Note that all the following commands are also used and executed by the Batch Monitor dashboard to provide real-time visual insights on current or completed processes .
The Fabric runtime environment enables the execution of the following sets of Batch commands:
Instances Migration
BATCH LU ('LUI','LUI2','LUI3','LUI4') FABRIC_COMMAND="sync_instance LU.?" with ASYNC='true';
Command Name |
Description |
Example |
BATCH <LU>[@<DC>] FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_WORKERS_PER_NODE=<number>]]; |
Start the Batch process and sync all LU instances:
- LU, name of the Logical Unit
- DC, specifies the DC name to force the Batch process in the specified DC.
- FABRIC_COMMAND, Fabric command to be executed by the Batch process. This command can be any command that includes a '?' to represent a singular Entity ID. One of the following commands must be set: (for Migration, "sync_instance |
BATCH CUSTOMER FABRIC_COMMAND="sync_instance Customer.?" with async=’true’; This command migrates all customers from the source systems into the Fabric CUSTOMER keyspace in the Fabric database. |
BATCH <LU>[@<DC>].<IG> FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]]; |
Batch-processes a subset of the LUI based on the Instance Group specified by the <IG> parameter. Optional parameters are the same as described above. |
BATCH CUSTOMER.ig10CustomersList FABRIC_COMMAND="sync_instance CUST.?" with async=’true’; This command migrates the customers defined in the ‘ig10CustomersList’ Instance Group into the CUSTOMER keyspace in the Fabric database. |
BATCH <LU>[@<DC>] from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>' [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]]; |
Batch-processes a subset of the LUI based on a query to a source interface defined in the <db_interface> parameter. Optional parameters are the same as described above. |
BATCH CUSTOMER FROM CRM_DB USING (‘select customer_id from CUSTOMER where customer_id <= 1000’) FABRIC_COMMAND="sync_instance CUSTOMER.?" with async=’true’; |
BATCH <LU>[@<DC>] from fabric fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]]; |
Batch-processes a subset of the LUI based on existing instances in Fabric in the entity table. Optional parameters are the same as described above. |
BATCH Customer from fabric fabric_command='sync_instance Customer.?'; |
BATCH <LU>[@<DC>].(<instance 1,instance 2,...>) fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]] |
Batch-processes a subset of the LUI based on a list of instances defined in the <instance 1,instance 2,...> parameters list. Optional parameters are the same as described above. |
BATCH Customer.('100', '101', '102','103') FABRIC_COMMAND="sync_instance CUSTOMER.?" with async=’true’; |
BATCHF |
Runs the Batch process using a function that returns the LUI (IID). For example if LUI is stored in Fabric as a combination of ID + Environment name, use a function to concatenate the ID and Environment name. The BATCHF command uses the same parameters as the BATCH command described above. |
1) BATCHF Customer.batchFtest4().ig20 FABRIC_COMMAND='sync_instance Customer.?'; 2) BATCHF Customer@DC1.batchFtest4() from HIS_DB using ('select customer_id from invoice where balance=12894') FABRIC_COMMAND='sync_instance Customer.?'; 3) BATCHF Customer.batchFtest4().(‘1’,’2’,’3’) FABRIC_COMMAND='sync_instance Customer.?'; |
BATCH_RETRY '<batch_id>' [allow_cancelled=true/false] |
If the Batch process has not been completed, resumes a previous Batch process by reprocessing all failed or unhandled entities. Otherwise, it retries the failed entities only. If the Batch process is completed before the Retry command, Fabric gets the list of instances from the source DB. If the Batch process is completed before the Retry command, Fabric gets the list of failed entities from the batchprocess_entities_errors Cassandra table. allow_cancelled - indicates if the command should enable a retry of cancelled batch process. By default, this parameter is set to false. |
BATCH_RETRY ‘161f9717-bd93-4882-a3aa-7b58c1f61b27’; |
CANCEL BATCH ['<batch_id>'] |
Cancels the last started Batch process coordinated by the current node. The Cancel command must be executed from the node that started the operation. When adding the '<batch_id>' parameter, the Batch process with the defined batch_ID is cancelled. Note that in this case, the Cancel command does not need to be run from the node coordinating the specific Batch process. |
CANCEL BATCH; CANCEL BATCH ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; Note that the KILL command can be used as an alternative, using the batch ID (*bid*): KILL ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; |
BATCH_PAUSE ['<batch_id>'] |
Pause a batch process operation. Introduced in Fabric 6.5.1 |
BATCH_PAUSE ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; Pause the async batch process with the defined batch id. BATCH_PAUSE; Pause the last async batch process that was created in the current session. |
BATCH_RETRY ['<batch_id>'] [allow_cancelled=true/false] |
If a batch process execution was not completed: retry a batch process by processing all failed or not handled entities. or If a batch process execution was completed, retry only failed entities. **allow_cancelled** , allows a retry on a cancelled batch process. Default is false. Introduced in Fabric 6.5.1 |
BATCH_RETRY ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; Retry a batch process according to the given input batch id. BATCH_RETRY ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’ allow_cancelled=true; Retry a batch process according to the given input batch id even if the original batch process was cancelled. |
BATCH_CANCEL ['<batch_id>'] |
Cancel a batch process operation. Introduced in Fabric 6.5.1 |
BATCH_CANCEL ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; Cancel the async batch process according to the defined batch id. BATCH_CANCEL Cancel the last async batch process that was created in the current session. |
BATCH_EDIT '<param1=value1'> param2='<value2'> |
Modify the specified parameters for the specified batch process. MAX_WORKERS_PER_NODE - Set the maximum workers per node for this batch. Note that this will not surpass the configured value for MAX_WORKERS_PER_NODE in the config.ini. Introduced in Fabric 6.5.1 |
BATCH_EDIT ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’ MAX_WORKERS_PER_NODE=10 ; Modify MAX_WORKERS_PER_NODE for the given batch id to 10. |
###### BATCH_INFO [' |
Returns static information about the execution of the input batch ID (such as the batch command or the Fabric command). Introduced in Fabric 6.5.2 |
BATCH_INFO ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; |
Command
BATCH_LIST status='all'
Result
|Id |Command |Start date |End date |Status|Created by|Completion %|Execution id |Error|
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------------+------+----------+------------+------------------------------------+-----+
|35408af6-b26a-4243-bc95-f114335bfa5e|BATCH AUTODATA_DELTA FROM idsFile USING ('select id from ids limit 200') FABRIC_COMMAND="sync_instance AUTODATA_DELTA.?" with JOB_AFFINITY='10.21.2.102' ASYNC='true';|2020-08-12 12:20:07|2020-08-12 12:20:09|DONE | |100 |75fd2291-36d1-451c-8916-e60320a4e76f|null |
Command
BATCH_SUMMARY '35408af6-b26a-4243-bc95-f114335bfa5e'
Result
This command returns execution information and statistics for a given bid on each node in the execution:
Note that all other fields are self-explanatory.
Instead of referring to an Instance Group, the Batch command can embed an SQL statement to select the entities on which the Batch command is executed:
BATCH <LU> from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?'
Example
BATCH Customer from CRM_DB USING('select customer_id from Customer where customer_id <=10') FABRIC_COMMAND="sync_instance CUSTOMER.?";
|Batch id |Execution id |Execution succeeded|Execution failed|Total|Duration|
+------------------------------------+------------------------------------+-------------------+----------------+-----+--------+
|83fade2f-2ae6-4359-8b7a-bdd1866d2191|7b7f5a4b-2e2c-4f0e-90d5-865bac3484ee|10 |0 |10 |1 |
Note that all the following commands are also used and executed by the Batch Monitor dashboard to provide real-time visual insights on current or completed processes .
The Fabric runtime environment enables the execution of the following sets of Batch commands:
Instances Migration
BATCH LU ('LUI','LUI2','LUI3','LUI4') FABRIC_COMMAND="sync_instance LU.?" with ASYNC='true';
Command Name |
Description |
Example |
BATCH <LU>[@<DC>] FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_WORKERS_PER_NODE=<number>]]; |
Start the Batch process and sync all LU instances:
- LU, name of the Logical Unit
- DC, specifies the DC name to force the Batch process in the specified DC.
- FABRIC_COMMAND, Fabric command to be executed by the Batch process. This command can be any command that includes a '?' to represent a singular Entity ID. One of the following commands must be set: (for Migration, "sync_instance |
BATCH CUSTOMER FABRIC_COMMAND="sync_instance Customer.?" with async=’true’; This command migrates all customers from the source systems into the Fabric CUSTOMER keyspace in the Fabric database. |
BATCH <LU>[@<DC>].<IG> FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]]; |
Batch-processes a subset of the LUI based on the Instance Group specified by the <IG> parameter. Optional parameters are the same as described above. |
BATCH CUSTOMER.ig10CustomersList FABRIC_COMMAND="sync_instance CUST.?" with async=’true’; This command migrates the customers defined in the ‘ig10CustomersList’ Instance Group into the CUSTOMER keyspace in the Fabric database. |
BATCH <LU>[@<DC>] from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>' [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]]; |
Batch-processes a subset of the LUI based on a query to a source interface defined in the <db_interface> parameter. Optional parameters are the same as described above. |
BATCH CUSTOMER FROM CRM_DB USING (‘select customer_id from CUSTOMER where customer_id <= 1000’) FABRIC_COMMAND="sync_instance CUSTOMER.?" with async=’true’; |
BATCH <LU>[@<DC>] from fabric fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]]; |
Batch-processes a subset of the LUI based on existing instances in Fabric in the entity table. Optional parameters are the same as described above. |
BATCH Customer from fabric fabric_command='sync_instance Customer.?'; |
BATCH <LU>[@<DC>].(<instance 1,instance 2,...>) fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]] |
Batch-processes a subset of the LUI based on a list of instances defined in the <instance 1,instance 2,...> parameters list. Optional parameters are the same as described above. |
BATCH Customer.('100', '101', '102','103') FABRIC_COMMAND="sync_instance CUSTOMER.?" with async=’true’; |
BATCHF |
Runs the Batch process using a function that returns the LUI (IID). For example if LUI is stored in Fabric as a combination of ID + Environment name, use a function to concatenate the ID and Environment name. The BATCHF command uses the same parameters as the BATCH command described above. |
1) BATCHF Customer.batchFtest4().ig20 FABRIC_COMMAND='sync_instance Customer.?'; 2) BATCHF Customer@DC1.batchFtest4() from HIS_DB using ('select customer_id from invoice where balance=12894') FABRIC_COMMAND='sync_instance Customer.?'; 3) BATCHF Customer.batchFtest4().(‘1’,’2’,’3’) FABRIC_COMMAND='sync_instance Customer.?'; |
BATCH_RETRY '<batch_id>' [allow_cancelled=true/false] |
If the Batch process has not been completed, resumes a previous Batch process by reprocessing all failed or unhandled entities. Otherwise, it retries the failed entities only. If the Batch process is completed before the Retry command, Fabric gets the list of instances from the source DB. If the Batch process is completed before the Retry command, Fabric gets the list of failed entities from the batchprocess_entities_errors Cassandra table. allow_cancelled - indicates if the command should enable a retry of cancelled batch process. By default, this parameter is set to false. |
BATCH_RETRY ‘161f9717-bd93-4882-a3aa-7b58c1f61b27’; |
CANCEL BATCH ['<batch_id>'] |
Cancels the last started Batch process coordinated by the current node. The Cancel command must be executed from the node that started the operation. When adding the '<batch_id>' parameter, the Batch process with the defined batch_ID is cancelled. Note that in this case, the Cancel command does not need to be run from the node coordinating the specific Batch process. |
CANCEL BATCH; CANCEL BATCH ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; Note that the KILL command can be used as an alternative, using the batch ID (*bid*): KILL ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; |
BATCH_PAUSE ['<batch_id>'] |
Pause a batch process operation. Introduced in Fabric 6.5.1 |
BATCH_PAUSE ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; Pause the async batch process with the defined batch id. BATCH_PAUSE; Pause the last async batch process that was created in the current session. |
BATCH_RETRY ['<batch_id>'] [allow_cancelled=true/false] |
If a batch process execution was not completed: retry a batch process by processing all failed or not handled entities. or If a batch process execution was completed, retry only failed entities. **allow_cancelled** , allows a retry on a cancelled batch process. Default is false. Introduced in Fabric 6.5.1 |
BATCH_RETRY ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; Retry a batch process according to the given input batch id. BATCH_RETRY ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’ allow_cancelled=true; Retry a batch process according to the given input batch id even if the original batch process was cancelled. |
BATCH_CANCEL ['<batch_id>'] |
Cancel a batch process operation. Introduced in Fabric 6.5.1 |
BATCH_CANCEL ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; Cancel the async batch process according to the defined batch id. BATCH_CANCEL Cancel the last async batch process that was created in the current session. |
BATCH_EDIT '<param1=value1'> param2='<value2'> |
Modify the specified parameters for the specified batch process. MAX_WORKERS_PER_NODE - Set the maximum workers per node for this batch. Note that this will not surpass the configured value for MAX_WORKERS_PER_NODE in the config.ini. Introduced in Fabric 6.5.1 |
BATCH_EDIT ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’ MAX_WORKERS_PER_NODE=10 ; Modify MAX_WORKERS_PER_NODE for the given batch id to 10. |
###### BATCH_INFO [' |
Returns static information about the execution of the input batch ID (such as the batch command or the Fabric command). Introduced in Fabric 6.5.2 |
BATCH_INFO ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; |
Command
BATCH_LIST status='all'
Result
|Id |Command |Start date |End date |Status|Created by|Completion %|Execution id |Error|
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------------+------+----------+------------+------------------------------------+-----+
|35408af6-b26a-4243-bc95-f114335bfa5e|BATCH AUTODATA_DELTA FROM idsFile USING ('select id from ids limit 200') FABRIC_COMMAND="sync_instance AUTODATA_DELTA.?" with JOB_AFFINITY='10.21.2.102' ASYNC='true';|2020-08-12 12:20:07|2020-08-12 12:20:09|DONE | |100 |75fd2291-36d1-451c-8916-e60320a4e76f|null |
Command
BATCH_SUMMARY '35408af6-b26a-4243-bc95-f114335bfa5e'
Result
This command returns execution information and statistics for a given bid on each node in the execution:
Note that all other fields are self-explanatory.
Instead of referring to an Instance Group, the Batch command can embed an SQL statement to select the entities on which the Batch command is executed:
BATCH <LU> from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?'
Example
BATCH Customer from CRM_DB USING('select customer_id from Customer where customer_id <=10') FABRIC_COMMAND="sync_instance CUSTOMER.?";
|Batch id |Execution id |Execution succeeded|Execution failed|Total|Duration|
+------------------------------------+------------------------------------+-------------------+----------------+-----+--------+
|83fade2f-2ae6-4359-8b7a-bdd1866d2191|7b7f5a4b-2e2c-4f0e-90d5-865bac3484ee|10 |0 |10 |1 |