Note that all the following commands are also used and executed by the Batch Monitor dashboard to provide real-time visual insights on current or completed processes .
The Fabric runtime environment enables the execution of the following sets of Batch commands:
Instances Migration
BATCH LU ('LUI','LUI2','LUI3','LUI4') FABRIC_COMMAND="sync_instance LU.?" with ASYNC='true';
Command Name |
Description |
Example |
BATCH <LU>[@<DC>] FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_WORKERS_PER_NODE=<number>] [ESTIMATED_ENTITIES_COUNT=<number>]; |
Start the Batch process and sync all LU instances. GENERATE_ENTITIES_FIRST - if set to true, generate all entities before processing them. ALLOW_MULTIPLE - if set to true, multiplies executions of the same Batch process command. Default is False. MAX_WORKERS_PER_NODE - enables setting a lower number of maximum workers to run on each node than the maximum number of workers defined in the MAX_NO_OF_WORKERS parameter in config.ini file. The number of workers allocated by a Fabric node cannot exceed the maximum number of workers set in the config.ini file. ESTIMATED_ENTITIES_COUNT - present the estimated number of entities to be processed, should be used in case of feeding a batch with stream of instances in such a case during the monitoring process a user wants to know what is the predicted number of instances to be processed. |
BATCH CUSTOMER FABRIC_COMMAND="sync_instance Customer.?" with async=’true’; This command migrates all customers from the source systems into the Fabric CUSTOMER keyspace in the Fabric database. |
BATCH <LU>[@<DC>].<IG> FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>] [ESTIMATED_ENTITIES_COUNT=<number>]; |
Batch-processes a subset of the LUI based on the Instance Group specified by the <IG> parameter. Optional parameters are the same as described above. |
BATCH CUSTOMER.ig10CustomersList FABRIC_COMMAND="sync_instance CUST.?" with async=’true’; This command migrates the customers defined in the ‘ig10CustomersList’ Instance Group into the CUSTOMER keyspace in the Fabric database. |
BATCH <LU>[@<DC>] from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>' [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>] [ESTIMATED_ENTITIES_COUNT=<number>]; |
Batch-processes a subset of the LUI based on a query to a source interface defined in the <db_interface> parameter. Optional parameters are the same as described above. |
BATCH CUSTOMER FROM CRM_DB USING (‘select customer_id from CUSTOMER where customer_id <= 1000’) FABRIC_COMMAND="sync_instance CUSTOMER.?" with async=’true’; |
BATCH <LU>[@<DC>] from fabric fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>] [ESTIMATED_ENTITIES_COUNT=<number>]; |
Batch-processes a subset of the LUI based on existing instances in Fabric in the entity table. Optional parameters are the same as described above. |
BATCH Customer from fabric fabric_command='sync_instance Customer.?'; |
BATCH <LU>[@<DC>] from MessageQueueInterface using('consumed messages topic name') fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>] [ESTIMATED_ENTITIES_COUNT=<number>]] |
Batch-processes a subset of the LUI based on a list of instances defined in the messaging queue consumed from the defined topic. To end such a batch process a batch_cancel command should run. this option was added to Fabric 7.2.2 HF14 and on to add the ability to run a batch dynamically on set of instances defined as input in a message queue, such as Kafka. |
BATCH Customer from KafkaInterface1 using ('topic1') FABRIC_COMMAND="sync_instance CUSTOMER.?" with async="true"; |
BATCH <LU>[@<DC>].<IG> FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>] [ESTIMATED_ENTITIES_COUNT=<number>]; |
Batch-processes a subset of the LUI based on the Instance Group specified by the <IG> parameter. Optional parameters are the same as described above. |
BATCH CUSTOMER.ig10CustomersList FABRIC_COMMAND="sync_instance CUST.?" with async=’true’; This command migrates the customers defined in the ‘ig10CustomersList’ Instance Group into the CUSTOMER keyspace in the Fabric database. |
BATCHF |
Runs the Batch process using a function that returns the LUI (IID). For example if LUI is stored in Fabric as a combination of ID + Environment name, use a function to concatenate the ID and Environment name. The BATCHF command uses the same parameters as the BATCH command described above. |
1) BATCHF Customer.batchFtest4().ig20 FABRIC_COMMAND='sync_instance Customer.?'; 2) BATCHF Customer@DC1.batchFtest4() from HIS_DB using ('select customer_id from invoice where balance=12894') FABRIC_COMMAND='sync_instance Customer.?'; 3) BATCHF Customer.batchFtest4().(‘1’,’2’,’3’) FABRIC_COMMAND='sync_instance Customer.?'; |
BATCH_PAUSE ['<batch_id>'] |
Pause a batch process operation. |
BATCH_PAUSE ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; Pause the async batch process with the defined batch id. BATCH_PAUSE; Pause the last async batch process that was created in the current session. |
BATCH_RETRY ['<batch_id>'] [allow_cancelled=true/false] |
If a batch process execution was not completed: retry a batch process by processing all failed or not handled entities. If a batch process execution was completed, retry only failed entities. The allow_cancelled parameter allows a retry a cancelled batch process. Default is false. |
BATCH_RETRY ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; Retry a batch process according to the given input batch id. BATCH_RETRY ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’ allow_cancelled=true; Retry a batch process according to the given input batch id even if the original batch process was cancelled. |
BATCH_CANCEL ['<batch_id>'] |
Cancel a batch process operation. |
BATCH_CANCEL ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; Cancel the async batch process according to the defined batch id. BATCH_CANCEL Cancel the last async batch process that was created in the current session. |
BATCH_EDIT ['<batch_id>'] param1=<value1> param2=<value2> |
Modify the specified parameters for the specified batch process. MAX_WORKERS_PER_NODE - Set the maximum workers per node for this batch. Note that this will not surpass the configured value for MAX_WORKERS_PER_NODE in the config.ini. |
BATCH_EDIT ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’ MAX_WORKERS_PER_NODE=10 ; Modify MAX_WORKERS_PER_NODE for the given batch id to 10. |
Command
BATCH_LIST status='all'
Result
|Id |Command |Start date |End date |Status|Created by|Completion %|Execution id |Error|
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------------+------+----------+------------+------------------------------------+-----+
|35408af6-b26a-4243-bc95-f114335bfa5e|BATCH AUTODATA_DELTA FROM idsFile USING ('select id from ids limit 200') FABRIC_COMMAND="sync_instance AUTODATA_DELTA.?" with JOB_AFFINITY='10.21.2.102' ASYNC='true';|2020-08-12 12:20:07|2020-08-12 12:20:09|DONE | |100 |75fd2291-36d1-451c-8916-e60320a4e76f|null |
Command
BATCH_SUMMARY '35408af6-b26a-4243-bc95-f114335bfa5e'
Result
This command returns execution information and statistics for a given bid on each node in the execution:
Note that all other fields are self-explanatory.
Instead of referring to an Instance Group, the Batch command can embed an SQL statement to select the entities on which the Batch command is executed.
Command
BATCH <LU> from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?'
Example
BATCH Customer from CRM_DB USING('select customer_id from Customer where customer_id <=10') FABRIC_COMMAND="sync_instance CUSTOMER.?";
Result
|Batch id |Execution id |Execution succeeded|Execution failed|Total|Duration|
+------------------------------------+------------------------------------+-------------------+----------------+-----+--------+
|83fade2f-2ae6-4359-8b7a-bdd1866d2191|7b7f5a4b-2e2c-4f0e-90d5-865bac3484ee|10 |0 |10 |1 |
Note that all the following commands are also used and executed by the Batch Monitor dashboard to provide real-time visual insights on current or completed processes .
The Fabric runtime environment enables the execution of the following sets of Batch commands:
Instances Migration
BATCH LU ('LUI','LUI2','LUI3','LUI4') FABRIC_COMMAND="sync_instance LU.?" with ASYNC='true';
Command Name |
Description |
Example |
BATCH <LU>[@<DC>] FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_WORKERS_PER_NODE=<number>] [ESTIMATED_ENTITIES_COUNT=<number>]; |
Start the Batch process and sync all LU instances. GENERATE_ENTITIES_FIRST - if set to true, generate all entities before processing them. ALLOW_MULTIPLE - if set to true, multiplies executions of the same Batch process command. Default is False. MAX_WORKERS_PER_NODE - enables setting a lower number of maximum workers to run on each node than the maximum number of workers defined in the MAX_NO_OF_WORKERS parameter in config.ini file. The number of workers allocated by a Fabric node cannot exceed the maximum number of workers set in the config.ini file. ESTIMATED_ENTITIES_COUNT - present the estimated number of entities to be processed, should be used in case of feeding a batch with stream of instances in such a case during the monitoring process a user wants to know what is the predicted number of instances to be processed. |
BATCH CUSTOMER FABRIC_COMMAND="sync_instance Customer.?" with async=’true’; This command migrates all customers from the source systems into the Fabric CUSTOMER keyspace in the Fabric database. |
BATCH <LU>[@<DC>].<IG> FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>] [ESTIMATED_ENTITIES_COUNT=<number>]; |
Batch-processes a subset of the LUI based on the Instance Group specified by the <IG> parameter. Optional parameters are the same as described above. |
BATCH CUSTOMER.ig10CustomersList FABRIC_COMMAND="sync_instance CUST.?" with async=’true’; This command migrates the customers defined in the ‘ig10CustomersList’ Instance Group into the CUSTOMER keyspace in the Fabric database. |
BATCH <LU>[@<DC>] from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>' [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>] [ESTIMATED_ENTITIES_COUNT=<number>]; |
Batch-processes a subset of the LUI based on a query to a source interface defined in the <db_interface> parameter. Optional parameters are the same as described above. |
BATCH CUSTOMER FROM CRM_DB USING (‘select customer_id from CUSTOMER where customer_id <= 1000’) FABRIC_COMMAND="sync_instance CUSTOMER.?" with async=’true’; |
BATCH <LU>[@<DC>] from fabric fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>] [ESTIMATED_ENTITIES_COUNT=<number>]; |
Batch-processes a subset of the LUI based on existing instances in Fabric in the entity table. Optional parameters are the same as described above. |
BATCH Customer from fabric fabric_command='sync_instance Customer.?'; |
BATCH <LU>[@<DC>] from MessageQueueInterface using('consumed messages topic name') fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>] [ESTIMATED_ENTITIES_COUNT=<number>]] |
Batch-processes a subset of the LUI based on a list of instances defined in the messaging queue consumed from the defined topic. To end such a batch process a batch_cancel command should run. this option was added to Fabric 7.2.2 HF14 and on to add the ability to run a batch dynamically on set of instances defined as input in a message queue, such as Kafka. |
BATCH Customer from KafkaInterface1 using ('topic1') FABRIC_COMMAND="sync_instance CUSTOMER.?" with async="true"; |
BATCH <LU>[@<DC>].<IG> FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>] [ESTIMATED_ENTITIES_COUNT=<number>]; |
Batch-processes a subset of the LUI based on the Instance Group specified by the <IG> parameter. Optional parameters are the same as described above. |
BATCH CUSTOMER.ig10CustomersList FABRIC_COMMAND="sync_instance CUST.?" with async=’true’; This command migrates the customers defined in the ‘ig10CustomersList’ Instance Group into the CUSTOMER keyspace in the Fabric database. |
BATCHF |
Runs the Batch process using a function that returns the LUI (IID). For example if LUI is stored in Fabric as a combination of ID + Environment name, use a function to concatenate the ID and Environment name. The BATCHF command uses the same parameters as the BATCH command described above. |
1) BATCHF Customer.batchFtest4().ig20 FABRIC_COMMAND='sync_instance Customer.?'; 2) BATCHF Customer@DC1.batchFtest4() from HIS_DB using ('select customer_id from invoice where balance=12894') FABRIC_COMMAND='sync_instance Customer.?'; 3) BATCHF Customer.batchFtest4().(‘1’,’2’,’3’) FABRIC_COMMAND='sync_instance Customer.?'; |
BATCH_PAUSE ['<batch_id>'] |
Pause a batch process operation. |
BATCH_PAUSE ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; Pause the async batch process with the defined batch id. BATCH_PAUSE; Pause the last async batch process that was created in the current session. |
BATCH_RETRY ['<batch_id>'] [allow_cancelled=true/false] |
If a batch process execution was not completed: retry a batch process by processing all failed or not handled entities. If a batch process execution was completed, retry only failed entities. The allow_cancelled parameter allows a retry a cancelled batch process. Default is false. |
BATCH_RETRY ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; Retry a batch process according to the given input batch id. BATCH_RETRY ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’ allow_cancelled=true; Retry a batch process according to the given input batch id even if the original batch process was cancelled. |
BATCH_CANCEL ['<batch_id>'] |
Cancel a batch process operation. |
BATCH_CANCEL ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’; Cancel the async batch process according to the defined batch id. BATCH_CANCEL Cancel the last async batch process that was created in the current session. |
BATCH_EDIT ['<batch_id>'] param1=<value1> param2=<value2> |
Modify the specified parameters for the specified batch process. MAX_WORKERS_PER_NODE - Set the maximum workers per node for this batch. Note that this will not surpass the configured value for MAX_WORKERS_PER_NODE in the config.ini. |
BATCH_EDIT ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’ MAX_WORKERS_PER_NODE=10 ; Modify MAX_WORKERS_PER_NODE for the given batch id to 10. |
Command
BATCH_LIST status='all'
Result
|Id |Command |Start date |End date |Status|Created by|Completion %|Execution id |Error|
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------------+------+----------+------------+------------------------------------+-----+
|35408af6-b26a-4243-bc95-f114335bfa5e|BATCH AUTODATA_DELTA FROM idsFile USING ('select id from ids limit 200') FABRIC_COMMAND="sync_instance AUTODATA_DELTA.?" with JOB_AFFINITY='10.21.2.102' ASYNC='true';|2020-08-12 12:20:07|2020-08-12 12:20:09|DONE | |100 |75fd2291-36d1-451c-8916-e60320a4e76f|null |
Command
BATCH_SUMMARY '35408af6-b26a-4243-bc95-f114335bfa5e'
Result
This command returns execution information and statistics for a given bid on each node in the execution:
Note that all other fields are self-explanatory.
Instead of referring to an Instance Group, the Batch command can embed an SQL statement to select the entities on which the Batch command is executed.
Command
BATCH <LU> from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?'
Example
BATCH Customer from CRM_DB USING('select customer_id from Customer where customer_id <=10') FABRIC_COMMAND="sync_instance CUSTOMER.?";
Result
|Batch id |Execution id |Execution succeeded|Execution failed|Total|Duration|
+------------------------------------+------------------------------------+-------------------+----------------+-----+--------+
|83fade2f-2ae6-4359-8b7a-bdd1866d2191|7b7f5a4b-2e2c-4f0e-90d5-865bac3484ee|10 |0 |10 |1 |