Batch Commands

The Fabric runtime environment provides the following sets of Batch commands:

Instances Migration

BATCH LU ('LUI','LUI2','LUI3','LUI4') FABRIC_COMMAND="sync_instance LU.?" with ASYNC='true';

Batch Commands Summary

Command Name

Description

Example

BATCH <LU>[@<DC>] FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLY=true/false] [MAX_WORKERS_PER_NODE=<number>]];/h6>

Start the Batch process and sync all LU instances: - DC, specifies the DC name to force the Batch process in the specified DC. - AFFINITY, list of nodes and DCs to be used during the Batch process execution. - JOB_AFFINITY, affinity for the Batch process Job. - ASYNC, defines whether the Batch process should run in a sync or async mode. Default is False. - GENERATE_ENTITIES_FIRST, if set to True, generate all entities before processing them. - FABRIC_COMMAND, Fabric command to be executed by the Batch process which can be any command that includes a '?' to represent a singular Entity ID. One of the following commands must be set: (for Migration, "sync_instance .?", for Broadway, "broadway LU.SampleFlow SampleIID=?", for CDC republish, "cdc_republish_instance Customer.?") - ALLOW_MULTIPLY, when set to True, multiplies executions of the same Batch process command. Default is False. - MAX_NODES, maximum (random) nodes participating in the Batch process. - MAX_WORKERS_PER_NODE, enables setting a lower number of maximum workers to run on each node than the maximum number of workers defined in the config.ini file. (MAX_WORKERS_PER_NODE parameter).

BATCH CUSTOMER FABRIC_COMMAND="sync_instance Customer.?" with async=’true’; This command migrates all customers from the source systems into the Fabric CUSTOMER keyspace in the Fabric database.

BATCH <LU>[@<DC>].<IG> fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLY=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]];

Batch-processes a subset of the LUI based on the Instance Group specified by the <IG> parameter.

BATCH CUSTOMER.ig10CustomersList FABRIC_COMMAND="sync_instance CUST.?" with async=’true’;

This command migrates the customers defined in the ‘ig10CustomersList’ Instance Group into the CUSTOMER keyspace in the Fabric database.

BATCH <LU>[@<DC>] from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>' [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLY=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]];

Batch-processes a subset of the LUI based on a query to a source interface defined in the <db_interface> parameter.

BATCH CUSTOMER FROM CRM_DB USING (‘select customer_id from CUSTOMER where customer_id <= 1000’) FABRIC_COMMAND="sync_instance CUSTOMER.?" with async=’true’;

BATCH <LU>[@<DC>] from fabric fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLY=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]];

Batch-processes a subset of the LUI based on existing instances in Fabric in the entity table.

BATCH Customer from fabric fabric_command='sync_instance Customer.?';

BATCH <LU>[@<DC>].(<instance 1,instance 2,...>) fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLY=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]]

Batch-processes a subset of the LUI based on a list of instances defined in the <instance 1,instance 2,...> parameters list.

BATCH Customer.('100', '101', '102','103') FABRIC_COMMAND="sync_instance CUSTOMER.?" with async=’true’;

BATCHF

Runs the Batch process using a function that returns the LUI (IID). For example if LUI is stored in Fabric as a combination of ID + Environment name, use a function to concatenate the ID and Environment name.

The BATCHF command uses the exact same parameters as the BATCH command described above.

1) BATCHF Customer.batchFtest4().ig20 FABRIC_COMMAND='sync_instance Customer.?';

2) BATCHF Customer@DC1.batchFtest4() from HIS_DB using ('select customer_id from invoice where balance=12894') FABRIC_COMMAND='sync_instance Customer.?';

3) BATCHF Customer.batchFtest4().(‘1’,’2’,’3’) FABRIC_COMMAND='sync_instance Customer.?';

BATCH_RETRY '<batch_id>'

If the Batch process has not been completed, resumes a previous Batch process by reprocessing all failed or unhandled entities. Otherwise, it retries the failed entities only. If the Batch process is completed before the Retry command, Fabric gets the list of instances from the source DB. If the Batch process is completed before the Retry command, Fabric gets the list of failed entities from the batchprocess_entities_errors Cassandra table.

BATCH_RETRY ‘161f9717-bd93-4882-a3aa-7b58c1f61b27’;

CANCEL BATCH ['<batch_id>']

Cancels the last started Batch process coordinated by the current node. The Cancel command must be executed from the node that started the operation. When adding the '<batch_id>' parameter, the Batch process with the defined batch_ID is cancelled. Note that in this case, the Cancel command does not need to be run from the node coordinating the specific Batch process.

CANCEL BATCH;

CANCEL BATCH ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’;

Note that the KILL command can be used as well, using the batch ID (*bid*): KILL ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’;

Batch Monitoring Commands Summary

Command Name

Description

Example

BATCH_LIST [STATUS='<status>'] [FROM_DATE='<from_date>' [TO_DATE='<to_date>']] [FILTER=<filter criteria>]

If there are no arguments, lists all active Batch processes together with their respective status: - NEW, GENERATE_IID_LIST, IN_PROGRESS, FAILED, CANCELLED, DONE, ALL - FROM/TO_DATE, support DATE_FORMAT/DATETIME_FORMAT according to the configuration in the config.ini. - FILTER, filters Batch processes. The filter field must be populated by a string in a Fabric command in the Batch process. Note that the filter supports regex.

BATCH_LIST STATUS='ALL'; – list the history of all batch processes.

BATCH_LIST STATUS='ALL' FILTER='sync_instance'; list the history of all batch processes. This command returns the same results as the migrate_list STATUS = ‘ALL’; command.

BATCH_SUMMARY '<batch_id>'

This report brings a table holding a summary about node, DC and cluster levels.

BATCH_SUMMARY '35408af6-b26a-4243-bc95-f114335bfa5e'

BATCH_IN_PROCESS filter='<filter regex>'

Lists all currently running instances by batch processes in the cluster level and returns the following information:

  • Node ID
  • Batch process ID
  • Entity ID
  • LU type
  • Time at work (ms)
  • exeid
  • command
  • Filter, must be a regex compatible argument.
  • BATCH_IN_PROCESS filter='^(cust)*$'

    This command will return all running batch processes with names containing the "cust" pattern

    BATCH_DETAILS '<batch_id>' [STATUS='<status>'] [ENTITIES='<entity 1,entity 2,...>'] [AFFINITY='<Affinity>'] [LIMIT=<limit>] [SORT_BY_PROCESS_TIME=<true/false>]

    Displays the status of instances of a given Batch process ID: - STATUS, which can be either WAITING, COMPLETED, FAILED. - ENTITIES, lists of entities separated by a comma. - AFFINITY, DCs or nodes. - SORT_BY_PROCESS_TIME, if True, shows only the entities with the highest process time. If set, ignore all other parameters. - LIMIT, default limit defined in the config.ini if no limit is provided as an argument.

    BATCH_DETAILS 'a4587541-b12d-4329-affd-7c25516c9cde';

    Batch Commands Examples

    BATCH_LIST

    Command

    BATCH_LIST status='all'

    Result

    |Id                                  |Command                                                                                                                                                                |Start date         |End date           |Status|Created by|Completion %|Error|
    +------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------------+------+----------+------------+-----+
    |35408af6-b26a-4243-bc95-f114335bfa5e|BATCH AUTODATA_DELTA FROM idsFile USING ('select id from ids  limit 200') FABRIC_COMMAND="sync_instance AUTODATA_DELTA.?" with JOB_AFFINITY='10.21.2.102' ASYNC='true';|2020-08-12 12:20:07|2020-08-12 12:20:09|DONE  |          |100         |null |
    

    BATCH_SUMMARY

    Command

    BATCH_SUMMARY '35408af6-b26a-4243-bc95-f114335bfa5e'

    Result

    This command returns execution information and statistics for a given bid on each node in the execution:

    • Name, the Node ID.
    • %Completed, percentage of executions run on each node.
    • Ent/sec, average entities executed per seconds (pace - counting the average on the last 100 sec).

    Note that all other fields are self-explanatory.

    Batch Command with Embedded SQL Statements

    Instead of referring to an Instance Group, the Batch command can embed an SQL statement to select the entities on which the Batch command is executed:

    BATCH <LU> from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?'
    

    Example

    BATCH Customer from CRM_DB USING('select customer_id from Customer where customer_id <=10') FABRIC_COMMAND="sync_instance CUSTOMER.?";
    
    |Batch id                            |Execution succeeded|Execution failed|Total|Duration|Batch id                            |
    +------------------------------------+-------------------+----------------+-----+--------+------------------------------------+
    |83fade2f-2ae6-4359-8b7a-bdd1866d2191|10                 |0               |10   |1       |83fade2f-2ae6-4359-8b7a-bdd1866d2191|
    

    Previous

    Batch Commands

    The Fabric runtime environment provides the following sets of Batch commands:

    Instances Migration

    BATCH LU ('LUI','LUI2','LUI3','LUI4') FABRIC_COMMAND="sync_instance LU.?" with ASYNC='true';

    Batch Commands Summary

    Command Name

    Description

    Example

    BATCH <LU>[@<DC>] FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLY=true/false] [MAX_WORKERS_PER_NODE=<number>]];/h6>

    Start the Batch process and sync all LU instances: - DC, specifies the DC name to force the Batch process in the specified DC. - AFFINITY, list of nodes and DCs to be used during the Batch process execution. - JOB_AFFINITY, affinity for the Batch process Job. - ASYNC, defines whether the Batch process should run in a sync or async mode. Default is False. - GENERATE_ENTITIES_FIRST, if set to True, generate all entities before processing them. - FABRIC_COMMAND, Fabric command to be executed by the Batch process which can be any command that includes a '?' to represent a singular Entity ID. One of the following commands must be set: (for Migration, "sync_instance .?", for Broadway, "broadway LU.SampleFlow SampleIID=?", for CDC republish, "cdc_republish_instance Customer.?") - ALLOW_MULTIPLY, when set to True, multiplies executions of the same Batch process command. Default is False. - MAX_NODES, maximum (random) nodes participating in the Batch process. - MAX_WORKERS_PER_NODE, enables setting a lower number of maximum workers to run on each node than the maximum number of workers defined in the config.ini file. (MAX_WORKERS_PER_NODE parameter).

    BATCH CUSTOMER FABRIC_COMMAND="sync_instance Customer.?" with async=’true’; This command migrates all customers from the source systems into the Fabric CUSTOMER keyspace in the Fabric database.

    BATCH <LU>[@<DC>].<IG> fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLY=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]];

    Batch-processes a subset of the LUI based on the Instance Group specified by the <IG> parameter.

    BATCH CUSTOMER.ig10CustomersList FABRIC_COMMAND="sync_instance CUST.?" with async=’true’;

    This command migrates the customers defined in the ‘ig10CustomersList’ Instance Group into the CUSTOMER keyspace in the Fabric database.

    BATCH <LU>[@<DC>] from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>' [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLY=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]];

    Batch-processes a subset of the LUI based on a query to a source interface defined in the <db_interface> parameter.

    BATCH CUSTOMER FROM CRM_DB USING (‘select customer_id from CUSTOMER where customer_id <= 1000’) FABRIC_COMMAND="sync_instance CUSTOMER.?" with async=’true’;

    BATCH <LU>[@<DC>] from fabric fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLY=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]];

    Batch-processes a subset of the LUI based on existing instances in Fabric in the entity table.

    BATCH Customer from fabric fabric_command='sync_instance Customer.?';

    BATCH <LU>[@<DC>].(<instance 1,instance 2,...>) fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLY=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]]

    Batch-processes a subset of the LUI based on a list of instances defined in the <instance 1,instance 2,...> parameters list.

    BATCH Customer.('100', '101', '102','103') FABRIC_COMMAND="sync_instance CUSTOMER.?" with async=’true’;

    BATCHF

    Runs the Batch process using a function that returns the LUI (IID). For example if LUI is stored in Fabric as a combination of ID + Environment name, use a function to concatenate the ID and Environment name.

    The BATCHF command uses the exact same parameters as the BATCH command described above.

    1) BATCHF Customer.batchFtest4().ig20 FABRIC_COMMAND='sync_instance Customer.?';

    2) BATCHF Customer@DC1.batchFtest4() from HIS_DB using ('select customer_id from invoice where balance=12894') FABRIC_COMMAND='sync_instance Customer.?';

    3) BATCHF Customer.batchFtest4().(‘1’,’2’,’3’) FABRIC_COMMAND='sync_instance Customer.?';

    BATCH_RETRY '<batch_id>'

    If the Batch process has not been completed, resumes a previous Batch process by reprocessing all failed or unhandled entities. Otherwise, it retries the failed entities only. If the Batch process is completed before the Retry command, Fabric gets the list of instances from the source DB. If the Batch process is completed before the Retry command, Fabric gets the list of failed entities from the batchprocess_entities_errors Cassandra table.

    BATCH_RETRY ‘161f9717-bd93-4882-a3aa-7b58c1f61b27’;

    CANCEL BATCH ['<batch_id>']

    Cancels the last started Batch process coordinated by the current node. The Cancel command must be executed from the node that started the operation. When adding the '<batch_id>' parameter, the Batch process with the defined batch_ID is cancelled. Note that in this case, the Cancel command does not need to be run from the node coordinating the specific Batch process.

    CANCEL BATCH;

    CANCEL BATCH ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’;

    Note that the KILL command can be used as well, using the batch ID (*bid*): KILL ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’;

    Batch Monitoring Commands Summary

    Command Name

    Description

    Example

    BATCH_LIST [STATUS='<status>'] [FROM_DATE='<from_date>' [TO_DATE='<to_date>']] [FILTER=<filter criteria>]

    If there are no arguments, lists all active Batch processes together with their respective status: - NEW, GENERATE_IID_LIST, IN_PROGRESS, FAILED, CANCELLED, DONE, ALL - FROM/TO_DATE, support DATE_FORMAT/DATETIME_FORMAT according to the configuration in the config.ini. - FILTER, filters Batch processes. The filter field must be populated by a string in a Fabric command in the Batch process. Note that the filter supports regex.

    BATCH_LIST STATUS='ALL'; – list the history of all batch processes.

    BATCH_LIST STATUS='ALL' FILTER='sync_instance'; list the history of all batch processes. This command returns the same results as the migrate_list STATUS = ‘ALL’; command.

    BATCH_SUMMARY '<batch_id>'

    This report brings a table holding a summary about node, DC and cluster levels.

    BATCH_SUMMARY '35408af6-b26a-4243-bc95-f114335bfa5e'

    BATCH_IN_PROCESS filter='<filter regex>'

    Lists all currently running instances by batch processes in the cluster level and returns the following information:

  • Node ID
  • Batch process ID
  • Entity ID
  • LU type
  • Time at work (ms)
  • exeid
  • command
  • Filter, must be a regex compatible argument.
  • BATCH_IN_PROCESS filter='^(cust)*$'

    This command will return all running batch processes with names containing the "cust" pattern

    BATCH_DETAILS '<batch_id>' [STATUS='<status>'] [ENTITIES='<entity 1,entity 2,...>'] [AFFINITY='<Affinity>'] [LIMIT=<limit>] [SORT_BY_PROCESS_TIME=<true/false>]

    Displays the status of instances of a given Batch process ID: - STATUS, which can be either WAITING, COMPLETED, FAILED. - ENTITIES, lists of entities separated by a comma. - AFFINITY, DCs or nodes. - SORT_BY_PROCESS_TIME, if True, shows only the entities with the highest process time. If set, ignore all other parameters. - LIMIT, default limit defined in the config.ini if no limit is provided as an argument.

    BATCH_DETAILS 'a4587541-b12d-4329-affd-7c25516c9cde';

    Batch Commands Examples

    BATCH_LIST

    Command

    BATCH_LIST status='all'

    Result

    |Id                                  |Command                                                                                                                                                                |Start date         |End date           |Status|Created by|Completion %|Error|
    +------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------------+------+----------+------------+-----+
    |35408af6-b26a-4243-bc95-f114335bfa5e|BATCH AUTODATA_DELTA FROM idsFile USING ('select id from ids  limit 200') FABRIC_COMMAND="sync_instance AUTODATA_DELTA.?" with JOB_AFFINITY='10.21.2.102' ASYNC='true';|2020-08-12 12:20:07|2020-08-12 12:20:09|DONE  |          |100         |null |
    

    BATCH_SUMMARY

    Command

    BATCH_SUMMARY '35408af6-b26a-4243-bc95-f114335bfa5e'

    Result

    This command returns execution information and statistics for a given bid on each node in the execution:

    • Name, the Node ID.
    • %Completed, percentage of executions run on each node.
    • Ent/sec, average entities executed per seconds (pace - counting the average on the last 100 sec).

    Note that all other fields are self-explanatory.

    Batch Command with Embedded SQL Statements

    Instead of referring to an Instance Group, the Batch command can embed an SQL statement to select the entities on which the Batch command is executed:

    BATCH <LU> from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?'
    

    Example

    BATCH Customer from CRM_DB USING('select customer_id from Customer where customer_id <=10') FABRIC_COMMAND="sync_instance CUSTOMER.?";
    
    |Batch id                            |Execution succeeded|Execution failed|Total|Duration|Batch id                            |
    +------------------------------------+-------------------+----------------+-----+--------+------------------------------------+
    |83fade2f-2ae6-4359-8b7a-bdd1866d2191|10                 |0               |10   |1       |83fade2f-2ae6-4359-8b7a-bdd1866d2191|
    

    Previous