Knowledge Base Academy Community My Tickets
Fabric 7.2
Fabric 7.2 Fabric 7.1 Fabric 7.0 Fabric 6.5 Fabric 6.4

Batch Commands

Note that all the following commands are also used and executed by the Batch Monitor dashboard to provide real-time visual insights on current or completed processes .

The Fabric runtime environment enables the execution of the following sets of Batch commands:

Instances Migration

BATCH LU ('LUI','LUI2','LUI3','LUI4') FABRIC_COMMAND="sync_instance LU.?" with ASYNC='true';

Batch Commands Summary

Command Name

Description

Example
BATCH <LU>[@<DC>] FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_WORKERS_PER_NODE=<number>]];

Start the Batch process and sync all LU instances. GENERATE_ENTITIES_FIRST - if set to true, generate all entities before processing them. ALLOW_MULTIPLE - if set to true, multiplies executions of the same Batch process command. Default is False. MAX_WORKERS_PER_NODE - enables setting a lower number of maximum workers to run on each node than the maximum number of workers defined in the MAX_NO_OF_WORKERS parameter in config.ini file. The number of workers allocated by a Fabric node cannot exceed the maximum number of workers set in the config.ini file.

BATCH CUSTOMER FABRIC_COMMAND="sync_instance Customer.?" with async=’true’; This command migrates all customers from the source systems into the Fabric CUSTOMER keyspace in the Fabric database.
BATCH <LU>[@<DC>].<IG> FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]];

Batch-processes a subset of the LUI based on the Instance Group specified by the <IG> parameter. Optional parameters are the same as described above.

BATCH CUSTOMER.ig10CustomersList FABRIC_COMMAND="sync_instance CUST.?" with async=’true’;

This command migrates the customers defined in the ‘ig10CustomersList’ Instance Group into the CUSTOMER keyspace in the Fabric database.

BATCH <LU>[@<DC>] from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>' [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]];

Batch-processes a subset of the LUI based on a query to a source interface defined in the <db_interface> parameter. Optional parameters are the same as described above.

BATCH CUSTOMER FROM CRM_DB USING (‘select customer_id from CUSTOMER where customer_id <= 1000’) FABRIC_COMMAND="sync_instance CUSTOMER.?" with async=’true’;

BATCH <LU>[@<DC>] from fabric fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]];

Batch-processes a subset of the LUI based on existing instances in Fabric in the entity table. Optional parameters are the same as described above.

BATCH Customer from fabric fabric_command='sync_instance Customer.?';
BATCH <LU>[@<DC>].(<instance 1,instance 2,...>) fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]]

Batch-processes a subset of the LUI based on a list of instances defined in the <instance 1,instance 2,...> parameters list. Optional parameters are the same as described above.

BATCH Customer.('100', '101', '102','103') FABRIC_COMMAND="sync_instance CUSTOMER.?" with async=’true’;
BATCHF

Runs the Batch process using a function that returns the LUI (IID). For example if LUI is stored in Fabric as a combination of ID + Environment name, use a function to concatenate the ID and Environment name.

The BATCHF command uses the same parameters as the BATCH command described above.

1) BATCHF Customer.batchFtest4().ig20 FABRIC_COMMAND='sync_instance Customer.?';

2) BATCHF Customer@DC1.batchFtest4() from HIS_DB using ('select customer_id from invoice where balance=12894') FABRIC_COMMAND='sync_instance Customer.?';

3) BATCHF Customer.batchFtest4().(‘1’,’2’,’3’) FABRIC_COMMAND='sync_instance Customer.?';

BATCH_PAUSE ['<batch_id>']

Pause a batch process operation.

BATCH_PAUSE ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’;

Pause the async batch process with the defined batch id.

BATCH_PAUSE;

Pause the last async batch process that was created in the current session.

BATCH_RETRY ['<batch_id>'] [allow_cancelled=true/false]

If a batch process execution was not completed: retry a batch process by processing all failed or not handled entities. If a batch process execution was completed, retry only failed entities. The allow_cancelled parameter allows a retry a cancelled batch process. Default is false.

BATCH_RETRY ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’;

Retry a batch process according to the given input batch id.

BATCH_RETRY ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’ allow_cancelled=true;

Retry a batch process according to the given input batch id even if the original batch process was cancelled.

BATCH_CANCEL ['<batch_id>']

Cancel a batch process operation.

BATCH_CANCEL ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’;

Cancel the async batch process according to the defined batch id.

BATCH_CANCEL

Cancel the last async batch process that was created in the current session.

BATCH_EDIT ['<batch_id>'] param1=<value1> param2=<value2>

Modify the specified parameters for the specified batch process. MAX_WORKERS_PER_NODE - Set the maximum workers per node for this batch. Note that this will not surpass the configured value for MAX_WORKERS_PER_NODE in the config.ini.

BATCH_EDIT ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’ MAX_WORKERS_PER_NODE=10 ;

Modify MAX_WORKERS_PER_NODE for the given batch id to 10.

Batch Monitoring Commands Summary

Command Name

Description

Example
BATCH_LIST [STATUS='<status>'] [FROM_DATE='<from_date>' [TO_DATE='<to_date>']] [FILTER=<filter criteria>]

If there are no arguments, lists all active Batch processes together with their respective status.

Statuses are: NEW, GENERATE_IID_LIST, IN_PROGRESS, FAILED, CANCELLED, DONE, ALL. FROM/TO_DATE - support DATE_FORMAT/DATETIME_FORMAT according to the configuration in the config.ini. FILTER - filters Batch processes. The filter field must be populated by a string and can be referred to the Batch command, Fabric command or the execution id of the batch process. The filter supports regex.

BATCH_LIST STATUS='ALL'; – list the history of all batch processes.

BATCH_LIST STATUS='ALL' FILTER='sync_instance'; list the history of the batch processes under the given filter. This command returns the same results as the migrate_list STATUS = ‘ALL’; command.
BATCH_INFO '<batch_id>'

Returns the general information of the batch process such as the batch command, Fabric command, LU name, Environment...

BATCH_INFO '35408af6-b26a-4243-bc95-f114335bfa5e'
BATCH_SUMMARY '<batch_id>'

This report brings a table holding a summary about node, DC and cluster levels.

BATCH_SUMMARY '35408af6-b26a-4243-bc95-f114335bfa5e'
BATCH_IN_PROCESS filter='<filter regex>'

Lists all currently running instances by batch processes in the cluster level and returns the following information:

  • Node ID
  • Batch process ID
  • Entity ID
  • LU type
  • Time at work (ms)
  • exeid
  • command
  • Filter, must be a regex compatible argument.

    		•

    BATCH_IN_PROCESS filter='^(cust)*$'

    This command will return all running batch processes with names containing the "cust" pattern
    BATCH_DETAILS '<batch_id>' [STATUS='<status>'] [ENTITIES='<entity 1,entity 2,...>'] [AFFINITY='<Affinity>'] [LIMIT=<limit>] [SORT_BY_PROCESS_TIME=<true/false>]

    Displays the status of instances of a given Batch process ID. This command returns up to 10,000 entities if no limit is set. The limitation of 10,000 is a default and is set by Fabric so that a result is given in a reasonable amount of time.

    STATUS - can be WAITING, COMPLETED, or FAILED. ENTITIES - lists of entities separated by a comma. AFFINITY - DCs or nodes. SORT_BY_PROCESS_TIME - if True, shows only the entities with the highest process time. If set, ignore all other parameters. LIMIT - limit a number of entities.

    BATCH_DETAILS 'a4587541-b12d-4329-affd-7c25516c9cde';

    Batch Commands Examples

    BATCH_LIST

    Command 

    BATCH_LIST status='all'

    Result 

    |Id                                  |Command                                                                                                                                                                |Start date         |End date           |Status|Created by|Completion %|Execution id                        |Error|
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------------+------+----------+------------+------------------------------------+-----+
|35408af6-b26a-4243-bc95-f114335bfa5e|BATCH AUTODATA_DELTA FROM idsFile USING ('select id from ids  limit 200') FABRIC_COMMAND="sync_instance AUTODATA_DELTA.?" with JOB_AFFINITY='10.21.2.102' ASYNC='true';|2020-08-12 12:20:07|2020-08-12 12:20:09|DONE  |          |100         |75fd2291-36d1-451c-8916-e60320a4e76f|null |

    BATCH_SUMMARY

    Command 

    BATCH_SUMMARY '35408af6-b26a-4243-bc95-f114335bfa5e'

    Result

    This command returns execution information and statistics for a given bid on each node in the execution:

    • Name, the Node ID.
    • % Completed, percentage of executions run on each node.
    • Ent/sec, average entities executed per seconds (pace - counting the average on the last 100 sec).

    Note that all other fields are self-explanatory.

    Batch Command with Embedded SQL Statements

    Instead of referring to an Instance Group, the Batch command can embed an SQL statement to select the entities on which the Batch command is executed.

    Command

    BATCH <LU> from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?'

    Example 

    BATCH Customer from CRM_DB USING('select customer_id from Customer where customer_id <=10') FABRIC_COMMAND="sync_instance CUSTOMER.?";

    Result

    |Batch id                            |Execution id                        |Execution succeeded|Execution failed|Total|Duration|
+------------------------------------+------------------------------------+-------------------+----------------+-----+--------+
|83fade2f-2ae6-4359-8b7a-bdd1866d2191|7b7f5a4b-2e2c-4f0e-90d5-865bac3484ee|10                 |0               |10   |1       |

    Previous

    Batch Commands

    Note that all the following commands are also used and executed by the Batch Monitor dashboard to provide real-time visual insights on current or completed processes .

    The Fabric runtime environment enables the execution of the following sets of Batch commands:

    Instances Migration

    BATCH LU ('LUI','LUI2','LUI3','LUI4') FABRIC_COMMAND="sync_instance LU.?" with ASYNC='true';

    Batch Commands Summary

    Command Name

    Description

    Example
    BATCH <LU>[@<DC>] FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_WORKERS_PER_NODE=<number>]];

    Start the Batch process and sync all LU instances. GENERATE_ENTITIES_FIRST - if set to true, generate all entities before processing them. ALLOW_MULTIPLE - if set to true, multiplies executions of the same Batch process command. Default is False. MAX_WORKERS_PER_NODE - enables setting a lower number of maximum workers to run on each node than the maximum number of workers defined in the MAX_NO_OF_WORKERS parameter in config.ini file. The number of workers allocated by a Fabric node cannot exceed the maximum number of workers set in the config.ini file.

    BATCH CUSTOMER FABRIC_COMMAND="sync_instance Customer.?" with async=’true’; This command migrates all customers from the source systems into the Fabric CUSTOMER keyspace in the Fabric database.
    BATCH <LU>[@<DC>].<IG> FABRIC_COMMAND='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]];

    Batch-processes a subset of the LUI based on the Instance Group specified by the <IG> parameter. Optional parameters are the same as described above.

    BATCH CUSTOMER.ig10CustomersList FABRIC_COMMAND="sync_instance CUST.?" with async=’true’;

    This command migrates the customers defined in the ‘ig10CustomersList’ Instance Group into the CUSTOMER keyspace in the Fabric database.

    BATCH <LU>[@<DC>] from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>' [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]];

    Batch-processes a subset of the LUI based on a query to a source interface defined in the <db_interface> parameter. Optional parameters are the same as described above.

    BATCH CUSTOMER FROM CRM_DB USING (‘select customer_id from CUSTOMER where customer_id <= 1000’) FABRIC_COMMAND="sync_instance CUSTOMER.?" with async=’true’;

    BATCH <LU>[@<DC>] from fabric fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]];

    Batch-processes a subset of the LUI based on existing instances in Fabric in the entity table. Optional parameters are the same as described above.

    BATCH Customer from fabric fabric_command='sync_instance Customer.?';
    BATCH <LU>[@<DC>].(<instance 1,instance 2,...>) fabric_command='<fabric command> ?' [WITH [AFFINITY='<affinity>'] [JOB_AFFINITY='<job affinity>'] [ASYNC=true/false] [GENERATE_ENTITIES_FIRST=true/false] [ALLOW_MULTIPLE=true/false] [MAX_NODES=<number>] [MAX_WORKERS_PER_NODE=<number>]]

    Batch-processes a subset of the LUI based on a list of instances defined in the <instance 1,instance 2,...> parameters list. Optional parameters are the same as described above.

    BATCH Customer.('100', '101', '102','103') FABRIC_COMMAND="sync_instance CUSTOMER.?" with async=’true’;
    BATCHF

    Runs the Batch process using a function that returns the LUI (IID). For example if LUI is stored in Fabric as a combination of ID + Environment name, use a function to concatenate the ID and Environment name.

    The BATCHF command uses the same parameters as the BATCH command described above.

    1) BATCHF Customer.batchFtest4().ig20 FABRIC_COMMAND='sync_instance Customer.?';

    2) BATCHF Customer@DC1.batchFtest4() from HIS_DB using ('select customer_id from invoice where balance=12894') FABRIC_COMMAND='sync_instance Customer.?';

    3) BATCHF Customer.batchFtest4().(‘1’,’2’,’3’) FABRIC_COMMAND='sync_instance Customer.?';

    BATCH_PAUSE ['<batch_id>']

    Pause a batch process operation.

    BATCH_PAUSE ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’;

    Pause the async batch process with the defined batch id.

    BATCH_PAUSE;

    Pause the last async batch process that was created in the current session.

    BATCH_RETRY ['<batch_id>'] [allow_cancelled=true/false]

    If a batch process execution was not completed: retry a batch process by processing all failed or not handled entities. If a batch process execution was completed, retry only failed entities. The allow_cancelled parameter allows a retry a cancelled batch process. Default is false.

    BATCH_RETRY ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’;

    Retry a batch process according to the given input batch id.

    BATCH_RETRY ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’ allow_cancelled=true;

    Retry a batch process according to the given input batch id even if the original batch process was cancelled.

    BATCH_CANCEL ['<batch_id>']

    Cancel a batch process operation.

    BATCH_CANCEL ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’;

    Cancel the async batch process according to the defined batch id.

    BATCH_CANCEL

    Cancel the last async batch process that was created in the current session.

    BATCH_EDIT ['<batch_id>'] param1=<value1> param2=<value2>

    Modify the specified parameters for the specified batch process. MAX_WORKERS_PER_NODE - Set the maximum workers per node for this batch. Note that this will not surpass the configured value for MAX_WORKERS_PER_NODE in the config.ini.

    BATCH_EDIT ‘568114fe-9ec8-4c9e-af11-6e3348eff6e9’ MAX_WORKERS_PER_NODE=10 ;

    Modify MAX_WORKERS_PER_NODE for the given batch id to 10.

    Batch Monitoring Commands Summary

    Command Name

    Description

    Example
    BATCH_LIST [STATUS='<status>'] [FROM_DATE='<from_date>' [TO_DATE='<to_date>']] [FILTER=<filter criteria>]

    If there are no arguments, lists all active Batch processes together with their respective status.

    Statuses are: NEW, GENERATE_IID_LIST, IN_PROGRESS, FAILED, CANCELLED, DONE, ALL. FROM/TO_DATE - support DATE_FORMAT/DATETIME_FORMAT according to the configuration in the config.ini. FILTER - filters Batch processes. The filter field must be populated by a string and can be referred to the Batch command, Fabric command or the execution id of the batch process. The filter supports regex.

    BATCH_LIST STATUS='ALL'; – list the history of all batch processes.

    BATCH_LIST STATUS='ALL' FILTER='sync_instance'; list the history of the batch processes under the given filter. This command returns the same results as the migrate_list STATUS = ‘ALL’; command.
    BATCH_INFO '<batch_id>'

    Returns the general information of the batch process such as the batch command, Fabric command, LU name, Environment...

    BATCH_INFO '35408af6-b26a-4243-bc95-f114335bfa5e'
    BATCH_SUMMARY '<batch_id>'

    This report brings a table holding a summary about node, DC and cluster levels.

    BATCH_SUMMARY '35408af6-b26a-4243-bc95-f114335bfa5e'
    BATCH_IN_PROCESS filter='<filter regex>'

    Lists all currently running instances by batch processes in the cluster level and returns the following information:

  • Node ID
  • Batch process ID
  • Entity ID
  • LU type
  • Time at work (ms)
  • exeid
  • command
  • Filter, must be a regex compatible argument.

    		•

    BATCH_IN_PROCESS filter='^(cust)*$'

    This command will return all running batch processes with names containing the "cust" pattern
    BATCH_DETAILS '<batch_id>' [STATUS='<status>'] [ENTITIES='<entity 1,entity 2,...>'] [AFFINITY='<Affinity>'] [LIMIT=<limit>] [SORT_BY_PROCESS_TIME=<true/false>]

    Displays the status of instances of a given Batch process ID. This command returns up to 10,000 entities if no limit is set. The limitation of 10,000 is a default and is set by Fabric so that a result is given in a reasonable amount of time.

    STATUS - can be WAITING, COMPLETED, or FAILED. ENTITIES - lists of entities separated by a comma. AFFINITY - DCs or nodes. SORT_BY_PROCESS_TIME - if True, shows only the entities with the highest process time. If set, ignore all other parameters. LIMIT - limit a number of entities.

    BATCH_DETAILS 'a4587541-b12d-4329-affd-7c25516c9cde';

    Batch Commands Examples

    BATCH_LIST

    Command 

    BATCH_LIST status='all'

    Result 

    |Id                                  |Command                                                                                                                                                                |Start date         |End date           |Status|Created by|Completion %|Execution id                        |Error|
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------+-------------------+------+----------+------------+------------------------------------+-----+
|35408af6-b26a-4243-bc95-f114335bfa5e|BATCH AUTODATA_DELTA FROM idsFile USING ('select id from ids  limit 200') FABRIC_COMMAND="sync_instance AUTODATA_DELTA.?" with JOB_AFFINITY='10.21.2.102' ASYNC='true';|2020-08-12 12:20:07|2020-08-12 12:20:09|DONE  |          |100         |75fd2291-36d1-451c-8916-e60320a4e76f|null |

    BATCH_SUMMARY

    Command 

    BATCH_SUMMARY '35408af6-b26a-4243-bc95-f114335bfa5e'

    Result

    This command returns execution information and statistics for a given bid on each node in the execution:

    • Name, the Node ID.
    • % Completed, percentage of executions run on each node.
    • Ent/sec, average entities executed per seconds (pace - counting the average on the last 100 sec).

    Note that all other fields are self-explanatory.

    Batch Command with Embedded SQL Statements

    Instead of referring to an Instance Group, the Batch command can embed an SQL statement to select the entities on which the Batch command is executed.

    Command

    BATCH <LU> from <db_interface> using ('<SQL>') fabric_command='<fabric command> ?'

    Example 

    BATCH Customer from CRM_DB USING('select customer_id from Customer where customer_id <=10') FABRIC_COMMAND="sync_instance CUSTOMER.?";

    Result

    |Batch id                            |Execution id                        |Execution succeeded|Execution failed|Total|Duration|
+------------------------------------+------------------------------------+-------------------+----------------+-----+--------+
|83fade2f-2ae6-4359-8b7a-bdd1866d2191|7b7f5a4b-2e2c-4f0e-90d5-865bac3484ee|10                 |0               |10   |1       |

    Previous

    © Copyright 2024 K2VIEW