Building Entity Lists on a Task's LUs

The task execution process executes the task's LUs from parent to child.

Click for more information about the execution order of hierarchical LUs.

The task execution process builds an entity list for each LU, as described below. Root LUs and Children LUs require different steps:

Root LUs

The entity list of the root LUs is based on the task's selection method:

Entity Selection Method

Where the Entities are Taken From

Task Types

Entity list

Run the task on the list of entities, set in the task itself or in the task overridden execution parameters

All task types

Custom logic

Run the selected Broadway flow to get the entity list

All task types

Select a predefined entity list

Run the SQL query or the Broadway flow defined in the trnMigrateList translation object for the LU.

Extract tasks

Select all entities of the selected version

Select all entities that are successfully extracted by the selected data version. The entities are selected from TASK_EXECUTION_ENTITIES TDM DB table based on the task_execution_id of the selected data version.

Load data versioning tasks

Parameters

Select the entities based on the task's parameters from a DB view, created in the TDM DB for each BE and source environment combination.

Regular load tasks (Data Versioning is cleared)

Entity clone

Create duplications on the target environment of the Entity ID, set in the task. Note that only one LUI is created on the entity ID.

Regular load tasks (Data Versioning is cleared)

Random selection

Randomly select the entities from the <LU Name>_<params> table.

Regular load tasks (Data Versioning is cleared)

Running a Broadway Flow to Get the Task's Entity List

The task's entity list is generated by a Broadway flow in the following scenarios:

The Broadway flow gets the entities for the task execution and inserts them into a dedicated Cassandra table in k2_tdm keyspace. A separate Cassandra entity table is created on each LU and has the following naming convention: [LU_NAME]_entity_list. This table is used by the task execution process to select the entity list for the task execution.

The tdmExecuteTask job runs the loadLuExternalEntityListTableJob TDM flow. This flow runs a BATCH job to run the loadLuExternalEntityListTable TDM Broadway flow with the following parameters:

  • LU_NAME: populated with the task's root LU.
  • EXTERNAL_TABLE_FLOW: populated with the name of the Broadway flow that gets the entity list.
  • NUM_OF_ENTITIES: populated with the number of entities for the task execution.

The batch_id is written to the task_execution_list TDM table to enable stopping the task execution during the entity list's Broadway flow execution.

The loadLuExternalEntityListTable Broadway flow creates the entity list's Cassandra table if needed and runs the Broadway flow that gets the entity list and populates the Cassandra table.

Children LUs

The entity list of a child LU must include all IDs related to parent IDs that have been successfully processed by the task execution.

Click for an execution of hierarchical BE example.

The generated entity list is based on a JOIN of the task_execution_entities and the TDM relationship tables:

Insert without Delete Load Task

Select the children IDs from the task_execution_entities and tdm_lu_type_relation_eid tables:

SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_relation_eid rel 
where t.task_execution_id= <task execution id> 
and t.execution_status = 'completed' 
and t.lu_name = <parent lu name> 
and t.source_env  = rel.source_env 
and t.lu_name = rel.lu_type_1 
and t.iid = rel. lu_type1_eid 
and rel.lu_type_2= <child lu name> 
and rel.version_name = <empty string on a regular task and the selected version name on a Data Flux task>;

Delete before Load Task

Select the children IDs from the task_execution_entities and TDM relationship tables: tdm_lu_type_relation_eid and tdm_lu_type_rel_tar_eid to get the children IDs from the source and target environments:

SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_relation_eid rel 
where t.task_execution_id= <task execution id> 
and t.execution_status = 'completed' 
and t.lu_name = <parent lu name> 
and t.source_env  = rel.source_env 
and t.lu_name = rel.lu_type_1 
and t.iid = rel. lu_type1_eid 
and rel.lu_type_2= <child lu name> 
and rel.version_name = <empty string on a regular task and the selected version name on a Data Flux task>
UNION
SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_rel_tar_eid rel 
where t.task_execution_id= <task execution id> 
and t.execution_status = 'completed' 
and t.lu_name = <parent lu name> 
and rel.target_env = <target environment name> 
and t.lu_name = rel.lu_type_1 
and t.iid = rel. lu_type1_eid 
and rel.lu_type_2= <child lu name>; 

Delete Entity without Load Task

Select the children IDs from the task_execution_entities and tdm_lu_type_rel_tar_eid to get the target children IDs:

SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_rel_tar_eid rel 
where t.task_execution_id= <task execution id> 
and t.execution_status = 'completed' 
and t.lu_name = <parent lu name> 
and rel.target_env = <target environment name> 
and t.lu_name = rel.lu_type_1 
and t.iid = rel. lu_type1_eid 
and rel.lu_type_2= <child lu name>; 

Previous

Building Entity Lists on a Task's LUs

The task execution process executes the task's LUs from parent to child.

Click for more information about the execution order of hierarchical LUs.

The task execution process builds an entity list for each LU, as described below. Root LUs and Children LUs require different steps:

Root LUs

The entity list of the root LUs is based on the task's selection method:

Entity Selection Method

Where the Entities are Taken From

Task Types

Entity list

Run the task on the list of entities, set in the task itself or in the task overridden execution parameters

All task types

Custom logic

Run the selected Broadway flow to get the entity list

All task types

Select a predefined entity list

Run the SQL query or the Broadway flow defined in the trnMigrateList translation object for the LU.

Extract tasks

Select all entities of the selected version

Select all entities that are successfully extracted by the selected data version. The entities are selected from TASK_EXECUTION_ENTITIES TDM DB table based on the task_execution_id of the selected data version.

Load data versioning tasks

Parameters

Select the entities based on the task's parameters from a DB view, created in the TDM DB for each BE and source environment combination.

Regular load tasks (Data Versioning is cleared)

Entity clone

Create duplications on the target environment of the Entity ID, set in the task. Note that only one LUI is created on the entity ID.

Regular load tasks (Data Versioning is cleared)

Random selection

Randomly select the entities from the <LU Name>_<params> table.

Regular load tasks (Data Versioning is cleared)

Running a Broadway Flow to Get the Task's Entity List

The task's entity list is generated by a Broadway flow in the following scenarios:

The Broadway flow gets the entities for the task execution and inserts them into a dedicated Cassandra table in k2_tdm keyspace. A separate Cassandra entity table is created on each LU and has the following naming convention: [LU_NAME]_entity_list. This table is used by the task execution process to select the entity list for the task execution.

The tdmExecuteTask job runs the loadLuExternalEntityListTableJob TDM flow. This flow runs a BATCH job to run the loadLuExternalEntityListTable TDM Broadway flow with the following parameters:

  • LU_NAME: populated with the task's root LU.
  • EXTERNAL_TABLE_FLOW: populated with the name of the Broadway flow that gets the entity list.
  • NUM_OF_ENTITIES: populated with the number of entities for the task execution.

The batch_id is written to the task_execution_list TDM table to enable stopping the task execution during the entity list's Broadway flow execution.

The loadLuExternalEntityListTable Broadway flow creates the entity list's Cassandra table if needed and runs the Broadway flow that gets the entity list and populates the Cassandra table.

Children LUs

The entity list of a child LU must include all IDs related to parent IDs that have been successfully processed by the task execution.

Click for an execution of hierarchical BE example.

The generated entity list is based on a JOIN of the task_execution_entities and the TDM relationship tables:

Insert without Delete Load Task

Select the children IDs from the task_execution_entities and tdm_lu_type_relation_eid tables:

SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_relation_eid rel 
where t.task_execution_id= <task execution id> 
and t.execution_status = 'completed' 
and t.lu_name = <parent lu name> 
and t.source_env  = rel.source_env 
and t.lu_name = rel.lu_type_1 
and t.iid = rel. lu_type1_eid 
and rel.lu_type_2= <child lu name> 
and rel.version_name = <empty string on a regular task and the selected version name on a Data Flux task>;

Delete before Load Task

Select the children IDs from the task_execution_entities and TDM relationship tables: tdm_lu_type_relation_eid and tdm_lu_type_rel_tar_eid to get the children IDs from the source and target environments:

SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_relation_eid rel 
where t.task_execution_id= <task execution id> 
and t.execution_status = 'completed' 
and t.lu_name = <parent lu name> 
and t.source_env  = rel.source_env 
and t.lu_name = rel.lu_type_1 
and t.iid = rel. lu_type1_eid 
and rel.lu_type_2= <child lu name> 
and rel.version_name = <empty string on a regular task and the selected version name on a Data Flux task>
UNION
SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_rel_tar_eid rel 
where t.task_execution_id= <task execution id> 
and t.execution_status = 'completed' 
and t.lu_name = <parent lu name> 
and rel.target_env = <target environment name> 
and t.lu_name = rel.lu_type_1 
and t.iid = rel. lu_type1_eid 
and rel.lu_type_2= <child lu name>; 

Delete Entity without Load Task

Select the children IDs from the task_execution_entities and tdm_lu_type_rel_tar_eid to get the target children IDs:

SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_rel_tar_eid rel 
where t.task_execution_id= <task execution id> 
and t.execution_status = 'completed' 
and t.lu_name = <parent lu name> 
and rel.target_env = <target environment name> 
and t.lu_name = rel.lu_type_1 
and t.iid = rel. lu_type1_eid 
and rel.lu_type_2= <child lu name>; 

Previous