Building Entity Lists on a Task's LUs
The task execution process executes the task's LUs from parent to child.
Click for more information about the execution order of hierarchical LUs.
The task execution process builds an entity list for each LU, as described below. Root LUs and Children LUs require different steps:
Root LUs
The entity list of the root LUs is based on the task's selection method:
|
Entity Selection Method |
Where the Entities are Taken From |
Task Types |
Entity list |
Run the task on the list of entities, set in the task itself or in the task overridden execution parameters |
All task types |
Custom logic |
Run the selected Broadway flow to get the entity list |
All task types |
Select a predefined entity list |
Run the SQL query or the Broadway flow defined in the trnMigrateList translation object for the LU. |
Extract tasks |
Select all entities of the selected version |
Select all entities that are successfully extracted by the selected data version. The entities are selected from TASK_EXECUTION_ENTITIES TDM DB table based on the task_execution_id of the selected data version. |
Load data versioning tasks |
Parameters |
Select the entities based on the task's parameters from a DB view, created in the TDM DB for each BE and source environment combination. |
Regular load tasks (Data Versioning is cleared) |
Entity clone |
Create duplications on the target environment of the Entity ID, set in the task. Note that only one LUI is created on the entity ID. |
Regular load tasks (Data Versioning is cleared) |
Random selection |
Randomly select the entities from the <LU Name>_<params> table. |
Regular load tasks (Data Versioning is cleared) |
Running a Broadway Flow to Get the Task's Entity List
The task's entity list is generated by a Broadway flow in the following scenarios:
- Using the Custom Logic selection method.
- Using the Select a predefined entity list selection method on an extract task when the trnMigrateList object is populated with a Broadway flow.
The Broadway flow gets the entities for the task execution and inserts them into a dedicated Cassandra table in the k2_tdm keyspace. A separate Cassandra entity table is created on each LU and has the following naming convention: [LU_NAME]_entity_list. The task execution process uses this table to select the entity list for the task execution.
The task execution process (tdmExecuteTask job) runs the createLuExternalEntityListTable Broadway flow to create the entity list's Cassandra table if needed. Then the task execution process runs a BATCH job to execute the Broadway flow that extracts the entity list and populates the Cassandra table. This Broadway flow always receives the following parameters:
- LU_NAME: populated with the task's root LU.
- EXTERNAL_TABLE_FLOW: populated with the name of the Broadway flow that gets the entity list.
- NUM_OF_ENTITIES: populated with the number of entities for the task execution.
The batch_id is written to the task_execution_list TDM table to enable stopping the task execution during the entity list's Broadway flow execution.
Children LUs
The entity list of a child LU must include all IDs related to parent IDs that have been successfully processed by the task execution.
Click for an execution of hierarchical BE example.
The generated entity list is based on a JOIN of the task_execution_entities and the TDM relationship tables:
Insert without Delete Load Task
Select the children IDs from the task_execution_entities and tdm_lu_type_relation_eid tables:
SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_relation_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and t.source_env = rel.source_env
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>
and rel.version_name = <empty string on a regular task and the selected version name on a Data Flux task>;
Delete before Load Task
Select the children IDs from the task_execution_entities and TDM relationship tables: tdm_lu_type_relation_eid and tdm_lu_type_rel_tar_eid to get the children IDs from the source and target environments:
SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_relation_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and t.source_env = rel.source_env
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>
and rel.version_name = <empty string on a regular task and the selected version name on a Data Flux task>
UNION
SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_rel_tar_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and rel.target_env = <target environment name>
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>;
Delete Entity without Load Task
Select the children IDs from the task_execution_entities and tdm_lu_type_rel_tar_eid to get the target children IDs:
SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_rel_tar_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and rel.target_env = <target environment name>
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>;
Building Entity Lists on a Task's LUs
The task execution process executes the task's LUs from parent to child.
Click for more information about the execution order of hierarchical LUs.
The task execution process builds an entity list for each LU, as described below. Root LUs and Children LUs require different steps:
Root LUs
The entity list of the root LUs is based on the task's selection method:
|
Entity Selection Method |
Where the Entities are Taken From |
Task Types |
Entity list |
Run the task on the list of entities, set in the task itself or in the task overridden execution parameters |
All task types |
Custom logic |
Run the selected Broadway flow to get the entity list |
All task types |
Select a predefined entity list |
Run the SQL query or the Broadway flow defined in the trnMigrateList translation object for the LU. |
Extract tasks |
Select all entities of the selected version |
Select all entities that are successfully extracted by the selected data version. The entities are selected from TASK_EXECUTION_ENTITIES TDM DB table based on the task_execution_id of the selected data version. |
Load data versioning tasks |
Parameters |
Select the entities based on the task's parameters from a DB view, created in the TDM DB for each BE and source environment combination. |
Regular load tasks (Data Versioning is cleared) |
Entity clone |
Create duplications on the target environment of the Entity ID, set in the task. Note that only one LUI is created on the entity ID. |
Regular load tasks (Data Versioning is cleared) |
Random selection |
Randomly select the entities from the <LU Name>_<params> table. |
Regular load tasks (Data Versioning is cleared) |
Running a Broadway Flow to Get the Task's Entity List
The task's entity list is generated by a Broadway flow in the following scenarios:
- Using the Custom Logic selection method.
- Using the Select a predefined entity list selection method on an extract task when the trnMigrateList object is populated with a Broadway flow.
The Broadway flow gets the entities for the task execution and inserts them into a dedicated Cassandra table in the k2_tdm keyspace. A separate Cassandra entity table is created on each LU and has the following naming convention: [LU_NAME]_entity_list. The task execution process uses this table to select the entity list for the task execution.
The task execution process (tdmExecuteTask job) runs the createLuExternalEntityListTable Broadway flow to create the entity list's Cassandra table if needed. Then the task execution process runs a BATCH job to execute the Broadway flow that extracts the entity list and populates the Cassandra table. This Broadway flow always receives the following parameters:
- LU_NAME: populated with the task's root LU.
- EXTERNAL_TABLE_FLOW: populated with the name of the Broadway flow that gets the entity list.
- NUM_OF_ENTITIES: populated with the number of entities for the task execution.
The batch_id is written to the task_execution_list TDM table to enable stopping the task execution during the entity list's Broadway flow execution.
Children LUs
The entity list of a child LU must include all IDs related to parent IDs that have been successfully processed by the task execution.
Click for an execution of hierarchical BE example.
The generated entity list is based on a JOIN of the task_execution_entities and the TDM relationship tables:
Insert without Delete Load Task
Select the children IDs from the task_execution_entities and tdm_lu_type_relation_eid tables:
SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_relation_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and t.source_env = rel.source_env
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>
and rel.version_name = <empty string on a regular task and the selected version name on a Data Flux task>;
Delete before Load Task
Select the children IDs from the task_execution_entities and TDM relationship tables: tdm_lu_type_relation_eid and tdm_lu_type_rel_tar_eid to get the children IDs from the source and target environments:
SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_relation_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and t.source_env = rel.source_env
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>
and rel.version_name = <empty string on a regular task and the selected version name on a Data Flux task>
UNION
SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_rel_tar_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and rel.target_env = <target environment name>
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>;
Delete Entity without Load Task
Select the children IDs from the task_execution_entities and tdm_lu_type_rel_tar_eid to get the target children IDs:
SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_rel_tar_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and rel.target_env = <target environment name>
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>;
