Building Entity Lists on a Task's LUs

The task execution process executes the task's LUs from parent to child.

Click for more information about the execution order of hierarchical LUs.

The task execution process builds an entity list for each LU, as described below. Root LUs and Children LUs require different steps:

Root LUs

The entity list of the root LUs is based on the task's entity selection method:

Entity Selection Method

Where the Entities are Taken From

Task Actions

Entity list

Run the task on the list of entities that are set in either the task itself or the task overridden execution parameters.

All task actions

Custom logic

Run the selected Broadway flow to get the entity list.

All task actions

Select a predefined entity list

Run either the SQL query or the Broadway flow that are defined in the MigrateList MTable for the LU.

Extract tasks

Select all entities of the selected version

Select all entities that are successfully extracted by the selected data version. The entities are selected from the TASK_EXECUTION_ENTITIES TDM DB table based on the task_execution_id of the selected data version.

Load data versioning tasks

Parameters

Select the entities based on the task's parameters from the LU parameters tables. These tables are created in the TDM DB for each BE and source environment combination.

Regular load tasks (Data Versioning is cleared)

Entity clone

Create duplications on the target environment of the first Entity ID returned by the task's subset. Only one LUI is created on the entity ID. Note that from TDM 9.0 onwards, entity clone is defined as an attribute on the target and it can be added to regardless of the subset's selection method.

Regular load tasks (Data Versioning is cleared)

Random selection

Randomly select the entities from the <LU Name>_<params> table.

Regular load tasks (Data Versioning is cleared)

Rule-based Synthetic Entities' Generation

Create a list of dummy entities for generating synthetic entities tasks.

Rule-based entites' generation.

AI-based Synthetic Entities' Generation

The AI generates the required entities with a dummy ID for each generated entity. TDM creates a unique LUI on each generated entity before importing the entity to Fabric. The mapping between the AI-generated ID and its Fabric LUI (imported LUI) is kept in tdm_ai_gen_iid_mapping TDM DB table.

AI-based entites' generation.

Running a Broadway Flow to Get the Task's Entity List

The task's entity list is generated by a Broadway flow in the following scenarios:

  • when using the Custom Logic selection method.

    Click here for more information about the custom logic modes.

  • when using the Select a predefined entity list selection method on an extract task when the MigrateList object is populated with a Broadway flow.

  • when generating a list of dummy entities for a synthetic data generation task. The prefix Synthetic is concatenated to each generated ID to create a dummy LUI, e.g., Synthetic_84.

    Click here for more information about synthetic data generation tasks.

Children LUs

The entity list of a child LU must include all IDs relating to parent IDs that have been successfully processed by the task execution.

Click for an execution of hierarchical BEs example.

The child LU's entity list is based on a JOIN of the task_execution_entities and the TDM relationship tables:

 

Task Actions

SQL Query
  • Extract
  • Extract + Load
  • Extract + Load + Reserve
  • Load
  • Load + Reserve
  • Rule-based Generate
  • Rule-based Generate + Load
  • Rule-based Generate + Load + Reserve

SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_relation_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and t.source_env = rel.source_env
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>
and rel.version_name = <empty string on a regular task
and the selected version name on a Data Versioning task>;

  • Extract + Load + Delete
  • Extract + Load + Delete+ Reserve
  • Load + Delete
  • Load + Delete + Reserve

SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_relation_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and t.source_env = rel.source_env
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>
and rel.version_name = <empty string on a regular task
and the selected version name on a Data Versioning task>
UNION
SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_rel_tar_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and rel.target_env = <target environment name>
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>;

  • Delete only

SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_rel_tar_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and rel.target_env = <target environment name>
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>;

Previous

Building Entity Lists on a Task's LUs

The task execution process executes the task's LUs from parent to child.

Click for more information about the execution order of hierarchical LUs.

The task execution process builds an entity list for each LU, as described below. Root LUs and Children LUs require different steps:

Root LUs

The entity list of the root LUs is based on the task's entity selection method:

Entity Selection Method

Where the Entities are Taken From

Task Actions

Entity list

Run the task on the list of entities that are set in either the task itself or the task overridden execution parameters.

All task actions

Custom logic

Run the selected Broadway flow to get the entity list.

All task actions

Select a predefined entity list

Run either the SQL query or the Broadway flow that are defined in the MigrateList MTable for the LU.

Extract tasks

Select all entities of the selected version

Select all entities that are successfully extracted by the selected data version. The entities are selected from the TASK_EXECUTION_ENTITIES TDM DB table based on the task_execution_id of the selected data version.

Load data versioning tasks

Parameters

Select the entities based on the task's parameters from the LU parameters tables. These tables are created in the TDM DB for each BE and source environment combination.

Regular load tasks (Data Versioning is cleared)

Entity clone

Create duplications on the target environment of the first Entity ID returned by the task's subset. Only one LUI is created on the entity ID. Note that from TDM 9.0 onwards, entity clone is defined as an attribute on the target and it can be added to regardless of the subset's selection method.

Regular load tasks (Data Versioning is cleared)

Random selection

Randomly select the entities from the <LU Name>_<params> table.

Regular load tasks (Data Versioning is cleared)

Rule-based Synthetic Entities' Generation

Create a list of dummy entities for generating synthetic entities tasks.

Rule-based entites' generation.

AI-based Synthetic Entities' Generation

The AI generates the required entities with a dummy ID for each generated entity. TDM creates a unique LUI on each generated entity before importing the entity to Fabric. The mapping between the AI-generated ID and its Fabric LUI (imported LUI) is kept in tdm_ai_gen_iid_mapping TDM DB table.

AI-based entites' generation.

Running a Broadway Flow to Get the Task's Entity List

The task's entity list is generated by a Broadway flow in the following scenarios:

  • when using the Custom Logic selection method.

    Click here for more information about the custom logic modes.

  • when using the Select a predefined entity list selection method on an extract task when the MigrateList object is populated with a Broadway flow.

  • when generating a list of dummy entities for a synthetic data generation task. The prefix Synthetic is concatenated to each generated ID to create a dummy LUI, e.g., Synthetic_84.

    Click here for more information about synthetic data generation tasks.

Children LUs

The entity list of a child LU must include all IDs relating to parent IDs that have been successfully processed by the task execution.

Click for an execution of hierarchical BEs example.

The child LU's entity list is based on a JOIN of the task_execution_entities and the TDM relationship tables:

 

Task Actions

SQL Query
  • Extract
  • Extract + Load
  • Extract + Load + Reserve
  • Load
  • Load + Reserve
  • Rule-based Generate
  • Rule-based Generate + Load
  • Rule-based Generate + Load + Reserve

SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_relation_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and t.source_env = rel.source_env
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>
and rel.version_name = <empty string on a regular task
and the selected version name on a Data Versioning task>;

  • Extract + Load + Delete
  • Extract + Load + Delete+ Reserve
  • Load + Delete
  • Load + Delete + Reserve

SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_relation_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and t.source_env = rel.source_env
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>
and rel.version_name = <empty string on a regular task
and the selected version name on a Data Versioning task>
UNION
SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_rel_tar_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and rel.target_env = <target environment name>
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>;

  • Delete only

SELECT rel.lu_type2_eid as child_entity_id
FROM task_execution_entities t, tdm_lu_type_rel_tar_eid rel
where t.task_execution_id= <task execution id>
and t.execution_status = 'completed'
and t.lu_name = <parent lu name>
and rel.target_env = <target environment name>
and t.lu_name = rel.lu_type_1
and t.iid = rel. lu_type1_eid
and rel.lu_type_2= <child lu name>;

Previous