TDM Extract Task
An Extract task extracts the selected entities and/or Reference tables from the selected source environment. The data can be saved in Fabric (TDM warehouse) for a later use.
An Extract task contains the following tabs:
- General
- Additional Execution Parameters
- Requested Entities - opens if the Data Type contains entities.
- Task Scheduling
When checking the Set Task Variables setting, a new Task Variables tab opens.
When setting the Data Type by checking the Reference checkbox, a Reference tab opens.
Additional Execution Parameters Tab
The following execution parameters are set on Extract tasks:
Data Versioning
- Check the checkbox to set the task mode to Data Versioning.
- Leave the Data Versioning checkbox unchecked to create a regular mode task.
Data Type
Check the Entities checkbox to extract the selected entities and/or the Reference checkbox to extract the selected reference tables into the TDM warehouse (Fabric).
Click here for more information about the reference handling.
Set Sync Policy
This setting enables the user to change the default LUI sync mode (Sync ON) and to extract the LUI from the data source whenever the task is executed (Sync FORCE).
Note that this setting is only available when the Data Versioning checkbox is clear (regular task). If the Data Versioning checkbox is checked, each task execution extracts the data from the data source and creates a new LUI.
Retention Period
The retention period is set on the extracted entities and/or Reference tables. When this period ends, the task's entities and/or Reference tables are automatically deleted from Fabric and are no longer available.
Retention Period Values
Do not Delete - do not delete from Fabric.
Do not Retain - avoid saving the task's entities and/or Reference tables in Fabric (instead of saving and deleting). This option can be used, for example, to run an Extract task on a large subset of entities in order to populate the TDM parameter’s tables on each entity without saving the entities into Fabric.
Setting of a time measuring unit (minutes/hours/days…) and a value. For example, save the data in Fabric for 2 days. After 2 days the data is automatically deleted from Fabric.
Note that the retention period can be set in minutes, hours, days, weeks or years, depending on the maximum retention period set in the TDM DB. Both parameters - default retention period and maximum retention period - are set in the TDM DB.
Default Retention Period
- Data versioning checkbox is clear - do not Delete.
- Data versioning checkbox is checked - 5 days. A retention period must be set on a Data Versioning Extract task, i.e., the retention period must be set to a value greater than zero when the Data Versioning checkbox is checked.
Additional Execution Parameters
Set Task Variables
Check to open the Task Variables tab and set the variable value on a task level.
Mask Sensitive Data
TDM 8.1 added a checkbox to the Environment window to indicate if the environment has a sensitive data that must be stored in Fabric with masked values on PII fields. If the task's source value is defined as containing sensitive data, the TDM task window display the PII icon to indicate to the user that the source sensitive data is masked.
Post Execution Processes
Select all, partial or one post execution process of the selected BE.
Requested Entities Tab
This tab opens when the task's Data Type includes entities. This tab defines the subset of entities for the task:
The following selection methods are available on extract tasks:
Select a Predefined Entity List: run the SQL query or the Broadway flow defined in the MigrateList MTable translation object for the LU. This option is available only for Admin and Environment owner users.
Entity list: this is the default option. Populate the list of entities for the task. The populated entities should be separated with a comma.
Custom logic: select a Broadway flow in order to both get the entity list for the task and set the maximum number of entities for the task.
Note that when populating the Custom logic parameters in the task’s tabs, it includes an integration of Broadway editors into the TDM portal, which has been added by TDM 8.0:
Notes:
The maximum number of entities populated by the tester user is limited by their environment's permission set. This is the maximum number of entities in the task.
The maximum number of entities in the task is limited to the number of entities returned from the Custom Logic flow or by the Max Number of Entities task parameter. For example: if the maximum number of entities in the task is 50, but the custom logic returns 30 entities only, the task will process 30 entities.
Populate the Entity ID as populated in the source environment. For example, populate the Entities List with 1, 2 in order to extract Customers 1 and 2. The TDM execution process concatenates the required components to each Entity ID when building its LUI.
TDM Extract Task
An Extract task extracts the selected entities and/or Reference tables from the selected source environment. The data can be saved in Fabric (TDM warehouse) for a later use.
An Extract task contains the following tabs:
- General
- Additional Execution Parameters
- Requested Entities - opens if the Data Type contains entities.
- Task Scheduling
When checking the Set Task Variables setting, a new Task Variables tab opens.
When setting the Data Type by checking the Reference checkbox, a Reference tab opens.
Additional Execution Parameters Tab
The following execution parameters are set on Extract tasks:
Data Versioning
- Check the checkbox to set the task mode to Data Versioning.
- Leave the Data Versioning checkbox unchecked to create a regular mode task.
Data Type
Check the Entities checkbox to extract the selected entities and/or the Reference checkbox to extract the selected reference tables into the TDM warehouse (Fabric).
Click here for more information about the reference handling.
Set Sync Policy
This setting enables the user to change the default LUI sync mode (Sync ON) and to extract the LUI from the data source whenever the task is executed (Sync FORCE).
Note that this setting is only available when the Data Versioning checkbox is clear (regular task). If the Data Versioning checkbox is checked, each task execution extracts the data from the data source and creates a new LUI.
Retention Period
The retention period is set on the extracted entities and/or Reference tables. When this period ends, the task's entities and/or Reference tables are automatically deleted from Fabric and are no longer available.
Retention Period Values
Do not Delete - do not delete from Fabric.
Do not Retain - avoid saving the task's entities and/or Reference tables in Fabric (instead of saving and deleting). This option can be used, for example, to run an Extract task on a large subset of entities in order to populate the TDM parameter’s tables on each entity without saving the entities into Fabric.
Setting of a time measuring unit (minutes/hours/days…) and a value. For example, save the data in Fabric for 2 days. After 2 days the data is automatically deleted from Fabric.
Note that the retention period can be set in minutes, hours, days, weeks or years, depending on the maximum retention period set in the TDM DB. Both parameters - default retention period and maximum retention period - are set in the TDM DB.
Default Retention Period
- Data versioning checkbox is clear - do not Delete.
- Data versioning checkbox is checked - 5 days. A retention period must be set on a Data Versioning Extract task, i.e., the retention period must be set to a value greater than zero when the Data Versioning checkbox is checked.
Additional Execution Parameters
Set Task Variables
Check to open the Task Variables tab and set the variable value on a task level.
Mask Sensitive Data
TDM 8.1 added a checkbox to the Environment window to indicate if the environment has a sensitive data that must be stored in Fabric with masked values on PII fields. If the task's source value is defined as containing sensitive data, the TDM task window display the PII icon to indicate to the user that the source sensitive data is masked.
Post Execution Processes
Select all, partial or one post execution process of the selected BE.
Requested Entities Tab
This tab opens when the task's Data Type includes entities. This tab defines the subset of entities for the task:
The following selection methods are available on extract tasks:
Select a Predefined Entity List: run the SQL query or the Broadway flow defined in the MigrateList MTable translation object for the LU. This option is available only for Admin and Environment owner users.
Entity list: this is the default option. Populate the list of entities for the task. The populated entities should be separated with a comma.
Custom logic: select a Broadway flow in order to both get the entity list for the task and set the maximum number of entities for the task.
Note that when populating the Custom logic parameters in the task’s tabs, it includes an integration of Broadway editors into the TDM portal, which has been added by TDM 8.0:
Notes:
The maximum number of entities populated by the tester user is limited by their environment's permission set. This is the maximum number of entities in the task.
The maximum number of entities in the task is limited to the number of entities returned from the Custom Logic flow or by the Max Number of Entities task parameter. For example: if the maximum number of entities in the task is 50, but the custom logic returns 30 entities only, the task will process 30 entities.
Populate the Entity ID as populated in the source environment. For example, populate the Entities List with 1, 2 in order to extract Customers 1 and 2. The TDM execution process concatenates the required components to each Entity ID when building its LUI.
