TDM Load Task

A Load task contains the Load task action; it loads (provisions) the task's entities and/or Reference tables to the target environment. By selecting multiple task actions (types), a Load task can also delete entities from target environments before loading them, and/or reserve the target entities in the TDM DB. Note that when the Load task action checkbox is checked - the Reserve task action checkbox is checked by default as well.

A Load task contains the following tabs:

• General
• Additional Execution Parameters
• Requested Entities - opens if the Data Type contains entities.
• Task Scheduling

When checking the Set Task Variables checkbox, a new Task Variables tab opens.

When checking the Reference Data Type checkbox, a Reference tab opens.

Additional Execution Parameters Tab

Loading Entities from a Physical Environment

The following execution parameters are set on Load tasks:

Data Versioning

• Check the checkbox to set the task mode to Data Versioning.
• Leave the Data Versioning checkbox unchecked to create a regular mode task.

Data Type

Check the Entities and/or the Reference checkboxes to load the task's entities and/or Reference tables to the target environment (set in 'Load (provision) to Environment' task setting).

Note that the Entities checkbox must be checked in the task if it also includes an entity reservation (the Reserve task action checkbox is checked together with the Load task action checkbox).

Click here for more information about the reference handling.

Set Sync Policy

This setting enables the user to change the default LUI sync mode (Sync ON). The following sync policies are available:

• Sync New Data - this is the default option (Sync ON).
• Do Not Sync From Source Data - get the data from the TDM warehouse (Fabric) and do not sync it from the data source (Sync OFF).
• Refresh all Data from Source - extract the data source whenever the task is executed (Sync FORCE).

Notes:

• This setting is only available when the Data Versioning checkbox is clear (regular task). A Load Data Versioning task gets the data version from the TDM warehouse (Fabric) and reloads it to the target environment.
• The Do Not Sync From Source Data option is not displayed if the user selects a combination of Extract and Load task actions.

Click for more information on how overriding the Sync mode and the Task Operation mode impacts the task execution process.

Retention Period

The Retention Period setting is displayed if the Sync Policy is not Do Not Sync From Source Data. This setting enables to limit of the time for which the synced entities are kept in Fabric. The default Retention Period value is Do not Delete (do not limit the time for which the entities are kept is Fabric).

Reservation Period

The reservation period settings are displayed if the load task also reserves the loaded entities on the target environment (the Reserve task action checkbox is checked together with the Load task action checkbox).

Note that when the Reservation Period is set to zero, the entities are reserved for an unlimited period.

Only Admin and Environment owner users can reserve entities for an unlimited period. Users that are attached to the target environment as testers must set a reservation period. The maximum number of days of a reservation period is set in the tdm_general_parameters TDB DB table in the MAX_RESERVATION_DAYS_FOR_TESTER parameter.

The start date of the reservation period is the task's execution time. The reservation period can be set in minutes, hours, days or weeks.

Reservation Note

This setting has been added by TDM 7.5.2. The user can populate it with a free text. The reservation note is displayed in the Reserved Entities window (Notes fields) and can be used for filtering the reserved entities display. The reservation note can be edited using the Reserved Entities window.

Additional Execution Parameters

Set Task Variables

Check to open the Task Variables tab and set the variable value on a task level.

Replace Sequence

When checked, the task execution process replaces the IDs of all selected entities before loading them into the target. This option is required to avoid key duplications if the testing environment is not empty and contains entities.

Notes:

• The Replace Sequence is not displayed on Data Versioning task since the data version is loaded 'as is'.
• The Replace Sequence setting is disabled if the tester user does not have permissions to replace the sequence in the Write TDM Environment permission setting on the task's target environment.
• The Replace Sequence must be implemented in the Fabric implementation.

Mask Sensitive Data

TDM 8.1 added a checkbox to the Environment window, to indicate whether the source data needs to be masked before it is saved into Fabric. If the task's source value is defined as containing sensitive data, the TDM task window display the PII icon to indicate to the user that the source sensitive data is masked.

Post Execution Processes

Select all, partial, or one post execution process of the selected BE.

Loading Previously Generated Entities

If the Extract from Environment is Synthetic, the Load task gets, from the TDM warehouse (Fabric), the synthetic entities previously generated by Generate tasks.

The following execution parameters are set on Load tasks when the Extract from Environment is Synthetic:

Main Differences Between Loading Synthetic Entities and Loading from a Physical Environment

The following settings are not displayed in the task when loading synthetic entities:

• The Data Versioning is not displayed. There is no Data Versioning on generated synthetic entities.

• The Data Type is Entities as the task loads generated synthetic entities. The Reference value is not displayed.

• The Set Sync Policy and Retention Period settings are not displayed as the task gets the synthetic entities from the TDM warehouse (Fabric).

Requested Entities Tab

This tab opens when the task's Data Type includes entities. This tab defines the subset of entities for the task:

The following selection methods are available on load tasks:

Select a Predefined Entity List

Run the SQL query or the Broadway flow defined in the MigrateList MTable translation object for the LU. This option is available only for Admin and Environment owner users.

Entity list

This is the default option. Populate the list of entities for the task with entities separated with a comma between them. Note that a warning is thrown if the entity list includes entities that are reserved for another user.

Custom Logic

Select a Broadway flow in order to both get the entity list for the task and set the maximum number of entities for the task.

Notes:

• TDM 8.0 added integration of Broadway editors into the TDM portal when populating the Custom logic parameters in the task’s tabs.
• The Filter out Reserved Entities checkbox indicates if entities that are reserved for other users must be filtered out from the task's entity list. If checked, these entities are filtered out from the task's entity list.

Parameters

Select entities from a predefined list of parameters. You should set the parameters and their values and the maximum number of entities for the task.

Note that entities reserved for another user in the environment are excluded from the task. You can select one or several parameters. Additionally, you can add the same parameter multiple times with different values:

The list of parameters should be defined for each LU in the task BE in the Fabric project.

Use Parameters with Random Selection Checkbox

The Parameters selection has 2 modes:

• When checked (default), TDM randomly selects - from the list of all entities - entities that match the selected parameters. Each task execution gets different lists of entities that match the selected parameters. The Selection Method displayed in the Tasks List window is Parameters - selection based on parameters with random selection.

  Example:

  • Creating a task to load 5 customers using selected parameters. There are 800 customers that match the selected parameters. The task execution gets a random list of 5 customers from the list of 800 customers that match the selected parameters.

• When unchecked, it gets the first entities that match the selected parameters. Each task execution gets the same list of entities that match the selected parameters. The Selection Method displayed in the Tasks List window is Parameters - selection based only on Parameters.

  Example:

  • Create a task to load 5 customers with selected parameters. There are 800 customers that match the selected parameters. The task execution gets the first 5 customers that match the selected parameters.

Filter out Reserved Entities Checkbox

The Filter out Reserved Entities checkbox has been added in TDM 7.6 and it indicates if entities that are reserved for other users must be filtered out from the task's entity list. If checked, these entities are filtered out from the task's entity list and from the Entities Matched setting.

For example:

• There are 500 customers with VIP Status 'Gold' and 10 of them are reserved for other users on the task's environments.
• If the Filter out Reserved Entities checkbox is checked, the Entities Matched result is 490.
• If the Filter out Reserved Entities checkbox is cleared, the Entities Matched result is 500.

How Do I Add a Condition?

To add a parameter:

• Click Add Condition.

• Select the parameter and the operator from the drop-down lists and populate their values. Note that from TDM 8.1 onwards it is possible to select the IN or NOT IN operators on combo parameters and on text parameters.

• Set the value on the parameter. You can set multiple values on a parameter if the selected operator is either IN or NOT IN:

  • Combo parameters - click on the value field to select a required value from the drop-down list; this can be done multiple times as more than one value can be selected in this field.

  • Free text parameters - populate the values separated by a comma. For example: NY, CA.

  • View the below example:

    

• Add the AND/OR operator to connect the parameter to the previous parameters or group. The TDM Portal displays the SQL query, which is built based on the selected parameters.

How Do I Populate a Parameter's Value?

There are several types of parameters:

• Combo - parameters with a limited number of values. The Task window displays a drop-down list of the parameters' values; select one of them.

  Click for more information about setting a parameter as a combo parameter.

• Number - the TDM Portal displays the minimum and maximum values of this parameter. If the populated values exceed a parameter's range, an error message is displayed.

• Date - populate the value using the following format: YYYY-MM-DD.

  • Notes:

  • The date format is defined in DATETIME_FORMAT parameter of the config.ini Fabric file. The default format is 'yyyy-MM-dd HH:mm:ss.SSS'.

  • The parameter's search may be based on either a Date (removing the time from the Date format) or a Datetime.

    Examples:

    • 2021-12-28
    • 2022-04-03 10:00:00

  • The date value is set without quotation marks.

• Text - populate the value using a free text.

How Do I Add a Group of Parameters?

Click Add Group.

Note that nested groups of parameters can be added, for example, in order to define an inner group in an outer group.

How Do I Remove a Parameter or a Parameters Group?

1. Click next to the parameter in order to delete it.
2. Click Remove Group to remove a parameter's group.

Getting the Number of Matching Entities

Click Refresh next to the Entities Matched in order to calculate the number of entities that match the selected parameters. The Parameters selection supports the parent-child hierarchy relationship between the LUs of the selected BE. It can cross-check the matching entities of a selected combination of parameters and take into consideration parameters from different LUs in the same BE hierarchy. For example, selecting customers based on their number of open cases, subscriber_type, and vip_status (see the above screenshot).

• Click to display the number of matching entities according to the parameters’ conditions.

Click for more information about the TDM parameters tables and View created by TDM in the TDM DB to display a hierarchical view of TDM parameters.

Entity Clone

Create X number of clones of the selected entity in the target environment. When selecting this method, you can either populate a real entity ID to be cloned or parameters. If you select the parameters option, then the first selected entity that is based on the task's parameters is cloned (replicated) in the target environment.

For example, if the Number of Clones setting is populated with 10 and the Entity ID is set to 645, the task creates 10 clones (replicas) of Entity ID 645 in the target environment.

The task replaces the sequences of each replica in order to avoid duplicated sequences in the target environment.

Testers may select this option only if they are permitted to clone an entity in the task's target environment.

Random Selection

Get a random list of entities from the LU PARAMS table created in the TDM DB for the root LU of the task's BE. Entities that are reserved for other users are excluded from the selected entities.

Testers can select this option only if they are permitted to do so in the task's target environment.

Filter out Reserved Entities Checkbox

The Filter out Reserved Entities checkbox indicates if entities that are reserved for other users must be filtered out from the task's entity list. If checked, these entities are filtered out from the task's entity list.