TDM Reference Processes

The list of Reference tables available for TDM tasks is populated in the RefList MTable. TDM 7.6 onwards stores the Reference tables in a dedicated LU: TDM_Reference. Each Reference table is stored as a separate LUI.

Previous TDM versions stored the extracted reference tables in the Cassandra DB: The TDM Extract tasks stored the selected reference data in the Cassandra DB and the TDM Load tasks selected the reference tables from Cassandra and loaded them into the target environment.

Task Execution Job

The main task execution process runs TDM Broadway flows in order to get the reference table into Fabric (TDM_Reference LU) and loads it into the target environment for a load task.

The job updates the status of the processed reference table in the task_ref_exe_stats TDM DB table:

• If the process succeeds, it sets the status to completed.
• If the process fails, it sets the status to failed and populates the error_msg field with the error message.

TDM_Reference LU

The LUI contains the following:

[LU name]|[source environment name]|[version id]|[table name]

By default, the version id is populated by ALL. When running a TDM Extract task in Data Versioning mode, this setting is populated by the task_execution_id of the task execution.

Examples:

• Customer|SRC|ALL|DEVICESTABLE2017
• Customer|SRC|155|DEVICESTABLE2017

Note that the Sync method LU property is set by default to None, i.e., each LUI (reference table) is synced only once. You need to edit this property in order to enable a recurring sync of the reference table from the source environment.

TDM_Reference LU - reference_table LU Table

The reference_table LU table keeps the reference data. The LU table has the following fields:

• TABLE_NAME - populated with the reference table name, e.g., CUSTOMER_TYPE.
• TDM_TASK_EXECUTION_ID - populated, by default, with ALL. When running a TDM Extract task in Data Versioning mode, this column is populated by the task_execution_id of the task execution.
• SOURCE_ENV_NAME - populated by the source environment.
• TABLE_DATA - the table data is extracted into a JSON structure. The JSON is compressed and saved in a BLOB field.
• RECORD_COUNT - number of records of the reference table.
• TABLE_FIELDS - list of the reference table's fields.

Example:

• CUSTOMER_TYPE Reference table. This table has 3 fields: CUSTOMER_TYPE, CUSTOMER_SUB_TYPE and DESCRIPTION.

• CUSTOMER_TYPE is populated in ENV1 as follows:

• CUSTOMER_TYPE is populated in ENV2 as follows:

• Creating and executing an Extract task with a regular mode (Data Versioning checkbox is unchecked) to extract CUSTOMER_TYPE Reference table from ENV1. A new LUI is created in TDM_Reference LU: Customer|ENV1|ALL|CUSTOMER_TYPE.

• The LU table is populated as follows:

• Creating an Extract Data Versioning task for CUSTOMER_TYPE Reference table on ENV1. A new LUI is created:

  The LUI is Customer|ENV1|1234|CUSTOMER_TYPE.

• The LU table is populated as follows:

• Creating an Extract Task with a regular mode for CUSTOMER_TYPE on ENV2. A new LUI is created:

  The LUI is Customer|ENV2|ALL|CUSTOMER_TYPE.

• The LU table is populated as follows: