Fabric TDM Library

The TDM Library has all the utilities required to implement a TDM project and to run TDM execution processes. It holds the following:

Note that the TDM Library must be imported to the Fabric project created for TDM. Download the TDM library from this link .

TDM Library - Shared Objects

TDM Web Services

Import and deploy all TDM Web Services (APIs) to Fabric project. These Web Services are invoked by the TDM GUI application and comprise the back-end layer of the TDM GUI application.

Since the TDM categories contain the product's Web Services, it is recommended to add the project's Web Services into separate categories to simplify upgrading the TDM version.

Generic TDM Interfaces

Import and deploy the following interfaces into the project's Shared Objects:

  • DB_CASSANDRA: This is the connection to the Cassandra DB. This interface is used by TDM utilities. Edit the IP address according to the environment.

  • TDM: This is the connection to the TDM PosgreSQL DB. Edit the IP address according to the environment.

  • FabricRedis: This is the Redis interface that connects to the environment's Redis storage. The Redis interface can be used for the sequence implementation. Edit the IP address and populate it with the IP address of the TDM server.

  • TDM_APIDOC_JSON: This is a local file system interface used to generate the JSON file of the TDM APIDOC if the APIDOC needs to be updated to include project custom APIs. Click here for more information about updating the TDM APIDOC.

    Note that it is important to set the TDM_APIDOC_JSON interface as disabled in the Environments to avoid an error when running the test connection on the task's environment (Fabric server has a different IP address than the local windows machine and cannot connect window's directory).

Shared Globals

Import the list of shared global variables required to execute TDM in your project.

Shared Functions

TDM shared functions are saved in the TDM Logic file.

Import the TDM shared functions to your project. Note that since the TDM category contains the product's functions, it is recommended to add the project's shared functions to a separate category (Logic file) to simplify upgrading the TDM version.

Shared Translations

Item 

Description 

Instructions 

trnMigrateList

Define the query and interface name to run the extract task on all entities of each LU. One record per LU.

Populate this translation for each Logical Unit. A separate record must be created for each Logical Unit in the Fabric project apart from TDM, TDM_LIBRARY and the dummy LU of the post-execution processes.  

If there is a need to define a query per source environment, populate the source environment name and create a separate record for each Logical Unit and source_env_name combination. Otherwise, leave the source environment empty.

Example 1:

  • LU_NAME= ORDER
  • SOURCE_ENV_NAME = ENV1
  • INTERFACE_NAME = TDM
  • IG_SQL = Select lu_type2_eid from tdm_lu_type_relation_eid where lu_type_2 = ‘ORDER’ and source_env = 'ENV1';

Example 2:

  • LU_NAME= CUSTOMER
  • SOURCE_ENV_NAME is empty
  • INTERFACE_NAME = CRM_DB
  • IG_SQL = Select customer_id from customer limit 1000;

trnMigrateListQueryFormats

Supports special syntax for extract tasks when creating the LU instance query based on the trnMigrateList translation. Each LUI consists of a concatenation of source environment, IID, version name and version datetime.

Click to read more about LUI structure for TDM implementation.

This translation is required for DBs that do not support the standard ‘||’ syntax for concatenated strings. For example, sqlServer.

Populate two records for each DB, one record with version_ind ‘true’ and another with version_ind ‘false’. 

Example 1:

  • interface_type = sqlserver
  • version_ind = true
  • query_format = CONCAT(<source_env_name>,'_',<entity_id>,'_',<task_name>,'_',<timestamp>)

Example 2:

  • interface_type = sqlserver
  • version_ind = false
  • query_format = CONCAT(<source_env_name>,'_',<entity_id>)

trnRefList

Define the list of available reference tables for extract tasks.

Click to read more about Reference implementation.

Populate this translation for each reference table. A separate record must be created for each reference table. Set the LU name on each record.

trnPostProcessList

Define the list of post-processes to run at the end of the task's execution. For example, a process that sends a mail to notify the user when the task's execution ends.

Each process is implemented as a Broadway flow.

Populate the list of Broadway flows and the LU of the Broadway flow. The LU can be empty if the post processes are defined under Shared Objects whereby the TDM task execution process sets the LU Name to TDM when running Batch commands to execute post execution processes.

trnTDMCleanUp

Define the list of the TDM DB tables to be cleaned by the TDM clean-up process. The translation defines the Delete statement on each table and a clean-up indicator indicating whether the table should be cleaned by the TDM clean-up process.

Clear the cleanup_ind to remove a table from the clean-up process. TDM tables can be added and Delete statements can be edited.

Broadway Generic Flows and Templates

The Fabric TDM library includes a set of built-in generic Broadway flows defined for easy adaptation of a generic TDM implementation for a specific data model.

Click for more information about Generic TDM Broadway Flows.

TDM LU

The TDM Logical Unit must be deployed to the Fabric project. It has the following tasks:

  • Saves information about executed TDM tasks. The TDM GUI provides execution statistics and reports based on the data in the TDM LU. The LUI of the TDM LU is a unique task_Execution_id generated by the TDM GUI for each executed task.
  • Task execution jobs are defined and run under the TDM LU.
  • The TDM cleanup job that cleans the TDM DB is defined under the TDM LU.

Note that you need to stop the TDM jobs on the local Fabric before deploying the TDM LU to a remote Fabric to avoid a parallel execution of TDM jobs on the same TDM DB.

TDM_LIBRARY LU

The TDM_LIBRARY LU holds utilities that must be copied to the project's LUs, as follows:

Globals

LU level Globals

  • Populate the ROOT_TABLE_NAME Global using the main source table or tables. You can populate several tables separated by a comma.

    Examples:

    • CUSTOMER.
    • CUSTOMER, ACCOUNT
  • Populate the ROOT_COLUMN_NAME Global using the entity ID's column. These Globals are needed to set the IS_INSTANCE_ID column correctly in TDM_SEQ_MAPPING TDM DB table. Note that the number and order of root column names must be aligned with the number and order of the tables, populated in ROOT_TABLE_NAME.

    Examples:

ROOT_TABLE_NAME ROOT_COLUMN_NAME
CUSTOMER CUSTOMER_ID
CUSTOMER, ACCOUNT CUSTOMER_ID, CUSTOMER_ID
CUSTOMER, ACCOUNT_DATA CUSTOMER_ID, ACC_CUST_ID

LU Tables

  • FABRIC_TDM_ROOT, the Root table of each LU. This table contains the following columns:

    • K2_TDM_EID, populated by the LU instance ID.
    • IID, populated by the entity ID without the concatenation of the source environment, version name and version datetime.
    • SOURCE_ENV, populated by the source environment name of the TDM task.
    • TASK_NAME, version name, populated by a DataFlux task by the task name.
    • TIMESTAMP, version datetime, populated by a DataFlux task.

    Example:

K2_TDM_EID

IID

SOURCE_ENV

TASK_NAME

TIMESTAMP

PROD_1

1

PROD

 

 

PROD_1_copyCust_20201105090000

1

PROD

copyCust

20201105090000

  • LU_PARAMS, parameters table. Must be added to each LU schema even when it is not required for defining parameters on the LU, whereby the LU_PARAM table only holds the ENTITY_ID and SOURCE_ENVIRONMENT fields.

    Click for more information about TDM parameters handling.

  • TDM_LU_TYPE_RELATION_EID and TDM_LU_TYPE_REL_TAR_EID, TDM relationship tables that map the parent to child IDs. Note that these tables are also created in the TDM DB.

    Click for more information about TDM Hierarchy implementation.

  • INSTANCE_TABLE_COUNT, this table holds the number of records populated on each LU table and is used to populate the [TDM execution Report]().

LU Level Translations

Item

Description 

Instructions 

trnChildLink

Translation for mapping parent and child IDs. 

Click for more information about TDM business entities and how to support a hierarchy when implementing the LUs.

This translation must be added and populated on each parent LU and is used to populate TDM relationship tables. The child_lu field must be populated by the name of the child LU.

Both SQLs populated in child_lu_eid_sql and child_lu_tar_eid_Sql fields must run on the parent LU and get the source and target child IDs for each parent ID.

Example:
Customer LU is the parent of the Orders LU.
trnChildLink of the Customer LU must be populated as follows:

  • Child_lu = Orders
  • Child_lu_eid_sql = select order_id from subscriber
  • Child_lu_tar_eid_sql = select order_id from tar_subscriber

Note that both tables, subscriber and tar_subscriber, must be defined in Customer LU schema.

trnLuParams

Translation for the population of the LU_PARAMS table. 

The COLUMN_NAME is populated by the name of the parameter and the SQL is populated by the SQL query that gets the values for the defined parameter.

Click for more information about handling parameters.

trnParsList

Translation for splitting large files. 

Copy the translation to the LU if needed and populate it with the list of large parsers. 

Split Large Parsers

parExecuteJob, the main parser that runs other parsers and splits them to run in parallel. The trnParsList translation object holds the large parser names to be split. Copy this parser to the LU if needed.

Previous

Fabric TDM Library

The TDM Library has all the utilities required to implement a TDM project and to run TDM execution processes. It holds the following:

Note that the TDM Library must be imported to the Fabric project created for TDM. Download the TDM library from this link .

TDM Library - Shared Objects

TDM Web Services

Import and deploy all TDM Web Services (APIs) to Fabric project. These Web Services are invoked by the TDM GUI application and comprise the back-end layer of the TDM GUI application.

Since the TDM categories contain the product's Web Services, it is recommended to add the project's Web Services into separate categories to simplify upgrading the TDM version.

Generic TDM Interfaces

Import and deploy the following interfaces into the project's Shared Objects:

  • DB_CASSANDRA: This is the connection to the Cassandra DB. This interface is used by TDM utilities. Edit the IP address according to the environment.

  • TDM: This is the connection to the TDM PosgreSQL DB. Edit the IP address according to the environment.

  • FabricRedis: This is the Redis interface that connects to the environment's Redis storage. The Redis interface can be used for the sequence implementation. Edit the IP address and populate it with the IP address of the TDM server.

  • TDM_APIDOC_JSON: This is a local file system interface used to generate the JSON file of the TDM APIDOC if the APIDOC needs to be updated to include project custom APIs. Click here for more information about updating the TDM APIDOC.

    Note that it is important to set the TDM_APIDOC_JSON interface as disabled in the Environments to avoid an error when running the test connection on the task's environment (Fabric server has a different IP address than the local windows machine and cannot connect window's directory).

Shared Globals

Import the list of shared global variables required to execute TDM in your project.

Shared Functions

TDM shared functions are saved in the TDM Logic file.

Import the TDM shared functions to your project. Note that since the TDM category contains the product's functions, it is recommended to add the project's shared functions to a separate category (Logic file) to simplify upgrading the TDM version.

Shared Translations

Item 

Description 

Instructions 

trnMigrateList

Define the query and interface name to run the extract task on all entities of each LU. One record per LU.

Populate this translation for each Logical Unit. A separate record must be created for each Logical Unit in the Fabric project apart from TDM, TDM_LIBRARY and the dummy LU of the post-execution processes.  

If there is a need to define a query per source environment, populate the source environment name and create a separate record for each Logical Unit and source_env_name combination. Otherwise, leave the source environment empty.

Example 1:

  • LU_NAME= ORDER
  • SOURCE_ENV_NAME = ENV1
  • INTERFACE_NAME = TDM
  • IG_SQL = Select lu_type2_eid from tdm_lu_type_relation_eid where lu_type_2 = ‘ORDER’ and source_env = 'ENV1';

Example 2:

  • LU_NAME= CUSTOMER
  • SOURCE_ENV_NAME is empty
  • INTERFACE_NAME = CRM_DB
  • IG_SQL = Select customer_id from customer limit 1000;

trnMigrateListQueryFormats

Supports special syntax for extract tasks when creating the LU instance query based on the trnMigrateList translation. Each LUI consists of a concatenation of source environment, IID, version name and version datetime.

Click to read more about LUI structure for TDM implementation.

This translation is required for DBs that do not support the standard ‘||’ syntax for concatenated strings. For example, sqlServer.

Populate two records for each DB, one record with version_ind ‘true’ and another with version_ind ‘false’. 

Example 1:

  • interface_type = sqlserver
  • version_ind = true
  • query_format = CONCAT(<source_env_name>,'_',<entity_id>,'_',<task_name>,'_',<timestamp>)

Example 2:

  • interface_type = sqlserver
  • version_ind = false
  • query_format = CONCAT(<source_env_name>,'_',<entity_id>)

trnRefList

Define the list of available reference tables for extract tasks.

Click to read more about Reference implementation.

Populate this translation for each reference table. A separate record must be created for each reference table. Set the LU name on each record.

trnPostProcessList

Define the list of post-processes to run at the end of the task's execution. For example, a process that sends a mail to notify the user when the task's execution ends.

Each process is implemented as a Broadway flow.

Populate the list of Broadway flows and the LU of the Broadway flow. The LU can be empty if the post processes are defined under Shared Objects whereby the TDM task execution process sets the LU Name to TDM when running Batch commands to execute post execution processes.

trnTDMCleanUp

Define the list of the TDM DB tables to be cleaned by the TDM clean-up process. The translation defines the Delete statement on each table and a clean-up indicator indicating whether the table should be cleaned by the TDM clean-up process.

Clear the cleanup_ind to remove a table from the clean-up process. TDM tables can be added and Delete statements can be edited.

Broadway Generic Flows and Templates

The Fabric TDM library includes a set of built-in generic Broadway flows defined for easy adaptation of a generic TDM implementation for a specific data model.

Click for more information about Generic TDM Broadway Flows.

TDM LU

The TDM Logical Unit must be deployed to the Fabric project. It has the following tasks:

  • Saves information about executed TDM tasks. The TDM GUI provides execution statistics and reports based on the data in the TDM LU. The LUI of the TDM LU is a unique task_Execution_id generated by the TDM GUI for each executed task.
  • Task execution jobs are defined and run under the TDM LU.
  • The TDM cleanup job that cleans the TDM DB is defined under the TDM LU.

Note that you need to stop the TDM jobs on the local Fabric before deploying the TDM LU to a remote Fabric to avoid a parallel execution of TDM jobs on the same TDM DB.

TDM_LIBRARY LU

The TDM_LIBRARY LU holds utilities that must be copied to the project's LUs, as follows:

Globals

LU level Globals

  • Populate the ROOT_TABLE_NAME Global using the main source table or tables. You can populate several tables separated by a comma.

    Examples:

    • CUSTOMER.
    • CUSTOMER, ACCOUNT
  • Populate the ROOT_COLUMN_NAME Global using the entity ID's column. These Globals are needed to set the IS_INSTANCE_ID column correctly in TDM_SEQ_MAPPING TDM DB table. Note that the number and order of root column names must be aligned with the number and order of the tables, populated in ROOT_TABLE_NAME.

    Examples:

ROOT_TABLE_NAME ROOT_COLUMN_NAME
CUSTOMER CUSTOMER_ID
CUSTOMER, ACCOUNT CUSTOMER_ID, CUSTOMER_ID
CUSTOMER, ACCOUNT_DATA CUSTOMER_ID, ACC_CUST_ID

LU Tables

  • FABRIC_TDM_ROOT, the Root table of each LU. This table contains the following columns:

    • K2_TDM_EID, populated by the LU instance ID.
    • IID, populated by the entity ID without the concatenation of the source environment, version name and version datetime.
    • SOURCE_ENV, populated by the source environment name of the TDM task.
    • TASK_NAME, version name, populated by a DataFlux task by the task name.
    • TIMESTAMP, version datetime, populated by a DataFlux task.

    Example:

K2_TDM_EID

IID

SOURCE_ENV

TASK_NAME

TIMESTAMP

PROD_1

1

PROD

 

 

PROD_1_copyCust_20201105090000

1

PROD

copyCust

20201105090000

  • LU_PARAMS, parameters table. Must be added to each LU schema even when it is not required for defining parameters on the LU, whereby the LU_PARAM table only holds the ENTITY_ID and SOURCE_ENVIRONMENT fields.

    Click for more information about TDM parameters handling.

  • TDM_LU_TYPE_RELATION_EID and TDM_LU_TYPE_REL_TAR_EID, TDM relationship tables that map the parent to child IDs. Note that these tables are also created in the TDM DB.

    Click for more information about TDM Hierarchy implementation.

  • INSTANCE_TABLE_COUNT, this table holds the number of records populated on each LU table and is used to populate the [TDM execution Report]().

LU Level Translations

Item

Description 

Instructions 

trnChildLink

Translation for mapping parent and child IDs. 

Click for more information about TDM business entities and how to support a hierarchy when implementing the LUs.

This translation must be added and populated on each parent LU and is used to populate TDM relationship tables. The child_lu field must be populated by the name of the child LU.

Both SQLs populated in child_lu_eid_sql and child_lu_tar_eid_Sql fields must run on the parent LU and get the source and target child IDs for each parent ID.

Example:
Customer LU is the parent of the Orders LU.
trnChildLink of the Customer LU must be populated as follows:

  • Child_lu = Orders
  • Child_lu_eid_sql = select order_id from subscriber
  • Child_lu_tar_eid_sql = select order_id from tar_subscriber

Note that both tables, subscriber and tar_subscriber, must be defined in Customer LU schema.

trnLuParams

Translation for the population of the LU_PARAMS table. 

The COLUMN_NAME is populated by the name of the parameter and the SQL is populated by the SQL query that gets the values for the defined parameter.

Click for more information about handling parameters.

trnParsList

Translation for splitting large files. 

Copy the translation to the LU if needed and populate it with the list of large parsers. 

Split Large Parsers

parExecuteJob, the main parser that runs other parsers and splits them to run in parallel. The trnParsList translation object holds the large parser names to be split. Copy this parser to the LU if needed.

Previous