The TDM Library has all the utilities required for implementing a TDM project and for running TDM execution processes. It holds the following:
The TDM Library must be imported to the Fabric project created for TDM.
Import and deploy all TDM Web Services (APIs) to the Fabric project. These Web Services are invoked by the TDM Portal application and they comprise the back-end layer of the TDM Portal application.
As the TDM categories contain the product's Web Services, it is recommended to add the project's Web Services into separate categories in order to simplify the TDM version upgrading.
Import and deploy the following interfaces into the project's Shared Objects:
DB_CASSANDRA - this is the connection to the Cassandra DB. This interface is used by TDM utilities. Edit the IP address according to the environment.
CASSANDRA_LD - a Cassandra Loader interface. This interface is used by the Reference upgrade script (upgrade to TDM 7.5.1).
POSTGRESQL_ADMIN - this is the admin connection to the TDM PosgreSQL DB. This interface is used by the TDMDB flow in the TDM LU to create the TDM DB in the PostgreSQL DB.
TDM - this is the connection to the TDM PosgreSQL DB. Edit the IP address according to the environment.
Note that if you work on a PostgreSQL with an SSL connection, you must edit the custom connection string of the POSTGRESQL_ADMIN and the TDM interfaces as follows:
FabricRedis - this is the Redis interface that connects to the environment's Redis storage. The Redis interface can be used for the sequence implementation. Edit the IP address and populate it with the IP address of the TDM server. Note that from Fabric 7.1 onwards, the sequence Actors can get the next sequence value from a newly created DB sequence in the sequence interface (the sequence interface can be the TDM DB). The sequence is created by the Actor, if it doesn't yet exist. It is therefore recommended to get the next value from a DB sequence instead of using the Redis.
For more information, view the Sequence Implementation Guide.
TDM_APIDOC_JSON - this is a local file system interface, used for generating the JSON file of the TDM APIDOC if the APIDOC needs to be updated to include project custom APIs. Click here for more information about updating the TDM APIDOC.
It is important to set the TDM_APIDOC_JSON interface as disabled in the Environments in order to prevent errors when running the test connection on the task's environment (the Fabric server has a different IP address than the local Windows machine and cannot connect to the local machine's directory).
Import the list of shared global variables required for executing TDM in your project.
TDM shared functions are saved in the TDM Logic file.
Import the TDM shared functions to your project. Note that since the TDM category contains the product's functions, it is recommended to add the project's shared functions to a separate category (Logic file) in order to simplify the TDM version upgrading.
TDM 8.1 replaces the previous TDM translation with MTables to support a development of the TDM on both Fabric Studios: Desktop-Studio and Web-Studio.
The following MTables have been added to the References in the TDM library. Note that you must deploy the Reference to Fabric after updating the MTables:
Item |
Description |
Instructions |
MigrateList |
Define the query and interface name, or the Broadway flow to generate the entity list when running the extract task on all entities of each LU; one record per LU. |
Populate this table for each Logical Unit. A separate record must be created for each Logical Unit in the Fabric project apart from TDM, TDM_LIBRARY and the dummy LU of the post-execution processes. If there is a need to define a query per source environment, populate the source environment name and create a separate record for each Logical Unit and source_env_name combination. Otherwise, leave the source environment empty. Click here for more information on how to implement a Broadway flow to get the entities (populated in external_table_flow field of MigrateList table). Example 1:
Example 2:
Example 3:
|
MigrateListQueryFormats |
Supports special syntax for extract tasks when creating the LU instance query based on the MigrateList table. Each LUI consists of a concatenation of the source environment, IID, version name and version datetime. Click to read more about LUI structure for TDM implementation. This table is required for databases that do not support the standard ‘||’ syntax for concatenated strings, e.g., sqlServer. |
Populate 2 records for each database, one record with version_ind ‘true’ and another with version_ind ‘false’. Example 1:
Example 2:
|
RefList |
Define the list of available reference tables for TDM tasks. Click to read more about Reference implementation. |
Populate this table for each reference table. A separate record must be created for each reference table. Set the LU name on each record. |
PostProcessList |
Define the list of post-processes to run at the end of the task's execution. For example, a process that sends a mail to notify the user when the task's execution ends. Each process is implemented as a Broadway flow. |
Populate the list of Broadway flows and the LU of the Broadway flow. The LU can be empty if the post processes are defined under Shared Objects, whereby the TDM task execution process sets the LU Name to TDM when running Batch commands to carry out post execution processes. Redeploy the LUs populated in this table, the TDM LU, and the Web-Services. |
ChildLink |
Mapping parent and child IDs. Click for more information about TDM business entities and how to support a hierarchy when implementing the LUs. |
A record must be added to this table for each parent-child relationship. The parent_lu field must be populated with the name of the parent LU and child_lu field must be populated with the name of the child LU. Both SQLs populated in child_lu_eid_sql and child_lu_tar_eid_Sql fields must run on the parent LU and get the source and target child IDs for each parent ID. Example:
The parameters: tables, subscriber and tar_subscriber, must all be defined in the CRM LU schema. |
LuParams |
Translation for the population of the LU_PARAMS table. |
The COLUMN_NAME is populated by the name of the parameter and the SQL is populated by the SQL query that gets the values for the defined parameter. Click for more information about handling parameters. |
The Fabric TDM library includes a set of built-in generic Broadway flows, defined for easy adaptation of a generic TDM implementation for a specific data model.
Click for more information about Generic TDM Broadway Flows.
The TDM Logical Unit must be deployed to the Fabric project. It has the following tasks:
TDM enables setting TTL (Time To Live) on the TDM LUIs. The default TTL period is 10 days. The TDM LUI's TTL depends on the following shared Globals (imported from the TDM Library):
The deploy.flow process runs the following activities upon the TDM LU deployment:
Deploy the TDM LU to Fabric. From TDM 7.6 onwards, the deployment of the TDM LU deploys the TDM Portal as well into the TDM web applications.
Notes:
The TDM_LIBRARY LU holds utilities that must be copied to the project's LUs. These utilities are described below.
It is recommended to duplicate the TDM_Library LU and use it as a template when creating a new LU in a TDM project.
Populate the ROOT_TABLE_NAME Global using the main source table/s. You can populate several tables, separated by a comma.
Examples:
Populate the ROOT_COLUMN_NAME Global using the entity ID's column. These Globals are needed for setting the IS_INSTANCE_ID column correctly in TDM_SEQ_MAPPING TDM DB table. Note that the number and order of root column names must be aligned with the number and order of the tables that are populated in ROOT_TABLE_NAME.
Examples:
ROOT_TABLE_NAME | ROOT_COLUMN_NAME |
CUSTOMER | CUSTOMER_ID |
CUSTOMER, ACCOUNT | CUSTOMER_ID, CUSTOMER_ID |
CUSTOMER, ACCOUNT_DATA | CUSTOMER_ID, ACC_CUST_ID |
FABRIC_TDM_ROOT - the Root table of each LU. This table contains the following columns:
Example:
K2_TDM_EID |
IID |
SOURCE_ENV |
TASK_NAME |
TIMESTAMP |
PROD_1 |
1 |
PROD |
|
|
PROD_1_copyCust_20201105090000 |
1 |
PROD |
copyCust |
20201105090000 |
LU_PARAMS - parameters table. Must be added to each LU schema even when it is not required for defining parameters in the LU. The LU_PARAM table holds only the ENTITY_ID and SOURCE_ENVIRONMENT fields.
Click for more information about TDM parameters handling.
TDM_LU_TYPE_RELATION_EID and TDM_LU_TYPE_REL_TAR_EID - TDM relationship tables that map the parent to child IDs. Note that these tables are also created in the TDM DB.
Click for more information about TDM Hierarchy implementation.
TDM 7.6 onwards stores the extracted Reference tables in a new LU - TDM_Reference - instead of storing them in Cassandra. Each Reference table is stored as a separate LUI. For more information see Reference Implementation.
The TDM Library has all the utilities required for implementing a TDM project and for running TDM execution processes. It holds the following:
The TDM Library must be imported to the Fabric project created for TDM.
Import and deploy all TDM Web Services (APIs) to the Fabric project. These Web Services are invoked by the TDM Portal application and they comprise the back-end layer of the TDM Portal application.
As the TDM categories contain the product's Web Services, it is recommended to add the project's Web Services into separate categories in order to simplify the TDM version upgrading.
Import and deploy the following interfaces into the project's Shared Objects:
DB_CASSANDRA - this is the connection to the Cassandra DB. This interface is used by TDM utilities. Edit the IP address according to the environment.
CASSANDRA_LD - a Cassandra Loader interface. This interface is used by the Reference upgrade script (upgrade to TDM 7.5.1).
POSTGRESQL_ADMIN - this is the admin connection to the TDM PosgreSQL DB. This interface is used by the TDMDB flow in the TDM LU to create the TDM DB in the PostgreSQL DB.
TDM - this is the connection to the TDM PosgreSQL DB. Edit the IP address according to the environment.
Note that if you work on a PostgreSQL with an SSL connection, you must edit the custom connection string of the POSTGRESQL_ADMIN and the TDM interfaces as follows:
FabricRedis - this is the Redis interface that connects to the environment's Redis storage. The Redis interface can be used for the sequence implementation. Edit the IP address and populate it with the IP address of the TDM server. Note that from Fabric 7.1 onwards, the sequence Actors can get the next sequence value from a newly created DB sequence in the sequence interface (the sequence interface can be the TDM DB). The sequence is created by the Actor, if it doesn't yet exist. It is therefore recommended to get the next value from a DB sequence instead of using the Redis.
For more information, view the Sequence Implementation Guide.
TDM_APIDOC_JSON - this is a local file system interface, used for generating the JSON file of the TDM APIDOC if the APIDOC needs to be updated to include project custom APIs. Click here for more information about updating the TDM APIDOC.
It is important to set the TDM_APIDOC_JSON interface as disabled in the Environments in order to prevent errors when running the test connection on the task's environment (the Fabric server has a different IP address than the local Windows machine and cannot connect to the local machine's directory).
Import the list of shared global variables required for executing TDM in your project.
TDM shared functions are saved in the TDM Logic file.
Import the TDM shared functions to your project. Note that since the TDM category contains the product's functions, it is recommended to add the project's shared functions to a separate category (Logic file) in order to simplify the TDM version upgrading.
TDM 8.1 replaces the previous TDM translation with MTables to support a development of the TDM on both Fabric Studios: Desktop-Studio and Web-Studio.
The following MTables have been added to the References in the TDM library. Note that you must deploy the Reference to Fabric after updating the MTables:
Item |
Description |
Instructions |
MigrateList |
Define the query and interface name, or the Broadway flow to generate the entity list when running the extract task on all entities of each LU; one record per LU. |
Populate this table for each Logical Unit. A separate record must be created for each Logical Unit in the Fabric project apart from TDM, TDM_LIBRARY and the dummy LU of the post-execution processes. If there is a need to define a query per source environment, populate the source environment name and create a separate record for each Logical Unit and source_env_name combination. Otherwise, leave the source environment empty. Click here for more information on how to implement a Broadway flow to get the entities (populated in external_table_flow field of MigrateList table). Example 1:
Example 2:
Example 3:
|
MigrateListQueryFormats |
Supports special syntax for extract tasks when creating the LU instance query based on the MigrateList table. Each LUI consists of a concatenation of the source environment, IID, version name and version datetime. Click to read more about LUI structure for TDM implementation. This table is required for databases that do not support the standard ‘||’ syntax for concatenated strings, e.g., sqlServer. |
Populate 2 records for each database, one record with version_ind ‘true’ and another with version_ind ‘false’. Example 1:
Example 2:
|
RefList |
Define the list of available reference tables for TDM tasks. Click to read more about Reference implementation. |
Populate this table for each reference table. A separate record must be created for each reference table. Set the LU name on each record. |
PostProcessList |
Define the list of post-processes to run at the end of the task's execution. For example, a process that sends a mail to notify the user when the task's execution ends. Each process is implemented as a Broadway flow. |
Populate the list of Broadway flows and the LU of the Broadway flow. The LU can be empty if the post processes are defined under Shared Objects, whereby the TDM task execution process sets the LU Name to TDM when running Batch commands to carry out post execution processes. Redeploy the LUs populated in this table, the TDM LU, and the Web-Services. |
ChildLink |
Mapping parent and child IDs. Click for more information about TDM business entities and how to support a hierarchy when implementing the LUs. |
A record must be added to this table for each parent-child relationship. The parent_lu field must be populated with the name of the parent LU and child_lu field must be populated with the name of the child LU. Both SQLs populated in child_lu_eid_sql and child_lu_tar_eid_Sql fields must run on the parent LU and get the source and target child IDs for each parent ID. Example:
The parameters: tables, subscriber and tar_subscriber, must all be defined in the CRM LU schema. |
LuParams |
Translation for the population of the LU_PARAMS table. |
The COLUMN_NAME is populated by the name of the parameter and the SQL is populated by the SQL query that gets the values for the defined parameter. Click for more information about handling parameters. |
The Fabric TDM library includes a set of built-in generic Broadway flows, defined for easy adaptation of a generic TDM implementation for a specific data model.
Click for more information about Generic TDM Broadway Flows.
The TDM Logical Unit must be deployed to the Fabric project. It has the following tasks:
TDM enables setting TTL (Time To Live) on the TDM LUIs. The default TTL period is 10 days. The TDM LUI's TTL depends on the following shared Globals (imported from the TDM Library):
The deploy.flow process runs the following activities upon the TDM LU deployment:
Deploy the TDM LU to Fabric. From TDM 7.6 onwards, the deployment of the TDM LU deploys the TDM Portal as well into the TDM web applications.
Notes:
The TDM_LIBRARY LU holds utilities that must be copied to the project's LUs. These utilities are described below.
It is recommended to duplicate the TDM_Library LU and use it as a template when creating a new LU in a TDM project.
Populate the ROOT_TABLE_NAME Global using the main source table/s. You can populate several tables, separated by a comma.
Examples:
Populate the ROOT_COLUMN_NAME Global using the entity ID's column. These Globals are needed for setting the IS_INSTANCE_ID column correctly in TDM_SEQ_MAPPING TDM DB table. Note that the number and order of root column names must be aligned with the number and order of the tables that are populated in ROOT_TABLE_NAME.
Examples:
ROOT_TABLE_NAME | ROOT_COLUMN_NAME |
CUSTOMER | CUSTOMER_ID |
CUSTOMER, ACCOUNT | CUSTOMER_ID, CUSTOMER_ID |
CUSTOMER, ACCOUNT_DATA | CUSTOMER_ID, ACC_CUST_ID |
FABRIC_TDM_ROOT - the Root table of each LU. This table contains the following columns:
Example:
K2_TDM_EID |
IID |
SOURCE_ENV |
TASK_NAME |
TIMESTAMP |
PROD_1 |
1 |
PROD |
|
|
PROD_1_copyCust_20201105090000 |
1 |
PROD |
copyCust |
20201105090000 |
LU_PARAMS - parameters table. Must be added to each LU schema even when it is not required for defining parameters in the LU. The LU_PARAM table holds only the ENTITY_ID and SOURCE_ENVIRONMENT fields.
Click for more information about TDM parameters handling.
TDM_LU_TYPE_RELATION_EID and TDM_LU_TYPE_REL_TAR_EID - TDM relationship tables that map the parent to child IDs. Note that these tables are also created in the TDM DB.
Click for more information about TDM Hierarchy implementation.
TDM 7.6 onwards stores the extracted Reference tables in a new LU - TDM_Reference - instead of storing them in Cassandra. Each Reference table is stored as a separate LUI. For more information see Reference Implementation.