The TDM Library includes all the utilities required to implement a TDM project and run TDM execution processes. It holds the following:
The TDM Library must be imported into the Fabric project created for TDM.
Deploy all TDM Web Services (APIs) to the Fabric project. These Web Services are invoked by the TDM self-service application and form its back-end layer.
Since TDM categories contain the product's Web Services, it is recommended to add the project's Web Services into separate categories, which would simplify the TDM version upgrade process.
Import and deploy the following interfaces into the project's Shared Objects:
POSTGRESQL_ADMIN — this is the admin connection to the TDM PostgreSQL DB. This interface is used by the TDMDB flow within the TDM LU to create the TDM PostgreSQL DB.
TDM — this is the connection to the TDM PosgreSQL DB. Edit the IP address according to the environment.
Note that if you are working on a PostgreSQL DB over an SSL connection, you must edit the custom connection string of both the POSTGRESQL_ADMIN and TDM interfaces as follows:
AI interfaces — AI DB and Kubernetes interfaces. Starting from TDM V9.0, an integrated AI solution is available for generating synthetic data. The AI-related interfaces must be disabled in case the AI machine is not installed and the AI-based data generation process is not in use.
From TDM V9.4 onwards, the TDM shared Globals are split into two files:
A new Global, SEQ_CACHE_INTERFACE, was introduced in TDM V8.1. This Global is populated with the DB interface of the k2masking DB (PostgreSQL or Cassandra) and must be aligned with the Fabric System DB. From TDM V9.0 onwards, TDM sets the POSTGRESQL_ADMIN as a default value for this Global:
The TDM shared functions are saved in the TDM Logic file.
Import the TDM shared functions into your project. Note that since the TDM category contains the product's functions, it is recommended to add the project's shared functions to a separate category (Logic file) in order to simplify the TDM version upgrade process.
In TDM V8.1, the previous TDM translation has been replaced with MTables to support development on both Fabric Studios: Desktop-Studio and Web-Studio.
The following MTables have been added to the References by the TDM Library. Note that you must deploy the Reference to Fabric after updating the MTables:
Item |
Description |
Instructions |
MigrateList |
Defines the interface name and the SQL query, or alternatively the Broadway flow, to generate the entity list when running a task with a Predefined entity list selection method for the entity's subset; one record per LU and environment name. The environment can be empty. |
Populate this table for each Logical Unit. A separate record must be created for each Logical Unit in the Fabric project, apart from TDM, TDM_LIBRARY, and the dummy LU of the post-execution processes. If there is a need to define a query per source environment, populate the source environment name and create a separate record for each Logical Unit and source_env_name combination. Otherwise, leave the source environment empty. Example 1:
Example 2:
|
MigrateListQueryFormats |
Supports special syntax for extract tasks when creating the LU instance query based on the MigrateList table. Each LUI consists of a concatenation of the source environment, IID, version name and version datetime. Click to read more about LUI structure for TDM implementation. This table is required for databases that do not support the standard ‘||’ syntax for concatenated strings, e.g., sqlServer. |
Populate 2 records for each database, one record with version_ind ‘true’ and another with version_ind ‘false’. Example 1:
Example 2:
|
RefList |
Defines the list of available tables related to a Business entity, a list that can be included on a TDM task for Entities and referential data. Additionally, it can be used for setting different interface/schema/table name between the source and target environments for table level Click to read more about Tables implementation. |
A separate record must be created for each table. Set the LU name on each record. |
TableLevelInterfaces and TableLevelDefinitions |
These MTables define special handling of tasks with tables. | A separate record must be set for each DB or table.
Click to read more about Tables implementation. |
PostAndPreExecutionProcess |
Defines the list of pre-execution and post-execution flows to run before or at the end of a task's execution. For example, a process that sends a mail to notify the user when the task's execution ends, or a process that populates a mapping table before the LU execution starts. Each process is implemented as a Broadway flow. |
Populate the list of Broadway flows, the LU of the Broadway flow and the process type (pre/post). Each flow can have external parameters that can be overridden by the task's creator. The LU can be empty if the processes are defined under Shared Objects, whereby the TDM task execution process sets the LU Name to TDM when running Batch commands to carry out pre/post execution processes. Redeploy the LUs populated in this table, the TDM LU, and the Web Services. |
ChildLink |
Mapping parent and child IDs. Click for more information about TDM Business Entities and how to support a hierarchy when implementing the LUs. |
A record must be added to this table for each parent-child relationship. The parent_lu field must be populated with the name of the parent LU and the child_lu field must be populated with the name of the child LU. Both SQLs, populated in child_lu_eid_sql and child_lu_tar_eid_Sql fields, must run on the parent LU and get the source and target child IDs for each parent ID. Example:
The parameters — tables, subscriber and tar_subscriber — must all be defined in the CRM LU schema. |
LuParams |
This table is used for populating business parameters. |
The COLUMN_NAME is populated by the name of the parameter and the SQL is populated by the SQL query that gets the values for the defined parameter. Click for more information about handling parameters. |
LuParamsMapping |
This table is used for populating business parameters when the |
Map each parameter to an LU table's field. Click for more information about handling parameters. |
AI configuration tables |
Configration settings for AI-based data generation. | Click for more information about AI-based data generation implementation |
The Fabric TDM Library includes a set of built-in generic Broadway flows and templates, designed to easily add the TDM setup to the LU.
The TDM Logical Unit must be deployed to the Fabric project. It has the following tasks:
TDM enables setting TTL (Time To Live) on the TDM LUIs. The default TTL period is 10 days. The TDM LUI's TTL depends on the following shared Globals (imported from the TDM Library):
The deploy.flow process runs the following activities upon the TDM LU deployment:
Deploy the TDM LU to Fabric. From TDM 7.6 onwards, deploying the TDM LU also deploys the TDM Portal into the Web applications.
Notes:
The TDM_LIBRARY LU holds utilities that must be copied to the project's LUs. These utilities are described below.
It is recommended to duplicate the TDM_Library LU and use it as a template when creating a new LU in a TDM project.
Populate the ROOT_TABLE_NAME Global using the main source table(s). You can populate several tables, separated by a comma.
Examples:
Populate the ROOT_COLUMN_NAME Global using the entity ID's column. These Globals are needed for setting the IS_INSTANCE_ID column correctly in TDM_SEQ_MAPPING TDM DB table. Note that the number and order of root column names must be aligned with the number and order of the tables that are populated in ROOT_TABLE_NAME.
Examples:
ROOT_TABLE_NAME | ROOT_COLUMN_NAME |
CUSTOMER | CUSTOMER_ID |
CUSTOMER, ACCOUNT | CUSTOMER_ID, CUSTOMER_ID |
CUSTOMER, ACCOUNT_DATA | CUSTOMER_ID, ACC_CUST_ID |
FABRIC_TDM_ROOT — the Root table of each LU. This table contains the following columns:
Example:
K2_TDM_EID |
IID |
SOURCE_ENV |
TASK_NAME |
TIMESTAMP |
PROD_1 |
1 |
PROD |
|
|
PROD_1_copyCust_20201105090000 |
1 |
PROD |
copyCust |
20201105090000 |
LU_PARAMS — parameters table. This table must be added to each LU schema even when it is not required for defining parameters in the LU. The LU_PARAM table holds only the ENTITY_ID and SOURCE_ENVIRONMENT fields.
Click for more information about TDM parameter handling.
Starting from TDM V9.0, extracted tables are stores in a new LU — TDM_TableLevel. Each table is stored as a separate LUI. For more information, read Tables Implementation.
The TDM Library includes all the utilities required to implement a TDM project and run TDM execution processes. It holds the following:
The TDM Library must be imported into the Fabric project created for TDM.
Deploy all TDM Web Services (APIs) to the Fabric project. These Web Services are invoked by the TDM self-service application and form its back-end layer.
Since TDM categories contain the product's Web Services, it is recommended to add the project's Web Services into separate categories, which would simplify the TDM version upgrade process.
Import and deploy the following interfaces into the project's Shared Objects:
POSTGRESQL_ADMIN — this is the admin connection to the TDM PostgreSQL DB. This interface is used by the TDMDB flow within the TDM LU to create the TDM PostgreSQL DB.
TDM — this is the connection to the TDM PosgreSQL DB. Edit the IP address according to the environment.
Note that if you are working on a PostgreSQL DB over an SSL connection, you must edit the custom connection string of both the POSTGRESQL_ADMIN and TDM interfaces as follows:
AI interfaces — AI DB and Kubernetes interfaces. Starting from TDM V9.0, an integrated AI solution is available for generating synthetic data. The AI-related interfaces must be disabled in case the AI machine is not installed and the AI-based data generation process is not in use.
From TDM V9.4 onwards, the TDM shared Globals are split into two files:
A new Global, SEQ_CACHE_INTERFACE, was introduced in TDM V8.1. This Global is populated with the DB interface of the k2masking DB (PostgreSQL or Cassandra) and must be aligned with the Fabric System DB. From TDM V9.0 onwards, TDM sets the POSTGRESQL_ADMIN as a default value for this Global:
The TDM shared functions are saved in the TDM Logic file.
Import the TDM shared functions into your project. Note that since the TDM category contains the product's functions, it is recommended to add the project's shared functions to a separate category (Logic file) in order to simplify the TDM version upgrade process.
In TDM V8.1, the previous TDM translation has been replaced with MTables to support development on both Fabric Studios: Desktop-Studio and Web-Studio.
The following MTables have been added to the References by the TDM Library. Note that you must deploy the Reference to Fabric after updating the MTables:
Item |
Description |
Instructions |
MigrateList |
Defines the interface name and the SQL query, or alternatively the Broadway flow, to generate the entity list when running a task with a Predefined entity list selection method for the entity's subset; one record per LU and environment name. The environment can be empty. |
Populate this table for each Logical Unit. A separate record must be created for each Logical Unit in the Fabric project, apart from TDM, TDM_LIBRARY, and the dummy LU of the post-execution processes. If there is a need to define a query per source environment, populate the source environment name and create a separate record for each Logical Unit and source_env_name combination. Otherwise, leave the source environment empty. Example 1:
Example 2:
|
MigrateListQueryFormats |
Supports special syntax for extract tasks when creating the LU instance query based on the MigrateList table. Each LUI consists of a concatenation of the source environment, IID, version name and version datetime. Click to read more about LUI structure for TDM implementation. This table is required for databases that do not support the standard ‘||’ syntax for concatenated strings, e.g., sqlServer. |
Populate 2 records for each database, one record with version_ind ‘true’ and another with version_ind ‘false’. Example 1:
Example 2:
|
RefList |
Defines the list of available tables related to a Business entity, a list that can be included on a TDM task for Entities and referential data. Additionally, it can be used for setting different interface/schema/table name between the source and target environments for table level Click to read more about Tables implementation. |
A separate record must be created for each table. Set the LU name on each record. |
TableLevelInterfaces and TableLevelDefinitions |
These MTables define special handling of tasks with tables. | A separate record must be set for each DB or table.
Click to read more about Tables implementation. |
PostAndPreExecutionProcess |
Defines the list of pre-execution and post-execution flows to run before or at the end of a task's execution. For example, a process that sends a mail to notify the user when the task's execution ends, or a process that populates a mapping table before the LU execution starts. Each process is implemented as a Broadway flow. |
Populate the list of Broadway flows, the LU of the Broadway flow and the process type (pre/post). Each flow can have external parameters that can be overridden by the task's creator. The LU can be empty if the processes are defined under Shared Objects, whereby the TDM task execution process sets the LU Name to TDM when running Batch commands to carry out pre/post execution processes. Redeploy the LUs populated in this table, the TDM LU, and the Web Services. |
ChildLink |
Mapping parent and child IDs. Click for more information about TDM Business Entities and how to support a hierarchy when implementing the LUs. |
A record must be added to this table for each parent-child relationship. The parent_lu field must be populated with the name of the parent LU and the child_lu field must be populated with the name of the child LU. Both SQLs, populated in child_lu_eid_sql and child_lu_tar_eid_Sql fields, must run on the parent LU and get the source and target child IDs for each parent ID. Example:
The parameters — tables, subscriber and tar_subscriber — must all be defined in the CRM LU schema. |
LuParams |
This table is used for populating business parameters. |
The COLUMN_NAME is populated by the name of the parameter and the SQL is populated by the SQL query that gets the values for the defined parameter. Click for more information about handling parameters. |
LuParamsMapping |
This table is used for populating business parameters when the |
Map each parameter to an LU table's field. Click for more information about handling parameters. |
AI configuration tables |
Configration settings for AI-based data generation. | Click for more information about AI-based data generation implementation |
The Fabric TDM Library includes a set of built-in generic Broadway flows and templates, designed to easily add the TDM setup to the LU.
The TDM Logical Unit must be deployed to the Fabric project. It has the following tasks:
TDM enables setting TTL (Time To Live) on the TDM LUIs. The default TTL period is 10 days. The TDM LUI's TTL depends on the following shared Globals (imported from the TDM Library):
The deploy.flow process runs the following activities upon the TDM LU deployment:
Deploy the TDM LU to Fabric. From TDM 7.6 onwards, deploying the TDM LU also deploys the TDM Portal into the Web applications.
Notes:
The TDM_LIBRARY LU holds utilities that must be copied to the project's LUs. These utilities are described below.
It is recommended to duplicate the TDM_Library LU and use it as a template when creating a new LU in a TDM project.
Populate the ROOT_TABLE_NAME Global using the main source table(s). You can populate several tables, separated by a comma.
Examples:
Populate the ROOT_COLUMN_NAME Global using the entity ID's column. These Globals are needed for setting the IS_INSTANCE_ID column correctly in TDM_SEQ_MAPPING TDM DB table. Note that the number and order of root column names must be aligned with the number and order of the tables that are populated in ROOT_TABLE_NAME.
Examples:
ROOT_TABLE_NAME | ROOT_COLUMN_NAME |
CUSTOMER | CUSTOMER_ID |
CUSTOMER, ACCOUNT | CUSTOMER_ID, CUSTOMER_ID |
CUSTOMER, ACCOUNT_DATA | CUSTOMER_ID, ACC_CUST_ID |
FABRIC_TDM_ROOT — the Root table of each LU. This table contains the following columns:
Example:
K2_TDM_EID |
IID |
SOURCE_ENV |
TASK_NAME |
TIMESTAMP |
PROD_1 |
1 |
PROD |
|
|
PROD_1_copyCust_20201105090000 |
1 |
PROD |
copyCust |
20201105090000 |
LU_PARAMS — parameters table. This table must be added to each LU schema even when it is not required for defining parameters in the LU. The LU_PARAM table holds only the ENTITY_ID and SOURCE_ENVIRONMENT fields.
Click for more information about TDM parameter handling.
Starting from TDM V9.0, extracted tables are stores in a new LU — TDM_TableLevel. Each table is stored as a separate LUI. For more information, read Tables Implementation.