TDM Error Handling and Statistics Flows
The TDM library includes a set of generic flows for error handling and statistics gathering that are based on Broadway capabilities and are tailored for TDM business requirements.
These generic flows gather errors and statistics during task executions and populate them into dedicated tables. This data is used for monitoring TDM tasks and creating TDM execution reports.
How Do I Perform Error Handling in TDM?
The TDM library includes 2 utility flows that handle errors during the execution of TDM tasks:
- PopulateTableErrorsWithFailed.flow
- PopulateTableErrorsWithReject.flow
Both utilities invoke the internal PopulateTableErrors.flow in order to populate data about errors into the task_exe_error_detailed table. The difference between the utilities is that PopulateTableErrorsWithFailed.flow sets the entity_status on a session level:
ENTITY_STATUS = failed
PopulateTableErrorsWithFailed.flow also sets the error category as Entity Failed in the task_exe_error_detailed table, while PopulateTableErrorsWithReject.flow sets a record as Record Rejected.
The error handling utility is invoked from each Load flow's Load Data To Target Stage. An error can be suppressed in order to continue a task execution and to reach the statistics gathering step.
By default, the PopulateTableErrorsWithFailed is invoked and the Suppress checkbox is unchecked, that is, the entity is rejected due to the error:
If a record needs to be rejected - instead of failing an entire entity - replace the Inner flow name with PopulateTableErrorsWithReject and check the Suppress checkbox.
Click to learn how to use the ErrorHandling Actor.
How Do I Gather Statistics in TDM?
TDM Task Execution Report
The task execution report includes the Statistics Report tab that compares the number of records in each table in the source and target environments.
The TDM library includes the StatsLoader Broadway Actor that populates the statistics data into task_exe_stats_detailed TDM DB table. The generated load flows include the following 2 Broadway Actors:
- StatsReader - gets the load statistics from the DbLoad Actor that loads the data to the target DB.
- StatsLoader - gets the statistical information from the StatsReader Actor and populates it in task_exe_stats_detailed TDM DB table. This table is extracted by the task execution report API to generate the Statistics Report tab of the Task Execution Report.
See an example below:
TDM Custom JMX Statistics
Fabric provides JMX metrics to enable comprehensive and low-resolution monitoring and management of applications. The JMX metrics can be accessed using monitoring tools such as Grafana. Additionally, Fabric enables adding customized JMX statistics for a better monitoring.
TDM 7.6 adds the following customized JMX metrics in order to have a better monitoring on the TDM executions. The metrics are extracted from the TDMDB by the TDM LU sync:
TotalLoadedRecordsPerLoadFlow - number of records loaded to the target DB by each load Broadway flow. The following information is provided:
- LU name (luName)
- Table name (tableName)
- Flow name (flowName)
- Number of records (RecCount)
Note that the custom metrics are added as the broadwayFlow JMX metrics provide data about the number of flow executions and not about the total number of records, loaded by each flow.
TaskExecutionPerBE - number of task executions per Business Entity (BE). The following information is provided:
- BE name (beName)
- Number of executions (RecCount)
TaskExecutionPerBEAndStatus - number of task executions per Business Entity (BE) and execution status. The following information is provided:
BE name and execution status, concatenated by '#'. For example: Customer#completed
Number of executions (RecCount)
TotalTaskExecutions - total number of task executions.
TotalTaskExecutionsPerStatus - total number of task executions per execution status.
Implementation Guidelines
Set the POPULATE_JMX_STATS shared Global to true to enable the execution of populationJMX population flow on TASK_EXECUTION TDM LU table.
Notes:
- Each task execution is accumulated as one execution even if the task contains multiple LUs.
- TDM 8.0 required adding the fnTDMJMXStats enrichment function to the TDM LU (TASK_EXECUTION LU table) to enable the TDM customized JMX metrics. TDM 8.1 does not require this manual edit of the TDM LU schema. Instead, you need to set the POPULATE_JMX_STATS Global to true.
TDM Error Handling and Statistics Flows
The TDM library includes a set of generic flows for error handling and statistics gathering that are based on Broadway capabilities and are tailored for TDM business requirements.
These generic flows gather errors and statistics during task executions and populate them into dedicated tables. This data is used for monitoring TDM tasks and creating TDM execution reports.
How Do I Perform Error Handling in TDM?
The TDM library includes 2 utility flows that handle errors during the execution of TDM tasks:
- PopulateTableErrorsWithFailed.flow
- PopulateTableErrorsWithReject.flow
Both utilities invoke the internal PopulateTableErrors.flow in order to populate data about errors into the task_exe_error_detailed table. The difference between the utilities is that PopulateTableErrorsWithFailed.flow sets the entity_status on a session level:
ENTITY_STATUS = failed
PopulateTableErrorsWithFailed.flow also sets the error category as Entity Failed in the task_exe_error_detailed table, while PopulateTableErrorsWithReject.flow sets a record as Record Rejected.
The error handling utility is invoked from each Load flow's Load Data To Target Stage. An error can be suppressed in order to continue a task execution and to reach the statistics gathering step.
By default, the PopulateTableErrorsWithFailed is invoked and the Suppress checkbox is unchecked, that is, the entity is rejected due to the error:
If a record needs to be rejected - instead of failing an entire entity - replace the Inner flow name with PopulateTableErrorsWithReject and check the Suppress checkbox.
Click to learn how to use the ErrorHandling Actor.
How Do I Gather Statistics in TDM?
TDM Task Execution Report
The task execution report includes the Statistics Report tab that compares the number of records in each table in the source and target environments.
The TDM library includes the StatsLoader Broadway Actor that populates the statistics data into task_exe_stats_detailed TDM DB table. The generated load flows include the following 2 Broadway Actors:
- StatsReader - gets the load statistics from the DbLoad Actor that loads the data to the target DB.
- StatsLoader - gets the statistical information from the StatsReader Actor and populates it in task_exe_stats_detailed TDM DB table. This table is extracted by the task execution report API to generate the Statistics Report tab of the Task Execution Report.
See an example below:
TDM Custom JMX Statistics
Fabric provides JMX metrics to enable comprehensive and low-resolution monitoring and management of applications. The JMX metrics can be accessed using monitoring tools such as Grafana. Additionally, Fabric enables adding customized JMX statistics for a better monitoring.
TDM 7.6 adds the following customized JMX metrics in order to have a better monitoring on the TDM executions. The metrics are extracted from the TDMDB by the TDM LU sync:
TotalLoadedRecordsPerLoadFlow - number of records loaded to the target DB by each load Broadway flow. The following information is provided:
- LU name (luName)
- Table name (tableName)
- Flow name (flowName)
- Number of records (RecCount)
Note that the custom metrics are added as the broadwayFlow JMX metrics provide data about the number of flow executions and not about the total number of records, loaded by each flow.
TaskExecutionPerBE - number of task executions per Business Entity (BE). The following information is provided:
- BE name (beName)
- Number of executions (RecCount)
TaskExecutionPerBEAndStatus - number of task executions per Business Entity (BE) and execution status. The following information is provided:
BE name and execution status, concatenated by '#'. For example: Customer#completed
Number of executions (RecCount)
TotalTaskExecutions - total number of task executions.
TotalTaskExecutionsPerStatus - total number of task executions per execution status.
Implementation Guidelines
Set the POPULATE_JMX_STATS shared Global to true to enable the execution of populationJMX population flow on TASK_EXECUTION TDM LU table.
Notes:
- Each task execution is accumulated as one execution even if the task contains multiple LUs.
- TDM 8.0 required adding the fnTDMJMXStats enrichment function to the TDM LU (TASK_EXECUTION LU table) to enable the TDM customized JMX metrics. TDM 8.1 does not require this manual edit of the TDM LU schema. Instead, you need to set the POPULATE_JMX_STATS Global to true.
