Knowledge Base Academy Community My Tickets

Home Knowledge Base Academy Community My Tickets

Fabric 8.1

Fabric 8.1 Fabric 8.0 Fabric 7.2 Fabric 7.1 Fabric 7.0 Fabric 6.5 Fabric 6.4

Knowledge Base Sync Methods

Sync Methods

Sync Properties

Sync properties can be defined on an LU schema, LU table, or Table Population level.

A Sync property contains the following settings:

Timeout (sec)	The timeout in seconds for syncing the LUI.
Sync Method	None, Time Interval, Inherited and Decision function.
Parameters	Settings of the selected sync method. For more details see the Sync Methods section below.
Delete Instance if Not Exists	When marked as True, Fabric deletes the LUI if the LUI is not found in the source system. When marked as False (default), the instance is retained in Fabric even if the instance is not found in the source system.

Click for more information about Set Timeout for Sync.

Sync Methods

None

Sync Method	None
Description	Do not sync
Parameters	N/A

Time Interval

Sync Method

Time interval.

Description

Synchronization is implemented automatically after each predefined time interval in order to check whether the latest update of the LU instance in Fabric occurred before the time interval of the specific instance.

· If it has, data is extracted from the source and reloaded to Fabric.

· If it hasn't, the current data in Fabric is used.

Parameters

Sync is performed every:

Format = D.HH:MM:SS.

When set to 00:00:00 - if sync mode is force or on - sync always takes place when getting the LUI.

Default = 1 day (1:00:00:00).

Decision Function

Sync Method	Decision.
Description	Synchronization is implemented according to the Decision function, which returns a Boolean (True/False) parameter. If the Decision function returns False, do not sync the object.
Parameters	Select the Decision function from the Predefined Decision Functions dropdown list.

Inherited

Sync Method

Inherited.

Description

Synchronization of each level inherits the sync rule of its direct parent branch according to the following hierarchy:

· LU table inherits from the LU schema.

· Table Population inherits from the LU table.

Parameters

N/A

Click for more information about LU Sync Levels

Delete Mode and Truncate Before Sync properties

Fabric 6.5.9 adds the Delete Mode property to the LU table. This property replaces the previous Truncate Before Sync LU table's property. The Delete Mode defines the policy regarding the deletion of the previous records (populated prior to the current sync) from the LU table by the current sync. Therefore, there is a logical dependency between the delete mode and the sync mode. The Delete Mode can be set either on the LU table or on the table's populations:

1. LU Table Settings

The Delete Mode property is set on the LU table and defines the delete mode of the previous records in the LU table (populated prior to the current sync). The values are All (default value), Off or NonUpdated:

All - the entire LU table is truncated before the populations are executed.
Off - the previous records are not deleted.
NonUpdated - deletes previous records (created by the earlier sync) where data no longer exists for the LUI in the source.

Notes:

It is recommended to set the NonUpdated value when the LU table has CDC fields in order to send CDC messages only for the updated records. If the Delete Mode is set to All, Fabric sends delete messages for all the truncated records and inserts messages for the newly inserted records.
If the Delete Mode is NonUpdated, it is recommended to define a PK on the LU table and set the LU table population mode to Upsert or Update in order to delete only the old data. If the LU table does not have a PK, new records are added to the LU table and all previous records are deleted.

2. LU Table Population Settings

2.1 Table Population Object

A table population object that is based on a source object contains the following property: Truncate Before Sync.

This property is set on the LU table population target properties. The Truncate Before Sync = True is equivalent to the All value in the Delete Mode LU table's property.

Notes:

When the Truncate Before Sync is set to True on a given LU table population, it truncates the entire LU table before populating it. Such setting overrides the LU table's Delete Mode.
When the Truncate Before Sync is set to False, the delete mode it taken from the LU table.

2.2. Table Populations Based on Broadway Flows

A new Broadway actor has been added in Fabric 6.5.9: SyncDeleteMode. This actor is automatically added to the generated Broadway flow with Off value.

The actor can have one of the following values:

Off (default) - when set, it gets the delete mode from the LU table.
All - deletes all records from the LU table before sync. This value is equivalent to Truncate Before Sync = True on a table population object.
NonUpdated - deletes previous records (created by the earlier sync) where data no longer exists for the LUI in the source.

Notes:

The SyncDeleteMode has the table external input parameter. By default, the table parameter gets the populated table name. In order to enable a population of multiple LU tables by a single Broadway population flow, it is possible to override the table input parameter in both actors in the population flow: SyncDeleteMode and DbLoad. This way it possible to add multiple instances of the DbLoad and SyncDeleteMode actors in one single population flow.
Unlike the table population object, the population flow deletes all the LU table's records before populating the table if the SyncDeleteMode actor is populated with All on the population flow. This may cause undesired results when the population flow deletes records populated by other populations in this table. Therefore, if running all LU table populations deletes the previous records from the LU table before its population, it is recommended to set the Delete Mode to All on the LU table level and to set the SyncDeleteMode to Off on the table's populations.

3. Delete Mode - LU Table and Population Levels

In general, the Delete Mode is set on the LU table. However, the Delete Mode can be overridden by the LU table population if the LU table's Delete Mode is not set to All as detailed in the table below:

LU table - Delete Records Property	Population - Truncate Before Sync	BF population - SyncDeleteMode value	Expected Behavior
All	FALSE/TRUE	All/Off/NonUpdated	Delete all previous records before running the table's populations.
Off	FALSE	Off	Do not delete the previous records before the sync.
Off	FALSE	All	Delete all previous records only if the Broadway population flow runs.
Off	FALSE	NonUpdated	Delete the non-updated records only if the Broadway population flow runs.
Off	TRUE	Off	Delete all previous records if the table population object runs.
Off	TRUE	All	Delete all previous records if any of the populations run.
Off	TRUE	NonUpdated	> Delete all previous records if the table population object runs. > Delete the non-updated records only if the Broadway flow population runs.
NonUpdated	FALSE	Off/NonUpdated	Delete the non-updated records if any of the populations run as set in the LU table.
NonUpdated	FALSE	All	> Delete all previous records if the Broadway population flow runs. > If only the population object runs, delete the non-updated records as defined in the LU table level.
NonUpdated	TRUE	Off/NonUpdated	> Delete all previous records if the table population object runs. > If only the Broadway population flow runs, delete the non-updated records as defined in the LU table level.
NonUpdated	TRUE	All	Delete all previous records in any of the population runs.

Sync Methods - Use Cases

When is a sync method selected?
None	The source either does not change or becomes unavailable, and therefore requires a one-time load only. After it is loaded, Fabric becomes the System of Record for the data and may get update transactions on the data. For example: adding a new payment.
Time Interval	Syncs every X times to ensure the data is up-to-date.
Decision Function	Requires a specific logic to check whether the data needs to be synced from the source.

EXAMPLES

Sync Method	Example 1	Example 2
None	Load task execution statistics. Once the execution is finished, load its statistics. The statistics are not updated.	Get initial Customer data loaded from the Billing system. Additional payment transactions may then be sent by the Payment Gateway system to update the Payment LUI table for the customer. There is no need to re-load the entire Customer object from the Billing system.
Time Interval	Sync the data every day or every week.	Sync the data every day.
Decision Function	Check the source environment: Do not run a sync on the Production environment. However, if the source environment is a Testing environment, run a sync.	An LU that can be populated by different source systems. Set a different population with a different logic for each table and then check the environment on the decision function of each population to decide whether the population needs to be executed based on the environment.

Sync Methods

Sync Properties

Sync properties can be defined on an LU schema, LU table, or Table Population level.

A Sync property contains the following settings:

Timeout (sec)	The timeout in seconds for syncing the LUI.
Sync Method	None, Time Interval, Inherited and Decision function.
Parameters	Settings of the selected sync method. For more details see the Sync Methods section below.
Delete Instance if Not Exists	When marked as True, Fabric deletes the LUI if the LUI is not found in the source system. When marked as False (default), the instance is retained in Fabric even if the instance is not found in the source system.

Click for more information about Set Timeout for Sync.

Sync Methods

None

Sync Method	None
Description	Do not sync
Parameters	N/A

Time Interval

Sync Method

Time interval.

Description

· If it has, data is extracted from the source and reloaded to Fabric.

· If it hasn't, the current data in Fabric is used.

Parameters

Sync is performed every:

Format = D.HH:MM:SS.

When set to 00:00:00 - if sync mode is force or on - sync always takes place when getting the LUI.

Default = 1 day (1:00:00:00).

Decision Function

Sync Method	Decision.
Description	Synchronization is implemented according to the Decision function, which returns a Boolean (True/False) parameter. If the Decision function returns False, do not sync the object.
Parameters	Select the Decision function from the Predefined Decision Functions dropdown list.

Inherited

Sync Method

Inherited.

Description

Synchronization of each level inherits the sync rule of its direct parent branch according to the following hierarchy:

· LU table inherits from the LU schema.

· Table Population inherits from the LU table.

Parameters

N/A

Click for more information about LU Sync Levels

Delete Mode and Truncate Before Sync properties

1. LU Table Settings

All - the entire LU table is truncated before the populations are executed.
Off - the previous records are not deleted.
NonUpdated - deletes previous records (created by the earlier sync) where data no longer exists for the LUI in the source.

Notes:

It is recommended to set the NonUpdated value when the LU table has CDC fields in order to send CDC messages only for the updated records. If the Delete Mode is set to All, Fabric sends delete messages for all the truncated records and inserts messages for the newly inserted records.
If the Delete Mode is NonUpdated, it is recommended to define a PK on the LU table and set the LU table population mode to Upsert or Update in order to delete only the old data. If the LU table does not have a PK, new records are added to the LU table and all previous records are deleted.

2. LU Table Population Settings

2.1 Table Population Object

A table population object that is based on a source object contains the following property: Truncate Before Sync.

This property is set on the LU table population target properties. The Truncate Before Sync = True is equivalent to the All value in the Delete Mode LU table's property.

Notes:

When the Truncate Before Sync is set to True on a given LU table population, it truncates the entire LU table before populating it. Such setting overrides the LU table's Delete Mode.
When the Truncate Before Sync is set to False, the delete mode it taken from the LU table.

2.2. Table Populations Based on Broadway Flows

A new Broadway actor has been added in Fabric 6.5.9: SyncDeleteMode. This actor is automatically added to the generated Broadway flow with Off value.

The actor can have one of the following values:

Off (default) - when set, it gets the delete mode from the LU table.
All - deletes all records from the LU table before sync. This value is equivalent to Truncate Before Sync = True on a table population object.
NonUpdated - deletes previous records (created by the earlier sync) where data no longer exists for the LUI in the source.

Notes:

The SyncDeleteMode has the table external input parameter. By default, the table parameter gets the populated table name. In order to enable a population of multiple LU tables by a single Broadway population flow, it is possible to override the table input parameter in both actors in the population flow: SyncDeleteMode and DbLoad. This way it possible to add multiple instances of the DbLoad and SyncDeleteMode actors in one single population flow.
Unlike the table population object, the population flow deletes all the LU table's records before populating the table if the SyncDeleteMode actor is populated with All on the population flow. This may cause undesired results when the population flow deletes records populated by other populations in this table. Therefore, if running all LU table populations deletes the previous records from the LU table before its population, it is recommended to set the Delete Mode to All on the LU table level and to set the SyncDeleteMode to Off on the table's populations.

3. Delete Mode - LU Table and Population Levels

LU table - Delete Records Property	Population - Truncate Before Sync	BF population - SyncDeleteMode value	Expected Behavior
All	FALSE/TRUE	All/Off/NonUpdated	Delete all previous records before running the table's populations.
Off	FALSE	Off	Do not delete the previous records before the sync.
Off	FALSE	All	Delete all previous records only if the Broadway population flow runs.
Off	FALSE	NonUpdated	Delete the non-updated records only if the Broadway population flow runs.
Off	TRUE	Off	Delete all previous records if the table population object runs.
Off	TRUE	All	Delete all previous records if any of the populations run.
Off	TRUE	NonUpdated	> Delete all previous records if the table population object runs. > Delete the non-updated records only if the Broadway flow population runs.
NonUpdated	FALSE	Off/NonUpdated	Delete the non-updated records if any of the populations run as set in the LU table.
NonUpdated	FALSE	All	> Delete all previous records if the Broadway population flow runs. > If only the population object runs, delete the non-updated records as defined in the LU table level.
NonUpdated	TRUE	Off/NonUpdated	> Delete all previous records if the table population object runs. > If only the Broadway population flow runs, delete the non-updated records as defined in the LU table level.
NonUpdated	TRUE	All	Delete all previous records in any of the population runs.

Sync Methods - Use Cases

When is a sync method selected?
None	The source either does not change or becomes unavailable, and therefore requires a one-time load only. After it is loaded, Fabric becomes the System of Record for the data and may get update transactions on the data. For example: adding a new payment.
Time Interval	Syncs every X times to ensure the data is up-to-date.
Decision Function	Requires a specific logic to check whether the data needs to be synced from the source.

EXAMPLES

Sync Method	Example 1	Example 2
None	Load task execution statistics. Once the execution is finished, load its statistics. The statistics are not updated.	Get initial Customer data loaded from the Billing system. Additional payment transactions may then be sent by the Payment Gateway system to update the Payment LUI table for the customer. There is no need to re-load the entire Customer object from the Billing system.
Time Interval	Sync the data every day or every week.	Sync the data every day.
Decision Function	Check the source environment: Do not run a sync on the Production environment. However, if the source environment is a Testing environment, run a sync.	An LU that can be populated by different source systems. Set a different population with a different logic for each table and then check the environment on the decision function of each population to decide whether the population needs to be executed based on the environment.