Sequence Implementation Guide
Broadway enables generating and setting new sequences before loading data into a target database. Various sequence patterns can be implemented via the Sequence and MaskingSequence Actors and other Broadway features.
This article describes the most useful use cases of sequence implementation by Broadway.
Sequence Caching
A common scenario of sequence implementation is when the same sequence needs to be used per entity across several flows during the same execution. In order to use the caching capabilities, use the MaskingSequence Actor, or wrap the Sequence Actor with the Masking Actor.
The following use cases of sequence caching are supported:
- Use a sequence across several tables of the same LU. For example, Customer ID is a sequential field in the CUSTOMER LU and is populated in several LU tables such as CUSTOMER and SUBSCRIBER.
- Use a sequence across different LUIs within the same LU. For example, the same ADDRESS ID can be used for different customers during the same execution.
- Use a sequence across different LU types. For example, the same CUSTOMER ID can be used in both a CUSTOMER LU and a Billing LU during the same execution.
To implement the above use cases, set a unique maskingId and populate it on either the MaskingSequence or the Masking Actors, where the same sequence is used. Keep the useEnvironment as true and the useExecutionID as true in each Actor's settings in order to generate a new masked value in each execution, in the same environment. Setting these arguments to false allows you to use the same masked value across different executions and environments.
Sequence Next Value
The sequence next value implementation method depends on the sequence definition set by the sequenceInterface input argument of the MaskingSequence and the Sequence Actors. The following use cases are supported:
- IN-MEMORY, useful for testing as it can only be used in a single node configuration.
- Redis or DB sequence. Getting the next value from the DB sequence is supported for Oracle, DB2 and PostgreSQL DBs. To implement the DB sequence, set the sequenceId input argument to hold the sequence name defined in the sequenceInterface DB.
Notes:
- The sequenceId input is mandatory for the Sequence Actor and optional for the MaskingSequence Actor. The MaskingSequence Actor has both input arguments: maskingId and sequenceId. If the input sequenceId is not set in the MaskingSequence Actor, the sequence name would be taken from the maskingId input argumenent.
- If the sequence interface is a DB, the Actor creates the sequence in the DB, in case it does not exist (lazy initiation). The initial value is set based on the initialValue setting, if populated.
Sequence Initiation Method
Sequence initiation can be performed using the initialValue and the increment settings of the Actor and is only relevant for IN-MEMORY or Redis interface. The initialValue is also relevant for a newly created DB sequence in order to set its initial value upon the sequence creation. In an existing DB sequence, these attributes are managed by the DB. Note that the initial value is cached upon the Actor's first execution. The following use cases are supported:
- Initialize the sequence using the constant initial value, e.g., 1000000.
- Initialize the sequence using another Broadway flow by setting the flow name in the initialValue argument. The Actor invokes the flow to calculate the sequence's initial value. Note that the flow must return an external variable, named initialValue. View the figures below:
Sequence Mapping
In Broadway, sequences can be mapped in a number of ways. The following use cases are supported:
Map the old value to the new value: send the old ID to the input value parameter of the Sequence Actor.
When there is no old value to be mapped to, and the target table requires a sequence, leave the input value empty and set the onEmpty parameter of the Sequence Actor to be MASK_NO_CACHE. The Actor generates a new sequence and returns it in its output, as in the below example:
Set the sequence as part of the attributes list. An example for an attributes list can be a string that concatenates several pairs of keys and values that include the sequence as one of them. To do so, generate the sequence and then create the concatenated attributes list using the JavaScript Actor or Actors belonging to the strings category.
Set the sequence value based on a condition. For example, generate the sequence value only for some entries, based on a given condition. To do so, define a Stage Condition in the Broadway flow. The below example shows a replacement of the customer ID with a new sequence, if the customer ID equals to 1. When no condition is given, it maps the original customer ID:
When parent-child relationships exist across Logical Units, the same sequence can exist in both. The updated flow can be executed on the parent LU in order to add a child sequence. For example, after completing the Customer and Order LUs population, update the parent (i.e., Customer LU) with the sequence from the child (i.e., Order LU).
Store the relationship between the old and the new sequence. To do so, create a flow that stores these values in the Cassandra TDM_SEQ_MAPPING table under the k2masking keyspace, e.g., for reporting purposes.
Clone the entities when required. Different sequence values are generated for each cloned entity. This functionality is supported as part of the TDM implementation.
Custom Sequence Mapping
Create your own function or Broadway flow for generating a new ID, using the MaskingLuFunction or MaskingInnerFlow Actors. Set the category to enable_sequences in order to use the Actor for sequence (ID) replacement.
Click here for more information about the custom masking actors.
Sequence Implementation Guide
Broadway enables generating and setting new sequences before loading data into a target database. Various sequence patterns can be implemented via the Sequence and MaskingSequence Actors and other Broadway features.
This article describes the most useful use cases of sequence implementation by Broadway.
Sequence Caching
A common scenario of sequence implementation is when the same sequence needs to be used per entity across several flows during the same execution. In order to use the caching capabilities, use the MaskingSequence Actor, or wrap the Sequence Actor with the Masking Actor.
The following use cases of sequence caching are supported:
- Use a sequence across several tables of the same LU. For example, Customer ID is a sequential field in the CUSTOMER LU and is populated in several LU tables such as CUSTOMER and SUBSCRIBER.
- Use a sequence across different LUIs within the same LU. For example, the same ADDRESS ID can be used for different customers during the same execution.
- Use a sequence across different LU types. For example, the same CUSTOMER ID can be used in both a CUSTOMER LU and a Billing LU during the same execution.
To implement the above use cases, set a unique maskingId and populate it on either the MaskingSequence or the Masking Actors, where the same sequence is used. Keep the useEnvironment as true and the useExecutionID as true in each Actor's settings in order to generate a new masked value in each execution, in the same environment. Setting these arguments to false allows you to use the same masked value across different executions and environments.
Sequence Next Value
The sequence next value implementation method depends on the sequence definition set by the sequenceInterface input argument of the MaskingSequence and the Sequence Actors. The following use cases are supported:
- IN-MEMORY, useful for testing as it can only be used in a single node configuration.
- Redis or DB sequence. Getting the next value from the DB sequence is supported for Oracle, DB2 and PostgreSQL DBs. To implement the DB sequence, set the sequenceId input argument to hold the sequence name defined in the sequenceInterface DB.
Notes:
- The sequenceId input is mandatory for the Sequence Actor and optional for the MaskingSequence Actor. The MaskingSequence Actor has both input arguments: maskingId and sequenceId. If the input sequenceId is not set in the MaskingSequence Actor, the sequence name would be taken from the maskingId input argumenent.
- If the sequence interface is a DB, the Actor creates the sequence in the DB, in case it does not exist (lazy initiation). The initial value is set based on the initialValue setting, if populated.
Sequence Initiation Method
Sequence initiation can be performed using the initialValue and the increment settings of the Actor and is only relevant for IN-MEMORY or Redis interface. The initialValue is also relevant for a newly created DB sequence in order to set its initial value upon the sequence creation. In an existing DB sequence, these attributes are managed by the DB. Note that the initial value is cached upon the Actor's first execution. The following use cases are supported:
- Initialize the sequence using the constant initial value, e.g., 1000000.
- Initialize the sequence using another Broadway flow by setting the flow name in the initialValue argument. The Actor invokes the flow to calculate the sequence's initial value. Note that the flow must return an external variable, named initialValue. View the figures below:
Sequence Mapping
In Broadway, sequences can be mapped in a number of ways. The following use cases are supported:
Map the old value to the new value: send the old ID to the input value parameter of the Sequence Actor.
When there is no old value to be mapped to, and the target table requires a sequence, leave the input value empty and set the onEmpty parameter of the Sequence Actor to be MASK_NO_CACHE. The Actor generates a new sequence and returns it in its output, as in the below example:
Set the sequence as part of the attributes list. An example for an attributes list can be a string that concatenates several pairs of keys and values that include the sequence as one of them. To do so, generate the sequence and then create the concatenated attributes list using the JavaScript Actor or Actors belonging to the strings category.
Set the sequence value based on a condition. For example, generate the sequence value only for some entries, based on a given condition. To do so, define a Stage Condition in the Broadway flow. The below example shows a replacement of the customer ID with a new sequence, if the customer ID equals to 1. When no condition is given, it maps the original customer ID:
When parent-child relationships exist across Logical Units, the same sequence can exist in both. The updated flow can be executed on the parent LU in order to add a child sequence. For example, after completing the Customer and Order LUs population, update the parent (i.e., Customer LU) with the sequence from the child (i.e., Order LU).
Store the relationship between the old and the new sequence. To do so, create a flow that stores these values in the Cassandra TDM_SEQ_MAPPING table under the k2masking keyspace, e.g., for reporting purposes.
Clone the entities when required. Different sequence values are generated for each cloned entity. This functionality is supported as part of the TDM implementation.
Custom Sequence Mapping
Create your own function or Broadway flow for generating a new ID, using the MaskingLuFunction or MaskingInnerFlow Actors. Set the category to enable_sequences in order to use the Actor for sequence (ID) replacement.
Click here for more information about the custom masking actors.
