Iterations
Overview
Iterations are used to repeatedly perform a section of a Broadway flow in a data set. Iterations are similar to a for…each loop in the sense that a logic is repeatedly run until no data remains to act upon.
The common use cases are iterating over a database result set, over a data returned by an HTTP call or an API or over Kafka messages. For example, Broadway population flow includes an iteration over the source DB data for loading it into the target DB.
This article described how Broadway performs the following:
- Define an iteration in a flow.
- Loop over one or more elements of a result set.
- Create a nested iteration within an iteration.
- Control the loops programmatically using JavaScript.
To learn about more complex iteration scenarios, such as iterating on different levels of the output object's hierarchy or combining multiple result sets, refer to the Complex Iteration Flows article.
How Do I Define an Iteration in a Flow?
To start the iteration, draw the line from the originating Actor's output and set the line's Link Type to Iterate. The connection line becomes double-dotted, the background of the loop's scope is highlighted grey and a thick divider line is displayed at the starting point.
The scope of the iteration's logic starts immediately after the iteration line and continues until the end of the flow or until the Iterate Close Stage. To mark an iteration Stage as closed, click 
in the Stage context menu > Iterate Close to display a second thick divider line at the iteration ending point.
In the below example, the StringsArray Actor is the iteration's originating Actor that returns an array of strings and the Iterate on array Stage runs on each array element. The iteration is closed at the end of Iterate on array Stage.
How Do I Iterate Over a Data Set?
The iteration over the data set starts once the originating Actor's output is connected to another Actor's input using an Iterate link type.
The originating Actor's output can be a simple object (for example, an SQL query result set) or a complex object that includes internal hierarchy (for example, a JSON object).
The Iterate connection can be drawn from the output object itself or from any of its elements.
Iterate Over a Row in a Result Array
The result output returns an array of maps. In each iteration, take the values of the entire map.
Iterate Over an Element in a Result Array
Iterate on all the values of the CUSTOMER_ID field in the result output array.
Iterate Over Two or More Elements
Iterate over several elements of the result set whereby it is possible to combine different link types.
Broadway takes the values of the connected elements from the same iteration.
In the below example: iterate over the result output array and in each iteration take the values of the current iteration.
Note that combining the link types will change the data flow as follows:
- Elements connected by an Iterate link type, will return the value of the current iteration.
- Elements connected by a Value (or First) link type, will return the value of the first iteration during all iterations.
Click for more information about Complex Iteration Flows.
How Can I Create a Nested Iteration?
Iterations can also be nested. For example, a value in an iteration can be used as an input for another iteration. The depth of the iteration is highlighted in shades of grey. To limit the loop's scope using Iterate Close, add a closing Stage to each level of the loop. There are no limitations on the iteration nesting level. However, to make a flow more readable, consider limiting a flow to 3-4 nesting levels and using Inner Flows when more are needed.
In the following image, the first name is an input to a query that gets a list of relevant phone numbers. Stage 2 is run on every entry in Stage 1 and Stage 3 on every entry in Stage 2.
ForLoop Actor
The ForLoop Actor can be used to create a virtual data set of integers in a given range. This enables creating a loop that runs N times over a synthetic data set and is useful for repeat iterations when there is no data set to traverse.
Programmatic Control
The Broadway Context object enables an Actor to programmatically access and control the loop using the Loop interface. A Loop Context object can be accessed via the Context.loop() method in Java or using the contextLoop instance in JavaScript.
The following methods are supported:
Loop.stop(), stops the current loop and continues execution after the loop. All Actors in the same Stage as the calling Actor are still invoked.
Loop.skip(), skips the current loop iteration and continues to the next data entry. All Actors in the current Stage are still invoked.
Loop.index(), returns the current loop index. The index of the first iteration = 0.
In a nested loop, you can only access the inner-most (deepest) loop that is running in the current Stage.
For more information, refer to the iterate-for-each.flow Broadway example.
Iterations
Overview
Iterations are used to repeatedly perform a section of a Broadway flow in a data set. Iterations are similar to a for…each loop in the sense that a logic is repeatedly run until no data remains to act upon.
The common use cases are iterating over a database result set, over a data returned by an HTTP call or an API or over Kafka messages. For example, Broadway population flow includes an iteration over the source DB data for loading it into the target DB.
This article described how Broadway performs the following:
- Define an iteration in a flow.
- Loop over one or more elements of a result set.
- Create a nested iteration within an iteration.
- Control the loops programmatically using JavaScript.
To learn about more complex iteration scenarios, such as iterating on different levels of the output object's hierarchy or combining multiple result sets, refer to the Complex Iteration Flows article.
How Do I Define an Iteration in a Flow?
To start the iteration, draw the line from the originating Actor's output and set the line's Link Type to Iterate. The connection line becomes double-dotted, the background of the loop's scope is highlighted grey and a thick divider line is displayed at the starting point.
The scope of the iteration's logic starts immediately after the iteration line and continues until the end of the flow or until the Iterate Close Stage. To mark an iteration Stage as closed, click in the Stage context menu > Iterate Close to display a second thick divider line at the iteration ending point.
In the below example, the StringsArray Actor is the iteration's originating Actor that returns an array of strings and the Iterate on array Stage runs on each array element. The iteration is closed at the end of Iterate on array Stage.
How Do I Iterate Over a Data Set?
The iteration over the data set starts once the originating Actor's output is connected to another Actor's input using an Iterate link type.
The originating Actor's output can be a simple object (for example, an SQL query result set) or a complex object that includes internal hierarchy (for example, a JSON object).
The Iterate connection can be drawn from the output object itself or from any of its elements.
Iterate Over a Row in a Result Array
The result output returns an array of maps. In each iteration, take the values of the entire map.
Iterate Over an Element in a Result Array
Iterate on all the values of the CUSTOMER_ID field in the result output array.
Iterate Over Two or More Elements
Iterate over several elements of the result set whereby it is possible to combine different link types.
Broadway takes the values of the connected elements from the same iteration.
In the below example: iterate over the result output array and in each iteration take the values of the current iteration.
Note that combining the link types will change the data flow as follows:
- Elements connected by an Iterate link type, will return the value of the current iteration.
- Elements connected by a Value (or First) link type, will return the value of the first iteration during all iterations.
Click for more information about Complex Iteration Flows.
How Can I Create a Nested Iteration?
Iterations can also be nested. For example, a value in an iteration can be used as an input for another iteration. The depth of the iteration is highlighted in shades of grey. To limit the loop's scope using Iterate Close, add a closing Stage to each level of the loop. There are no limitations on the iteration nesting level. However, to make a flow more readable, consider limiting a flow to 3-4 nesting levels and using Inner Flows when more are needed.
In the following image, the first name is an input to a query that gets a list of relevant phone numbers. Stage 2 is run on every entry in Stage 1 and Stage 3 on every entry in Stage 2.
ForLoop Actor
The ForLoop Actor can be used to create a virtual data set of integers in a given range. This enables creating a loop that runs N times over a synthetic data set and is useful for repeat iterations when there is no data set to traverse.
Programmatic Control
The Broadway Context object enables an Actor to programmatically access and control the loop using the Loop interface. A Loop Context object can be accessed via the Context.loop() method in Java or using the contextLoop instance in JavaScript.
The following methods are supported:
Loop.stop(), stops the current loop and continues execution after the loop. All Actors in the same Stage as the calling Actor are still invoked.
Loop.skip(), skips the current loop iteration and continues to the next data entry. All Actors in the current Stage are still invoked.
Loop.index(), returns the current loop index. The index of the first iteration = 0.
In a nested loop, you can only access the inner-most (deepest) loop that is running in the current Stage.
For more information, refer to the iterate-for-each.flow Broadway example.
