Broadway has a streams category of Actors that handle streams such as reading from and writing to a file or compressing and decompressing data.
For example:
To prevent memory overload, when Broadways reads from a stream or file, the data is split into blocks of a constant size of 8K. As a result, data can be cut in the middle of an object (e.g. in the middle of a line in the file), whereby the stream must then be handled by a Parser Actor.
The following example shows a flow where the FileRead Actor reads the data from a CSV file followed by the CsvParser Actor which parses the data into separate lines.
Since the FileRead Actor can only work on one file in the flow at a time, it can be preceded by the file system handling Actors. For example, using the ls Actor to list the files in a given directory and iterating over the files using the FileRead Actor.
Another way to handle multiple input files is by creating an Interface Listener Job that listens to the predefined interface and path and invokes the Broadway flow when a file arrives.
The file.flow example shows a flow with various steps that handle streams including compressing data, creating a directory and files, listing the files and moving them between directories.
Click Actions > Examples in the Main menu to open the file.flow example.
Broadway has a streams category of Actors that handle streams such as reading from and writing to a file or compressing and decompressing data.
For example:
To prevent memory overload, when Broadways reads from a stream or file, the data is split into blocks of a constant size of 8K. As a result, data can be cut in the middle of an object (e.g. in the middle of a line in the file), whereby the stream must then be handled by a Parser Actor.
The following example shows a flow where the FileRead Actor reads the data from a CSV file followed by the CsvParser Actor which parses the data into separate lines.
Since the FileRead Actor can only work on one file in the flow at a time, it can be preceded by the file system handling Actors. For example, using the ls Actor to list the files in a given directory and iterating over the files using the FileRead Actor.
Another way to handle multiple input files is by creating an Interface Listener Job that listens to the predefined interface and path and invokes the Broadway flow when a file arrives.
The file.flow example shows a flow with various steps that handle streams including compressing data, creating a directory and files, listing the files and moving them between directories.
Click Actions > Examples in the Main menu to open the file.flow example.