Google Cloud Storage Interface

Google Cloud Storage interface type is used to define the connections between a blob storage and a data stream.

When creating an Interface Listener for a Broadway flow, an Google Cloud Storage interface is needed to detect new files added to the storage.

To create a new Google Cloud Storage interface, do the following:

  1. Go to Project Tree > Shared Objects, right click Interfaces, select New Interface and then select Google Cloud Storage from the File System section to open the New Interface window.

    image

  2. Populate the connection's settings and click Save.

  1. Go to Project Tree > Shared Objects, right click Interfaces, select New Interface and then select Google Cloud Storage from the Interface Type dropdown menu to open the New Interface window.

  2. Enter a suitable name for your new Google Cloud Storage Interface, then click Create

    image

  3. Populate the connection's settings and click Save.

    image

Connection Settings

Test Connection. Click to test the connection.

Add an Interface Listener as a Broadway job. Click to create an Interface Listener job under the specified Logical Unit.

Parameter Description
Bucket name The name of the Google Cloud Storage bucket where your files are stored. This is a required field. Must be globally unique, 3-63 characters, lowercase letters, numbers, periods, and hyphens only.
Working Path The specific folder path within the bucket where the connector will look for files.
Files Filter Filters files using regular expressions to specify which files to look for.
Project ID The Google Cloud Platform project ID where your bucket is located. This is a required field with a globally unique identifier, typically 6-30 characters using lowercase letters, numbers, and hyphens.
Location ID The Google Cloud region or multi-region where your bucket is located (e.g., us-central1, europe-west1, asia). This is a required field and must match the actual location of your bucket.
Credentials file The location of the credentials file. This file includes the private key and service account details required to access the GCS bucket securely.
Discovery Discoveryoptions for analyzing and cataloging GCS bucket contents. Available options include Get Metadata, Get Files List, and Get File Data for different levels of bucket analysis.

Example of Using an Google Cloud Storage Interface

To create an Interface Listener Job that runs on an Google Cloud Storage interface, do the following:

  1. Create an interface using an Google Cloud Storage interface type.

  2. Create a Broadway flow either under Shared Objects or under the same Logical Unit. The flow reads data from a file using the predefined interface and populates it into the DB.

images

  • Note that the interface and the path input arguments of the FileRead Actor are defined as External link type. Their values are passed from the defined interface by the Listener.
  1. Add an InterfaceListener Actor to the "deploy.flow" flow, located at the Broadway folder. Use the Broadway flow, which you created in the previous step, as the flowName property in this actor.
  2. Deploy the LU to activate the Listener.

Using the InterfaceListener Actor

The InterfaceListener Actor enables the flow in which it is instantiated to listen to Google Cloud Storage interface and trigger another Broadway flow upon arrival of a new file on the interface.

To create an Interface Listener job from a Broadway flow, add the InterfaceListener Actor to the flow.

images

Fill in the following parameters in the Actor's Properties tab:

  • flowName, the flow to be triggered by the Interface Listener.

  • interfaceName, the interface that is being listened and used to trigger the flow defined above, once a new file is detected on the file system to which the interface points.

  • affinity, sets which node/DC name IP address is to be used to run the Interface Listener job.

  • params, refer to the arguments that can be passed to the flow. For example, multiple parameters can be parsed as a key/value object from an external link or from a Const or JavaScript Actor.

Previous

Google Cloud Storage Interface

Google Cloud Storage interface type is used to define the connections between a blob storage and a data stream.

When creating an Interface Listener for a Broadway flow, an Google Cloud Storage interface is needed to detect new files added to the storage.

To create a new Google Cloud Storage interface, do the following:

  1. Go to Project Tree > Shared Objects, right click Interfaces, select New Interface and then select Google Cloud Storage from the File System section to open the New Interface window.

    image

  2. Populate the connection's settings and click Save.

  1. Go to Project Tree > Shared Objects, right click Interfaces, select New Interface and then select Google Cloud Storage from the Interface Type dropdown menu to open the New Interface window.

  2. Enter a suitable name for your new Google Cloud Storage Interface, then click Create

    image

  3. Populate the connection's settings and click Save.

    image

Connection Settings

Test Connection. Click to test the connection.

Add an Interface Listener as a Broadway job. Click to create an Interface Listener job under the specified Logical Unit.

Parameter Description
Bucket name The name of the Google Cloud Storage bucket where your files are stored. This is a required field. Must be globally unique, 3-63 characters, lowercase letters, numbers, periods, and hyphens only.
Working Path The specific folder path within the bucket where the connector will look for files.
Files Filter Filters files using regular expressions to specify which files to look for.
Project ID The Google Cloud Platform project ID where your bucket is located. This is a required field with a globally unique identifier, typically 6-30 characters using lowercase letters, numbers, and hyphens.
Location ID The Google Cloud region or multi-region where your bucket is located (e.g., us-central1, europe-west1, asia). This is a required field and must match the actual location of your bucket.
Credentials file The location of the credentials file. This file includes the private key and service account details required to access the GCS bucket securely.
Discovery Discoveryoptions for analyzing and cataloging GCS bucket contents. Available options include Get Metadata, Get Files List, and Get File Data for different levels of bucket analysis.

Example of Using an Google Cloud Storage Interface

To create an Interface Listener Job that runs on an Google Cloud Storage interface, do the following:

  1. Create an interface using an Google Cloud Storage interface type.

  2. Create a Broadway flow either under Shared Objects or under the same Logical Unit. The flow reads data from a file using the predefined interface and populates it into the DB.

images

  • Note that the interface and the path input arguments of the FileRead Actor are defined as External link type. Their values are passed from the defined interface by the Listener.
  1. Add an InterfaceListener Actor to the "deploy.flow" flow, located at the Broadway folder. Use the Broadway flow, which you created in the previous step, as the flowName property in this actor.
  2. Deploy the LU to activate the Listener.

Using the InterfaceListener Actor

The InterfaceListener Actor enables the flow in which it is instantiated to listen to Google Cloud Storage interface and trigger another Broadway flow upon arrival of a new file on the interface.

To create an Interface Listener job from a Broadway flow, add the InterfaceListener Actor to the flow.

images

Fill in the following parameters in the Actor's Properties tab:

  • flowName, the flow to be triggered by the Interface Listener.

  • interfaceName, the interface that is being listened and used to trigger the flow defined above, once a new file is detected on the file system to which the interface points.

  • affinity, sets which node/DC name IP address is to be used to run the Interface Listener job.

  • params, refer to the arguments that can be passed to the flow. For example, multiple parameters can be parsed as a key/value object from an external link or from a Const or JavaScript Actor.

Previous