Azure Blob Storage Interface

Azure Blob Storage interface type is used to define the connections between a blob storage and a data stream.

When creating an Interface Listener for a Broadway flow, an Azure Blob Storage interface is needed to detect new files added to the storage.

To create a new Azure Blob Storage interface, do the following:

  1. Go to Project Tree > Shared Objects, right click Interfaces, select New Interface and then select Azure Blob Storage from the File System section to open the New Interface window.

    image

  2. Populate the connection's settings and click Save.

  1. Go to Project Tree > Shared Objects, right click Interfaces, select New Interface and then select Azure Blob Storage from the Interface Type dropdown menu to open the New Interface window.

  2. Enter a suitable name for your new Asuze Blob Storage Interface, then click Create

    image

  3. Populate the connection's settings and click Save.

    image

  4. If the interface is supposed to be used for File Cataloging, expand the Discovery section and populate the names of 3 Broadway flows. This option is available starting from Fabric V8.3. Click here for more information about the File Cataloging solution.

Connection Settings

Add an Interface Listener as a Broadway job. Click to create an Interface Listener job under the specified Logical Unit.

Parameter Description
Container name The name of the blob container within your storage account where files are stored. Must be lowercase, 3-63 characters, and contain only letters, numbers, and hyphens.
Working Path The specific folder path within the container where the connector will look for files
Files Filter Filters files based on the below filter type.
Files Filter Type

Two types are supported:

  • Wildcard – supports filtering using files wildcard pattern.
  • Regular expression – supports filtering using regex.
Recursive Indicator, to allow displaying all files in embedded folders.
Account name The unique name of your Azure Storage Account. Must be 3-24 characters, lowercase letters and numbers only.
Account key The primary or secondary access key for your storage account. This is a base64-encoded string that provides authentication to access the storage account.
Tenant Id Your Azure Active Directory (Azure AD) tenant identifier. A GUID that uniquely identifies your organization's Azure AD instance.
Client Id The Application (client) ID of the Azure AD app registration used for authentication. Identifies the specific application requesting access.
Resource Id The full Azure Resource Manager path to your storage account, including subscription, resource group, and storage account name.
Client secret The secret key associated with your Azure AD app registration. Works with the Client Id to authenticate application access.
Endpoint templates URL templates for accessing blob storage endpoints. Typically follows the format: https://{accountname}.blob.core.windows.net/
Credentials type Authentication method for accessing the storage account.
Discovery Broadway flows, invoked upon running Discovery for analyzing and cataloging S3 bucket contents. These flows define mapping and transformation rules to convert the files into the Catalog’s standard hierarchy:
  • Get Metadata - Retrieves metadata information about files and objects
  • Get Files List - Generates a list of all files in the specified bucket/path
  • Get File Data - Extracts actual file content and data for processing

Example of Using an Azure Blob Storage Interface

To create an Interface Listener Job that runs on an Azure Blob Storage interface, do the following:

  1. Create an interface using an Azure Blob Storage interface type.

  2. Create a Broadway flow either under Shared Objects or under the same Logical Unit. The flow reads data from a file using the predefined interface and populates it into the DB.

images

  • Note that the interface and the path input arguments of the FileRead Actor are defined as External link type. Their values are passed from the defined interface by the Listener.
  1. Add an InterfaceListener Actor to the "deploy.flow" flow, located at the Broadway folder. Use the Broadway flow, which you created in the previous step, as the flowName property in this actor.
  2. Deploy the LU to activate the Listener.

Using the InterfaceListener Actor

The InterfaceListener Actor enables the flow in which it is instantiated to listen to Azure Blob Storage interface and trigger another Broadway flow upon arrival of a new file on the interface.

To create an Interface Listener job from a Broadway flow, add the InterfaceListener Actor to the flow.

images

Fill in the following parameters in the Actor's Properties tab:

  • flowName, the flow to be triggered by the Interface Listener.

  • interfaceName, the interface that is being listened and used to trigger the flow defined above, once a new file is detected on the file system to which the interface points.

  • affinity, sets which node/DC name IP address is to be used to run the Interface Listener job.

  • params, refer to the arguments that can be passed to the flow. For example, multiple parameters can be parsed as a key/value object from an external link or from a Const or JavaScript Actor.

Previous

Azure Blob Storage Interface

Azure Blob Storage interface type is used to define the connections between a blob storage and a data stream.

When creating an Interface Listener for a Broadway flow, an Azure Blob Storage interface is needed to detect new files added to the storage.

To create a new Azure Blob Storage interface, do the following:

  1. Go to Project Tree > Shared Objects, right click Interfaces, select New Interface and then select Azure Blob Storage from the File System section to open the New Interface window.

    image

  2. Populate the connection's settings and click Save.

  1. Go to Project Tree > Shared Objects, right click Interfaces, select New Interface and then select Azure Blob Storage from the Interface Type dropdown menu to open the New Interface window.

  2. Enter a suitable name for your new Asuze Blob Storage Interface, then click Create

    image

  3. Populate the connection's settings and click Save.

    image

  4. If the interface is supposed to be used for File Cataloging, expand the Discovery section and populate the names of 3 Broadway flows. This option is available starting from Fabric V8.3. Click here for more information about the File Cataloging solution.

Connection Settings

Add an Interface Listener as a Broadway job. Click to create an Interface Listener job under the specified Logical Unit.

Parameter Description
Container name The name of the blob container within your storage account where files are stored. Must be lowercase, 3-63 characters, and contain only letters, numbers, and hyphens.
Working Path The specific folder path within the container where the connector will look for files
Files Filter Filters files based on the below filter type.
Files Filter Type

Two types are supported:

  • Wildcard – supports filtering using files wildcard pattern.
  • Regular expression – supports filtering using regex.
Recursive Indicator, to allow displaying all files in embedded folders.
Account name The unique name of your Azure Storage Account. Must be 3-24 characters, lowercase letters and numbers only.
Account key The primary or secondary access key for your storage account. This is a base64-encoded string that provides authentication to access the storage account.
Tenant Id Your Azure Active Directory (Azure AD) tenant identifier. A GUID that uniquely identifies your organization's Azure AD instance.
Client Id The Application (client) ID of the Azure AD app registration used for authentication. Identifies the specific application requesting access.
Resource Id The full Azure Resource Manager path to your storage account, including subscription, resource group, and storage account name.
Client secret The secret key associated with your Azure AD app registration. Works with the Client Id to authenticate application access.
Endpoint templates URL templates for accessing blob storage endpoints. Typically follows the format: https://{accountname}.blob.core.windows.net/
Credentials type Authentication method for accessing the storage account.
Discovery Broadway flows, invoked upon running Discovery for analyzing and cataloging S3 bucket contents. These flows define mapping and transformation rules to convert the files into the Catalog’s standard hierarchy:
  • Get Metadata - Retrieves metadata information about files and objects
  • Get Files List - Generates a list of all files in the specified bucket/path
  • Get File Data - Extracts actual file content and data for processing

Example of Using an Azure Blob Storage Interface

To create an Interface Listener Job that runs on an Azure Blob Storage interface, do the following:

  1. Create an interface using an Azure Blob Storage interface type.

  2. Create a Broadway flow either under Shared Objects or under the same Logical Unit. The flow reads data from a file using the predefined interface and populates it into the DB.

images

  • Note that the interface and the path input arguments of the FileRead Actor are defined as External link type. Their values are passed from the defined interface by the Listener.
  1. Add an InterfaceListener Actor to the "deploy.flow" flow, located at the Broadway folder. Use the Broadway flow, which you created in the previous step, as the flowName property in this actor.
  2. Deploy the LU to activate the Listener.

Using the InterfaceListener Actor

The InterfaceListener Actor enables the flow in which it is instantiated to listen to Azure Blob Storage interface and trigger another Broadway flow upon arrival of a new file on the interface.

To create an Interface Listener job from a Broadway flow, add the InterfaceListener Actor to the flow.

images

Fill in the following parameters in the Actor's Properties tab:

  • flowName, the flow to be triggered by the Interface Listener.

  • interfaceName, the interface that is being listened and used to trigger the flow defined above, once a new file is detected on the file system to which the interface points.

  • affinity, sets which node/DC name IP address is to be used to run the Interface Listener job.

  • params, refer to the arguments that can be passed to the flow. For example, multiple parameters can be parsed as a key/value object from an external link or from a Const or JavaScript Actor.

Previous