Kafka Interface

A Kafka interface type defines the connections to Apache Kafka and can be used by the Broadway Pub / Sub Actors, iidFinder or by User Jobs.

To create a new Kafka interface, do the following:

  1. Go to Project Tree > Shared Objects, right click Interfaces and select New Interface and then select Kafka from the Interface Type dropdown menu to open the New Interface window.

    image

  2. Populate the connection's settings and click Save.

Connection Settings

Parameter Description
Topic

Apache Kafka topic name.

Can be overriden by the Broadway Actor's setting.

Bootstrap Servers Server IP.
Group ID

Apache Kafka group ID.

Can be overriden by the Broadway Actor's setting. 

Data Type 

Supported data types: String, byte[], JSON, long.

The message type to be processed by Broadway Actors must be aligned with the Data type definition.

Max Poll Records

Maximum poll records.

Can be overriden by the Broadway Actor's setting. 

Affinity (Optional) Populate this parameter with the IP address of a Fabric node, DC name, or logical identifier for Fabric nodes. Fabric allocates an execution server based on the populated affinity. If the affinity is not populated, Fabric allocates an available server for the execution.

Example of an affinity: ’10.21.1.169’, ‘DC1’.

Can be overriden by the User Job's settings.

SSL properties Mandatory properties if Kafka is defined with SSL.
SSL optional properties Optional properties if Kafka is defined with SSL.

Example of Publish to Kafka Broadway Flow

image

The above Broadway flow uses a Publish Actor to publish the data to the predefined Kafka interface. The data is published to the Example topic which overrides the topic in the Interface definition.

Previous

Kafka Interface

A Kafka interface type defines the connections to Apache Kafka and can be used by the Broadway Pub / Sub Actors, iidFinder or by User Jobs.

To create a new Kafka interface, do the following:

  1. Go to Project Tree > Shared Objects, right click Interfaces and select New Interface and then select Kafka from the Interface Type dropdown menu to open the New Interface window.

    image

  2. Populate the connection's settings and click Save.

Connection Settings

Parameter Description
Topic

Apache Kafka topic name.

Can be overriden by the Broadway Actor's setting.

Bootstrap Servers Server IP.
Group ID

Apache Kafka group ID.

Can be overriden by the Broadway Actor's setting. 

Data Type 

Supported data types: String, byte[], JSON, long.

The message type to be processed by Broadway Actors must be aligned with the Data type definition.

Max Poll Records

Maximum poll records.

Can be overriden by the Broadway Actor's setting. 

Affinity (Optional) Populate this parameter with the IP address of a Fabric node, DC name, or logical identifier for Fabric nodes. Fabric allocates an execution server based on the populated affinity. If the affinity is not populated, Fabric allocates an available server for the execution.

Example of an affinity: ’10.21.1.169’, ‘DC1’.

Can be overriden by the User Job's settings.

SSL properties Mandatory properties if Kafka is defined with SSL.
SSL optional properties Optional properties if Kafka is defined with SSL.

Example of Publish to Kafka Broadway Flow

image

The above Broadway flow uses a Publish Actor to publish the data to the predefined Kafka interface. The data is published to the Example topic which overrides the topic in the Interface definition.

Previous