Kafka Interface

A Kafka interface type defines the connections to Apache Kafka and can be used by the Broadway Pub / Sub Actors, iidFinder or by User Jobs.

To create a new Kafka interface, do the following:

  1. Go to Project Tree > Shared Objects, right click Interfaces and select New Interface and then select Kafka from the Interface Type dropdown menu to open the New Interface window.

    image

  2. Populate the connection's settings and click Save.

Connection Settings

Parameter Description
Topic

Apache Kafka topic name.

Can be overriden by the Broadway Actor's setting.

Bootstrap Servers Server IP.
Group ID

Apache Kafka group ID.

Can be overriden by the Broadway Actor's setting. 

Data Type 

Supported data types: String, byte[], JSON, long.

The message type to be processed by Broadway Actors must be aligned with the Data type definition.

Max Poll Records

Maximum poll records.

Can be overriden by the Broadway Actor's setting. 

Affinity (Optional) Populate this parameter with the IP address of a Fabric node, DC name, or logical identifier for Fabric nodes. Fabric allocates an execution server based on the populated affinity. If the affinity is not populated, Fabric allocates an available server for the execution.

Example of an affinity: ’10.21.1.169’, ‘DC1’.

Can be overriden by the User Job's settings.

SSL / SASL properties Mandatory properties if Kafka is defined with SSL or SASL (available starting from V6.5.1).
SSL optional properties Optional properties if Kafka is defined with SSL.

Example of SSL Properties Definition

SSL Security Protocol

When SSL authentication is required on Kafka, set Enable SSL/SASL property to True and populate the SSL properties on the Kafka Interface. Below is the example of Interface definition when the Security Protocol = SSL.

image

SASL_SSL / SASL_PLAINTEXT Security Protocols

When you need to connect to Kafka using a user and a password, set the Security Protocol = SASL_SSL or SASL_PLAINTEXT and populate the Session user name and password attributes.

In addition, in case of SASL_SSL - populate the Truststore and Keystore properties as below.

image

Example of Publish to Kafka Broadway Flow

image

The above Broadway flow uses a Publish Actor to publish the data to the predefined Kafka interface. The data is published to the Example topic which overrides the topic in the Interface definition.

Previous

Kafka Interface

A Kafka interface type defines the connections to Apache Kafka and can be used by the Broadway Pub / Sub Actors, iidFinder or by User Jobs.

To create a new Kafka interface, do the following:

  1. Go to Project Tree > Shared Objects, right click Interfaces and select New Interface and then select Kafka from the Interface Type dropdown menu to open the New Interface window.

    image

  2. Populate the connection's settings and click Save.

Connection Settings

Parameter Description
Topic

Apache Kafka topic name.

Can be overriden by the Broadway Actor's setting.

Bootstrap Servers Server IP.
Group ID

Apache Kafka group ID.

Can be overriden by the Broadway Actor's setting. 

Data Type 

Supported data types: String, byte[], JSON, long.

The message type to be processed by Broadway Actors must be aligned with the Data type definition.

Max Poll Records

Maximum poll records.

Can be overriden by the Broadway Actor's setting. 

Affinity (Optional) Populate this parameter with the IP address of a Fabric node, DC name, or logical identifier for Fabric nodes. Fabric allocates an execution server based on the populated affinity. If the affinity is not populated, Fabric allocates an available server for the execution.

Example of an affinity: ’10.21.1.169’, ‘DC1’.

Can be overriden by the User Job's settings.

SSL / SASL properties Mandatory properties if Kafka is defined with SSL or SASL (available starting from V6.5.1).
SSL optional properties Optional properties if Kafka is defined with SSL.

Example of SSL Properties Definition

SSL Security Protocol

When SSL authentication is required on Kafka, set Enable SSL/SASL property to True and populate the SSL properties on the Kafka Interface. Below is the example of Interface definition when the Security Protocol = SSL.

image

SASL_SSL / SASL_PLAINTEXT Security Protocols

When you need to connect to Kafka using a user and a password, set the Security Protocol = SASL_SSL or SASL_PLAINTEXT and populate the Session user name and password attributes.

In addition, in case of SASL_SSL - populate the Truststore and Keystore properties as below.

image

Example of Publish to Kafka Broadway Flow

image

The above Broadway flow uses a Publish Actor to publish the data to the predefined Kafka interface. The data is published to the Example topic which overrides the topic in the Interface definition.

Previous