Kafka Interface

A Kafka interface type defines the connections to Apache Kafka and can be used by the Broadway Pub / Sub Actors, iidFinder or by User Jobs.

To create a new Kafka interface, do the following:

  1. Go to Project Tree > Shared Objects, right click Interfaces and select New Interface. Then select Kafka from the Interface Type dropdown menu to open the New Interface window.

    image

  2. Populate the relevant properties and click Save.

Connection Settings

Parameter Description
Topic

Apache Kafka topic name.

Can be overriden by the Broadway Actor's setting.

Bootstrap Servers Server IP.
Group ID

Apache Kafka group ID.

Can be overriden by the Broadway Actor's setting. 

Data Type 

Supported data types: String, byte[], JSON, long.

The message type to be processed by Broadway Actors must be aligned with the Data type definition.

Max Poll Records

Maximum poll records.

Can be overriden by the Broadway Actor's setting. 

Affinity (Optional) Populate this parameter with the IP address of a Fabric node, DC name, or logical identifier for Fabric nodes. Fabric allocates an execution server based on the populated affinity. If the affinity is not populated, Fabric allocates an available server for the execution.

Example of an affinity: ’10.21.1.169’, ‘DC1’.

Can be overriden by the User Job's settings.

Message Processing Timeout (ms) (Optional) The timeout to wait for a new message, default is 10,000 ms.

Can be overriden by the Broadway Actor's setting.

SSL properties Mandatory properties if Kafka is defined with SSL (available starting from V6.5.1).
SSL optional properties Optional properties if Kafka is defined with SSL.
SASL properties Mandatory properties if Kafka is defined with SASL authentication (available starting from V6.5.4).

The following SASL flavors are supported: SASL_PLAIN, SASL_SCRAM, SASL_LDAP, SASL_GSSAPI.

When SASL_SCRAM is used, you must fill in the Session Hash Policy in addition to the Session User Name & Password.

Example of Publish to Kafka Broadway Flow

image

The above Broadway flow uses a Publish Actor to publish the data to the predefined Kafka interface. The data is published to the Example topic which overrides the topic in the Interface definition.

Previous

Kafka Interface

A Kafka interface type defines the connections to Apache Kafka and can be used by the Broadway Pub / Sub Actors, iidFinder or by User Jobs.

To create a new Kafka interface, do the following:

  1. Go to Project Tree > Shared Objects, right click Interfaces and select New Interface. Then select Kafka from the Interface Type dropdown menu to open the New Interface window.

    image

  2. Populate the relevant properties and click Save.

Connection Settings

Parameter Description
Topic

Apache Kafka topic name.

Can be overriden by the Broadway Actor's setting.

Bootstrap Servers Server IP.
Group ID

Apache Kafka group ID.

Can be overriden by the Broadway Actor's setting. 

Data Type 

Supported data types: String, byte[], JSON, long.

The message type to be processed by Broadway Actors must be aligned with the Data type definition.

Max Poll Records

Maximum poll records.

Can be overriden by the Broadway Actor's setting. 

Affinity (Optional) Populate this parameter with the IP address of a Fabric node, DC name, or logical identifier for Fabric nodes. Fabric allocates an execution server based on the populated affinity. If the affinity is not populated, Fabric allocates an available server for the execution.

Example of an affinity: ’10.21.1.169’, ‘DC1’.

Can be overriden by the User Job's settings.

Message Processing Timeout (ms) (Optional) The timeout to wait for a new message, default is 10,000 ms.

Can be overriden by the Broadway Actor's setting.

SSL properties Mandatory properties if Kafka is defined with SSL (available starting from V6.5.1).
SSL optional properties Optional properties if Kafka is defined with SSL.
SASL properties Mandatory properties if Kafka is defined with SASL authentication (available starting from V6.5.4).

The following SASL flavors are supported: SASL_PLAIN, SASL_SCRAM, SASL_LDAP, SASL_GSSAPI.

When SASL_SCRAM is used, you must fill in the Session Hash Policy in addition to the Session User Name & Password.

Example of Publish to Kafka Broadway Flow

image

The above Broadway flow uses a Publish Actor to publish the data to the predefined Kafka interface. The data is published to the Example topic which overrides the topic in the Interface definition.

Previous