The Fabric Change Data Capture (CDC) solution can notify external systems about data changes published via Kafka. Fabric CDC can send different data segments to different CDC consumers and different LU table columns can be defined as CDC fields for different CDC consumers. Each CDC consumer has its own Kafka topic and gets its own CDC messages so that each CDC consumer gets only the data changes that are relevant for it.
Fabric CDC also integrates with the Elasticsearch search provider, enabling cross-instance search capabilities.
For example:
The following HL flow describes the CDC flow and the population of CDC data in consumers:
Each CDC message is sent to Kafka. Each CDC consumer has its own Kafka topic.
The Fabric CDC_TRANSACTION_CONSUMER job consumes the search topic from Kafka and updates Elasticsearch. Other consumers must create their own consumer processes to consume Kafka CDC messages.
Note that the publication of CDC changes must be predefined in the Fabric Studio. When defining an LU in the Fabric Studio, selected LU table columns can be set to publish CDC messages each time they are updated.
To publish CDC columns to CDC consumers, LUs with CDC indexes must be deployed to Fabric:
The Fabric Change Data Capture (CDC) solution can notify external systems about data changes published via Kafka. Fabric CDC can send different data segments to different CDC consumers and different LU table columns can be defined as CDC fields for different CDC consumers. Each CDC consumer has its own Kafka topic and gets its own CDC messages so that each CDC consumer gets only the data changes that are relevant for it.
Fabric CDC also integrates with the Elasticsearch search provider, enabling cross-instance search capabilities.
For example:
The following HL flow describes the CDC flow and the population of CDC data in consumers:
Each CDC message is sent to Kafka. Each CDC consumer has its own Kafka topic.
The Fabric CDC_TRANSACTION_CONSUMER job consumes the search topic from Kafka and updates Elasticsearch. Other consumers must create their own consumer processes to consume Kafka CDC messages.
Note that the publication of CDC changes must be predefined in the Fabric Studio. When defining an LU in the Fabric Studio, selected LU table columns can be set to publish CDC messages each time they are updated.
To publish CDC columns to CDC consumers, LUs with CDC indexes must be deployed to Fabric: