Fabric's Change Data Capture (CDC) solution notifies external systems about data changes published via Kafka and also offers cross-instance Search capabilities through its built-in integration with Elasticsearch.
When defining LU in the Fabric Studio, selected tables and columns can be set to publish CDC messages each time they are updated. For example, to notify an external consumer system about a customer's change of address the following columns are defined as CDC columns in the ADDRESS table in the CUSTOMER LU:
Note that Fabric Studio does not enable defining more than 63 columns in a CDC field in the same LU table, assuming that all columns are positioned according to 1 to 63 in the LU table.
A specific CDC message is generated for each type of change in the CDC column.
To control whether a CDC message is published to external systems, do either:
Fabric has the following CDC messages:
CDC Message | Description |
CDC Schema |
Generated either when an LU is deployed to Fabric for the first time, or when executing the CDC_REPUBLISH_SCHEMA Fabric command. These messages hold the LU Schema's name and information about its CDC tables. See the example in Appendix A. |
CDC Schema Update |
Generated by either one of the following scenarios:
These messages hold the LU Schema's name and updates on the affected CDC LU tables. See the examples in Appendix B. |
CDC Delete Tables |
Generated by either one of the following scenarios:
See the example in Appendix C. |
CDC Table Change Info |
Generated by either one of the following scenarios:
These messages hold a list of PK columns in LU tables and relevant changes in CDC columns. See the examples in Appendix D. |
Note that you can find samples of CDC messages under $K2_HOME/fabric/samples/cdc-messages directory of Fabric server.
CDC messages are stored in Kafka topics by their CDC Consumer Name as defined in the Fabric Studio and can be serialized or deserialized using the com.k2view.fabric.cdc.Serialization Fabric Java Class.
The Serialization class offers two methods:
To view the list of Fabric APIs, click http://[Fabric IP address]:3213/static/doc/user-api/index.html.
{
"info":{
"trxId":"6e9ace70-6e20-461d-966a-8f1c88fbf91c",
"ts":1587369930093,
"msgNo":1,
"msgCount":1
},
"luName":"CTN",
"dataType":"SCHEMA",
"schema":{
"name":"CTN",
"tables":[
{
"name":"CTNMAININFO",
"columns":[
{
"name":"ACCOUNT_STATUS",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"MAIL",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"ACCOUNT_NUMBER",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"CTN",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"ACCOUNT_TYPE",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"IP_ADDRESS",
"type":"TEXT",
"tags":[
""
],
"isPk":false
}
]
},
{
"name":"SUBSCRIBERADDRESS",
"columns":[
{
"name":"ID_NUMBER",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"CITY",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"STREET_NO",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"APARTMENT_NO",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"POSTAL_CODE",
"type":"TEXT",
"tags":[
""
],
"isPk":false
},
{
"name":"MAIL",
"type":"TEXT",
"tags":[
""
],
"isPk":false
},
{
"name":"STREET",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
}
]
}
]
}
}
{
"info":{
"trxId":"f0626955-2aee-4e83-93d8-d657edc12283",
"ts":1587370059952,
"msgNo":1,
"msgCount":1
},
"luName":"CTN",
"dataType":"SCHEMA_UPDATE",
"schemaUpdate":{
"name":"CTN",
"tables":[
{
"name":"CTNMAININFO",
"updateType":"UPDATED",
"columns":[
{
"name":"IP_ADDRESS",
"type":"TEXT",
"tags":[
""
],
"updateType":"REMOVED"
}
]
},
{
"name":"T1",
"updateType":"UPDATED",
"columns":[
{
"name":"F2",
"type":"TEXT",
"tags":[
""
],
"updateType":"REMOVED"
},
{
"name":"F3",
"type":"TEXT",
"tags":[
""
],
"updateType":"REMOVED"
},
{
"name":"F4",
"type":"TEXT",
"tags":[
""
],
"updateType":"REMOVED"
}
]
}
]
}
}
{
"info":{
"trxId":"5a96898d-a9df-42c0-be66-81bba9339746",
"ts":1587369528918,
"msgNo":1,
"msgCount":1
},
"luName":"CTN",
"dataType":"SCHEMA_UPDATE",
"schemaUpdate":{
"name":"CTN",
"tables":[
{
"name":"CTNACCOUNTBALANCE",
"updateType":"REMOVED",
"columns":[
]
},
{
"name":"CTNMAININFO",
"updateType":"REMOVED",
"columns":[
]
},
{
"name":"SUBSCRIBERADDRESS",
"updateType":"REMOVED",
"columns":[
]
},
{
"name":"T1",
"updateType":"REMOVED",
"columns":[
]
}
]
}
}
{
"info":{
"trxId":"db7cea26-458d-4477-a2e7-6c67fd1d8781",
"ts":1587369340824,
"msgNo":23,
"msgCount":24
},
"luName":"CTN",
"iid":"1",
"dataType":"DELETE_TABLES",
"deleteTables":{
"tables":[
"T1", “T2”, “T3”
]
}
}
{
"info":{
"trxId":"db198b31-736c-4566-aac7-a0891055bbf1",
"ts":1587369152081,
"msgNo":5,
"msgCount":5
},
"luName":"CTN",
"iid":"4",
"dataType":"DATA_CHANGE",
"tblChange":{
"pk":[
],
"dataChange":{
"type":"INSERT",
"table":"SUBSCRIBERADDRESS",
"names":[
"STREET",
"STREET_NO",
"APARTMENT_NO",
"FIRST_NAME",
"MAIL"
],
"oldValues":[
],
"newValues":[
"Straco",
"412",
"69",
"EINHORN",
"ovAMthQr@gmail.com"
]
}
}
}
{
"info":{
"trxId":"db7cea26-458d-4477-a2e7-6c67fd1d8781",
"ts":1587369340825,
"msgNo":24,
"msgCount":24
},
"luName":"CTN",
"iid":"1",
"dataType":"DATA_REPUBLISH",
"tblChange":{
"pk":[
],
"dataChange":{
"type":"INSERT",
"table":"T1",
"names":[
"F1",
"NOW"
],
"oldValues":[
],
"newValues":[
"1",
"2020/04/16 16:11:43:399"
]
}
}
}
Fabric's Change Data Capture (CDC) solution notifies external systems about data changes published via Kafka and also offers cross-instance Search capabilities through its built-in integration with Elasticsearch.
When defining LU in the Fabric Studio, selected tables and columns can be set to publish CDC messages each time they are updated. For example, to notify an external consumer system about a customer's change of address the following columns are defined as CDC columns in the ADDRESS table in the CUSTOMER LU:
Note that Fabric Studio does not enable defining more than 63 columns in a CDC field in the same LU table, assuming that all columns are positioned according to 1 to 63 in the LU table.
A specific CDC message is generated for each type of change in the CDC column.
To control whether a CDC message is published to external systems, do either:
Fabric has the following CDC messages:
CDC Message | Description |
CDC Schema |
Generated either when an LU is deployed to Fabric for the first time, or when executing the CDC_REPUBLISH_SCHEMA Fabric command. These messages hold the LU Schema's name and information about its CDC tables. See the example in Appendix A. |
CDC Schema Update |
Generated by either one of the following scenarios:
These messages hold the LU Schema's name and updates on the affected CDC LU tables. See the examples in Appendix B. |
CDC Delete Tables |
Generated by either one of the following scenarios:
See the example in Appendix C. |
CDC Table Change Info |
Generated by either one of the following scenarios:
These messages hold a list of PK columns in LU tables and relevant changes in CDC columns. See the examples in Appendix D. |
Note that you can find samples of CDC messages under $K2_HOME/fabric/samples/cdc-messages directory of Fabric server.
CDC messages are stored in Kafka topics by their CDC Consumer Name as defined in the Fabric Studio and can be serialized or deserialized using the com.k2view.fabric.cdc.Serialization Fabric Java Class.
The Serialization class offers two methods:
To view the list of Fabric APIs, click http://[Fabric IP address]:3213/static/doc/user-api/index.html.
{
"info":{
"trxId":"6e9ace70-6e20-461d-966a-8f1c88fbf91c",
"ts":1587369930093,
"msgNo":1,
"msgCount":1
},
"luName":"CTN",
"dataType":"SCHEMA",
"schema":{
"name":"CTN",
"tables":[
{
"name":"CTNMAININFO",
"columns":[
{
"name":"ACCOUNT_STATUS",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"MAIL",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"ACCOUNT_NUMBER",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"CTN",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"ACCOUNT_TYPE",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"IP_ADDRESS",
"type":"TEXT",
"tags":[
""
],
"isPk":false
}
]
},
{
"name":"SUBSCRIBERADDRESS",
"columns":[
{
"name":"ID_NUMBER",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"CITY",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"STREET_NO",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"APARTMENT_NO",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
},
{
"name":"POSTAL_CODE",
"type":"TEXT",
"tags":[
""
],
"isPk":false
},
{
"name":"MAIL",
"type":"TEXT",
"tags":[
""
],
"isPk":false
},
{
"name":"STREET",
"type":"TEXT",
"tags":[
"data"
],
"isPk":false
}
]
}
]
}
}
{
"info":{
"trxId":"f0626955-2aee-4e83-93d8-d657edc12283",
"ts":1587370059952,
"msgNo":1,
"msgCount":1
},
"luName":"CTN",
"dataType":"SCHEMA_UPDATE",
"schemaUpdate":{
"name":"CTN",
"tables":[
{
"name":"CTNMAININFO",
"updateType":"UPDATED",
"columns":[
{
"name":"IP_ADDRESS",
"type":"TEXT",
"tags":[
""
],
"updateType":"REMOVED"
}
]
},
{
"name":"T1",
"updateType":"UPDATED",
"columns":[
{
"name":"F2",
"type":"TEXT",
"tags":[
""
],
"updateType":"REMOVED"
},
{
"name":"F3",
"type":"TEXT",
"tags":[
""
],
"updateType":"REMOVED"
},
{
"name":"F4",
"type":"TEXT",
"tags":[
""
],
"updateType":"REMOVED"
}
]
}
]
}
}
{
"info":{
"trxId":"5a96898d-a9df-42c0-be66-81bba9339746",
"ts":1587369528918,
"msgNo":1,
"msgCount":1
},
"luName":"CTN",
"dataType":"SCHEMA_UPDATE",
"schemaUpdate":{
"name":"CTN",
"tables":[
{
"name":"CTNACCOUNTBALANCE",
"updateType":"REMOVED",
"columns":[
]
},
{
"name":"CTNMAININFO",
"updateType":"REMOVED",
"columns":[
]
},
{
"name":"SUBSCRIBERADDRESS",
"updateType":"REMOVED",
"columns":[
]
},
{
"name":"T1",
"updateType":"REMOVED",
"columns":[
]
}
]
}
}
{
"info":{
"trxId":"db7cea26-458d-4477-a2e7-6c67fd1d8781",
"ts":1587369340824,
"msgNo":23,
"msgCount":24
},
"luName":"CTN",
"iid":"1",
"dataType":"DELETE_TABLES",
"deleteTables":{
"tables":[
"T1", “T2”, “T3”
]
}
}
{
"info":{
"trxId":"db198b31-736c-4566-aac7-a0891055bbf1",
"ts":1587369152081,
"msgNo":5,
"msgCount":5
},
"luName":"CTN",
"iid":"4",
"dataType":"DATA_CHANGE",
"tblChange":{
"pk":[
],
"dataChange":{
"type":"INSERT",
"table":"SUBSCRIBERADDRESS",
"names":[
"STREET",
"STREET_NO",
"APARTMENT_NO",
"FIRST_NAME",
"MAIL"
],
"oldValues":[
],
"newValues":[
"Straco",
"412",
"69",
"EINHORN",
"ovAMthQr@gmail.com"
]
}
}
}
{
"info":{
"trxId":"db7cea26-458d-4477-a2e7-6c67fd1d8781",
"ts":1587369340825,
"msgNo":24,
"msgCount":24
},
"luName":"CTN",
"iid":"1",
"dataType":"DATA_REPUBLISH",
"tblChange":{
"pk":[
],
"dataChange":{
"type":"INSERT",
"table":"T1",
"names":[
"F1",
"NOW"
],
"oldValues":[
],
"newValues":[
"1",
"2020/04/16 16:11:43:399"
]
}
}
}