CDC Messages

Fabric's Change Data Capture (CDC) solution notifies external systems about data changes published via Kafka and also offers cross-instance Search capabilities through its built-in integration with Elasticsearch.

When defining LU in the Fabric Studio, selected tables and columns can be set to publish CDC messages each time they are updated. For example, to notify an external consumer system about a customer's change of address the following columns are defined as CDC columns in the ADDRESS table in the CUSTOMER LU:

  • STATE
  • CITY
  • STREET
  • HOUSE_NO
  • ZIP_CODE

Note that Fabric Studio does not enable defining more than 63 columns in a CDC field in the same LU table, assuming that all columns are positioned according to 1 to 63 in the LU table.

A specific CDC message is generated for each type of change in the CDC column.

Controlling Publication of CDC Messages

To control whether a CDC message is published to external systems, do either:

  • Update the config.ini. Set the CDC_PUBLISH_MODE parameter to:
    • ON (default), when implemented, publish the CDC message.
    • OFF, never publish the CDC message.
    • IF_SETUP, (Studio Debug default) when implemented, publish the CDC message only if the CDC publisher has been configured in the config.ini file.
  • Run the SET CDC_PUBLISH Fabric command to set the CDC mode on a session level. The values of the CDC_PUBLISH are true (enabling CDC publication) or false (disabling CDC publication). The default value is true.

CDC Message Types

Fabric has the following CDC messages:

CDC Message Description

CDC Schema

Generated either when an LU is deployed to Fabric for the first time, or when executing the CDC_REPUBLISH_SCHEMA Fabric command. These messages hold the LU Schema's name and information about its CDC tables. 

See the example in Appendix A.

CDC Schema Update

Generated by either one of the following scenarios:

These messages hold the LU Schema's name and updates on the affected CDC LU tables.

See the examples in Appendix B.

CDC Delete Tables

Generated by either one of the following scenarios:

  • Delete LUI command.
  • Run CDC_REPUBLISH_INSTANCE Fabric command to republish the CDC data of a given LUI. The CDC data of the LU tables of the republished LUI must be recreated by the CDC consumer.

See the example in Appendix C.

CDC Table Change Info

Generated by either one of the following scenarios:

  • The MicroDB is updated either by INSERT, UPDATE or DELETE statements.
  • Run the CDC_REPUBLISH_INSTANCE Fabric command to republish the CDC data of a given LUI.

These messages hold a list of PK columns in LU tables and relevant changes in CDC columns.

See the examples in Appendix D.

Note that you can find samples of CDC messages under $K2_HOME/fabric/samples/cdc-messages directory of Fabric server.

Serialization

CDC messages are stored in Kafka topics by their CDC Consumer Name as defined in the Fabric Studio and can be serialized or deserialized using the com.k2view.fabric.cdc.Serialization Fabric Java Class.

The Serialization class offers two methods:

Method

Description

public static CdcMessage fromJson(String msg)

Creates a CdcMessage object from an input message.

public static String toJson(CdcMessage msg)

Serializes the CDC message to JSON format.

To view the list of Fabric APIs, click http://[Fabric IP address]:3213/static/doc/user-api/index.html.

Appendices

Appendix A: CDC Schema Message - Example

{
  "info":{
    "trxId":"6e9ace70-6e20-461d-966a-8f1c88fbf91c",
    "ts":1587369930093,
    "msgNo":1,
    "msgCount":1
  },
  "luName":"CTN",
  "dataType":"SCHEMA",
  "schema":{
    "name":"CTN",
    "tables":[
     {
       "name":"CTNMAININFO",
       "columns":[
        {
          "name":"ACCOUNT_STATUS",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"MAIL",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"ACCOUNT_NUMBER",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"CTN",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"ACCOUNT_TYPE",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"IP_ADDRESS",
          "type":"TEXT",
          "tags":[
           ""
          ],
          "isPk":false
        }
       ]
     },
     {
       "name":"SUBSCRIBERADDRESS",
       "columns":[
        {
          "name":"ID_NUMBER",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"CITY",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"STREET_NO",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"APARTMENT_NO",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"POSTAL_CODE",
          "type":"TEXT",
          "tags":[
           ""
          ],
          "isPk":false
        },
        {
          "name":"MAIL",
          "type":"TEXT",
          "tags":[
           ""
          ],
          "isPk":false
        },
        {
          "name":"STREET",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        }
       ]
     }
    ]
  }
 }

Appendix B: CDC Schema Update Message - Example

Schema Update

{
   "info":{
      "trxId":"f0626955-2aee-4e83-93d8-d657edc12283",
      "ts":1587370059952,
      "msgNo":1,
      "msgCount":1
   },
   "luName":"CTN",
   "dataType":"SCHEMA_UPDATE",
   "schemaUpdate":{
      "name":"CTN",
      "tables":[
         {
            "name":"CTNMAININFO",
            "updateType":"UPDATED",
            "columns":[
               {
                  "name":"IP_ADDRESS",
                  "type":"TEXT",
                  "tags":[
                     ""
                  ],
                  "updateType":"REMOVED"
               }
            ]
         },
         {
            "name":"T1",
            "updateType":"UPDATED",
            "columns":[
               {
                  "name":"F2",
                  "type":"TEXT",
                  "tags":[
                     ""
                  ],
                  "updateType":"REMOVED"
               },
               {
                  "name":"F3",
                  "type":"TEXT",
                  "tags":[
                     ""
                  ],
                  "updateType":"REMOVED"
               },
               {
                  "name":"F4",
                  "type":"TEXT",
                  "tags":[
                     ""
                  ],
                  "updateType":"REMOVED"
               }
            ]
         }
      ]
   }
}

Schema Drop

{
  "info":{
    "trxId":"5a96898d-a9df-42c0-be66-81bba9339746",
    "ts":1587369528918,
    "msgNo":1,
    "msgCount":1
  },
  "luName":"CTN",
  "dataType":"SCHEMA_UPDATE",
  "schemaUpdate":{
    "name":"CTN",
    "tables":[
      {
        "name":"CTNACCOUNTBALANCE",
        "updateType":"REMOVED",
        "columns":[

        ]
      },
      {
        "name":"CTNMAININFO",
        "updateType":"REMOVED",
        "columns":[

        ]
      },
      {
        "name":"SUBSCRIBERADDRESS",
        "updateType":"REMOVED",
        "columns":[

        ]
      },
      {
        "name":"T1",
        "updateType":"REMOVED",
        "columns":[

        ]
      }
    ]
  }
}

Appendix C: CDC Delete Tables Message - Example

{
   "info":{
      "trxId":"db7cea26-458d-4477-a2e7-6c67fd1d8781",
      "ts":1587369340824,
      "msgNo":23,
      "msgCount":24
   },
   "luName":"CTN",
   "iid":"1",
   "dataType":"DELETE_TABLES",
   "deleteTables":{
      "tables":[
         "T1", “T2”, “T3”
      ]
   }
}

Appendix D: CDC Table Change Info Message - Example

Data Change

{
   "info":{
      "trxId":"db198b31-736c-4566-aac7-a0891055bbf1",
      "ts":1587369152081,
      "msgNo":5,
      "msgCount":5
   },
   "luName":"CTN",
   "iid":"4",
   "dataType":"DATA_CHANGE",
   "tblChange":{
      "pk":[

      ],
      "dataChange":{
         "type":"INSERT",
         "table":"SUBSCRIBERADDRESS",
         "names":[
            "STREET",
            "STREET_NO",
            "APARTMENT_NO",
            "FIRST_NAME",
            "MAIL"
         ],
         "oldValues":[

         ],
         "newValues":[
            "Straco",
            "412",
            "69",
            "EINHORN",
            "ovAMthQr@gmail.com"
         ]
      }
   }
}

Data Republish

{
  "info":{
    "trxId":"db7cea26-458d-4477-a2e7-6c67fd1d8781",
    "ts":1587369340825,
    "msgNo":24,
    "msgCount":24
  },
  "luName":"CTN",
  "iid":"1",
  "dataType":"DATA_REPUBLISH",
  "tblChange":{
    "pk":[

    ],
    "dataChange":{
      "type":"INSERT",
      "table":"T1",
      "names":[
        "F1",
        "NOW"
      ],
      "oldValues":[

      ],
      "newValues":[
        "1",
        "2020/04/16 16:11:43:399"
      ]
    }
  }
}

Previous

CDC Messages

Fabric's Change Data Capture (CDC) solution notifies external systems about data changes published via Kafka and also offers cross-instance Search capabilities through its built-in integration with Elasticsearch.

When defining LU in the Fabric Studio, selected tables and columns can be set to publish CDC messages each time they are updated. For example, to notify an external consumer system about a customer's change of address the following columns are defined as CDC columns in the ADDRESS table in the CUSTOMER LU:

  • STATE
  • CITY
  • STREET
  • HOUSE_NO
  • ZIP_CODE

Note that Fabric Studio does not enable defining more than 63 columns in a CDC field in the same LU table, assuming that all columns are positioned according to 1 to 63 in the LU table.

A specific CDC message is generated for each type of change in the CDC column.

Controlling Publication of CDC Messages

To control whether a CDC message is published to external systems, do either:

  • Update the config.ini. Set the CDC_PUBLISH_MODE parameter to:
    • ON (default), when implemented, publish the CDC message.
    • OFF, never publish the CDC message.
    • IF_SETUP, (Studio Debug default) when implemented, publish the CDC message only if the CDC publisher has been configured in the config.ini file.
  • Run the SET CDC_PUBLISH Fabric command to set the CDC mode on a session level. The values of the CDC_PUBLISH are true (enabling CDC publication) or false (disabling CDC publication). The default value is true.

CDC Message Types

Fabric has the following CDC messages:

CDC Message Description

CDC Schema

Generated either when an LU is deployed to Fabric for the first time, or when executing the CDC_REPUBLISH_SCHEMA Fabric command. These messages hold the LU Schema's name and information about its CDC tables. 

See the example in Appendix A.

CDC Schema Update

Generated by either one of the following scenarios:

These messages hold the LU Schema's name and updates on the affected CDC LU tables.

See the examples in Appendix B.

CDC Delete Tables

Generated by either one of the following scenarios:

  • Delete LUI command.
  • Run CDC_REPUBLISH_INSTANCE Fabric command to republish the CDC data of a given LUI. The CDC data of the LU tables of the republished LUI must be recreated by the CDC consumer.

See the example in Appendix C.

CDC Table Change Info

Generated by either one of the following scenarios:

  • The MicroDB is updated either by INSERT, UPDATE or DELETE statements.
  • Run the CDC_REPUBLISH_INSTANCE Fabric command to republish the CDC data of a given LUI.

These messages hold a list of PK columns in LU tables and relevant changes in CDC columns.

See the examples in Appendix D.

Note that you can find samples of CDC messages under $K2_HOME/fabric/samples/cdc-messages directory of Fabric server.

Serialization

CDC messages are stored in Kafka topics by their CDC Consumer Name as defined in the Fabric Studio and can be serialized or deserialized using the com.k2view.fabric.cdc.Serialization Fabric Java Class.

The Serialization class offers two methods:

Method

Description

public static CdcMessage fromJson(String msg)

Creates a CdcMessage object from an input message.

public static String toJson(CdcMessage msg)

Serializes the CDC message to JSON format.

To view the list of Fabric APIs, click http://[Fabric IP address]:3213/static/doc/user-api/index.html.

Appendices

Appendix A: CDC Schema Message - Example

{
  "info":{
    "trxId":"6e9ace70-6e20-461d-966a-8f1c88fbf91c",
    "ts":1587369930093,
    "msgNo":1,
    "msgCount":1
  },
  "luName":"CTN",
  "dataType":"SCHEMA",
  "schema":{
    "name":"CTN",
    "tables":[
     {
       "name":"CTNMAININFO",
       "columns":[
        {
          "name":"ACCOUNT_STATUS",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"MAIL",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"ACCOUNT_NUMBER",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"CTN",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"ACCOUNT_TYPE",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"IP_ADDRESS",
          "type":"TEXT",
          "tags":[
           ""
          ],
          "isPk":false
        }
       ]
     },
     {
       "name":"SUBSCRIBERADDRESS",
       "columns":[
        {
          "name":"ID_NUMBER",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"CITY",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"STREET_NO",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"APARTMENT_NO",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"POSTAL_CODE",
          "type":"TEXT",
          "tags":[
           ""
          ],
          "isPk":false
        },
        {
          "name":"MAIL",
          "type":"TEXT",
          "tags":[
           ""
          ],
          "isPk":false
        },
        {
          "name":"STREET",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        }
       ]
     }
    ]
  }
 }

Appendix B: CDC Schema Update Message - Example

Schema Update

{
   "info":{
      "trxId":"f0626955-2aee-4e83-93d8-d657edc12283",
      "ts":1587370059952,
      "msgNo":1,
      "msgCount":1
   },
   "luName":"CTN",
   "dataType":"SCHEMA_UPDATE",
   "schemaUpdate":{
      "name":"CTN",
      "tables":[
         {
            "name":"CTNMAININFO",
            "updateType":"UPDATED",
            "columns":[
               {
                  "name":"IP_ADDRESS",
                  "type":"TEXT",
                  "tags":[
                     ""
                  ],
                  "updateType":"REMOVED"
               }
            ]
         },
         {
            "name":"T1",
            "updateType":"UPDATED",
            "columns":[
               {
                  "name":"F2",
                  "type":"TEXT",
                  "tags":[
                     ""
                  ],
                  "updateType":"REMOVED"
               },
               {
                  "name":"F3",
                  "type":"TEXT",
                  "tags":[
                     ""
                  ],
                  "updateType":"REMOVED"
               },
               {
                  "name":"F4",
                  "type":"TEXT",
                  "tags":[
                     ""
                  ],
                  "updateType":"REMOVED"
               }
            ]
         }
      ]
   }
}

Schema Drop

{
  "info":{
    "trxId":"5a96898d-a9df-42c0-be66-81bba9339746",
    "ts":1587369528918,
    "msgNo":1,
    "msgCount":1
  },
  "luName":"CTN",
  "dataType":"SCHEMA_UPDATE",
  "schemaUpdate":{
    "name":"CTN",
    "tables":[
      {
        "name":"CTNACCOUNTBALANCE",
        "updateType":"REMOVED",
        "columns":[

        ]
      },
      {
        "name":"CTNMAININFO",
        "updateType":"REMOVED",
        "columns":[

        ]
      },
      {
        "name":"SUBSCRIBERADDRESS",
        "updateType":"REMOVED",
        "columns":[

        ]
      },
      {
        "name":"T1",
        "updateType":"REMOVED",
        "columns":[

        ]
      }
    ]
  }
}

Appendix C: CDC Delete Tables Message - Example

{
   "info":{
      "trxId":"db7cea26-458d-4477-a2e7-6c67fd1d8781",
      "ts":1587369340824,
      "msgNo":23,
      "msgCount":24
   },
   "luName":"CTN",
   "iid":"1",
   "dataType":"DELETE_TABLES",
   "deleteTables":{
      "tables":[
         "T1", “T2”, “T3”
      ]
   }
}

Appendix D: CDC Table Change Info Message - Example

Data Change

{
   "info":{
      "trxId":"db198b31-736c-4566-aac7-a0891055bbf1",
      "ts":1587369152081,
      "msgNo":5,
      "msgCount":5
   },
   "luName":"CTN",
   "iid":"4",
   "dataType":"DATA_CHANGE",
   "tblChange":{
      "pk":[

      ],
      "dataChange":{
         "type":"INSERT",
         "table":"SUBSCRIBERADDRESS",
         "names":[
            "STREET",
            "STREET_NO",
            "APARTMENT_NO",
            "FIRST_NAME",
            "MAIL"
         ],
         "oldValues":[

         ],
         "newValues":[
            "Straco",
            "412",
            "69",
            "EINHORN",
            "ovAMthQr@gmail.com"
         ]
      }
   }
}

Data Republish

{
  "info":{
    "trxId":"db7cea26-458d-4477-a2e7-6c67fd1d8781",
    "ts":1587369340825,
    "msgNo":24,
    "msgCount":24
  },
  "luName":"CTN",
  "iid":"1",
  "dataType":"DATA_REPUBLISH",
  "tblChange":{
    "pk":[

    ],
    "dataChange":{
      "type":"INSERT",
      "table":"T1",
      "names":[
        "F1",
        "NOW"
      ],
      "oldValues":[

      ],
      "newValues":[
        "1",
        "2020/04/16 16:11:43:399"
      ]
    }
  }
}

Previous