CDC Messages

Fabric's Change Data Capture (CDC) solution notifies external systems about data changes published via Kafka and also offers cross-instance Search capabilities through its built-in integration with Elasticsearch.

When defining LU in the Fabric Studio, selected tables and columns can be set to publish CDC messages each time they are updated. For example, to notify an external consumer system about a customer's change of address the following columns are defined as CDC columns in the ADDRESS table in the CUSTOMER LU:

• STATE
• CITY
• STREET
• HOUSE_NO
• ZIP_CODE

Note that Fabric Studio does not enable defining more than 63 columns in a CDC field in the same LU table, assuming that all columns are positioned according to 1 to 63 in the LU table.

A specific CDC message is generated for each type of change in the CDC column.

Controlling Publication of CDC Messages

To control whether a CDC message is published to external systems, do either:

• Update the config.ini. Set the CDC_PUBLISH_MODE parameter to:
  • ON (default), when implemented, publish the CDC message.
  • OFF, never publish the CDC message.
  • IF_SETUP, (Studio Debug default) when implemented, publish the CDC message only if the CDC publisher has been configured in the config.ini file.
• Run the SET CDC_PUBLISH Fabric command to set the CDC mode on a session level. The values of the CDC_PUBLISH are true (enabling CDC publication) or false (disabling CDC publication). The default value is true.

CDC Message Types

Fabric has the following CDC messages:

Note that you can find samples of CDC messages under $K2_HOME/fabric/samples/cdc-messages directory of Fabric server.

Serialization

CDC messages are stored in Kafka topics by their CDC Consumer Name as defined in the Fabric Studio and can be serialized or deserialized using the com.k2view.fabric.cdc.Serialization Fabric Java Class.

The Serialization class offers two methods:

To view the list of Fabric APIs, click http://[Fabric IP address]:3213/static/doc/user-api/index.html.

Appendices

Appendix A: CDC Schema Message - Example

{
  "info":{
    "trxId":"6e9ace70-6e20-461d-966a-8f1c88fbf91c",
    "ts":1587369930093,
    "msgNo":1,
    "msgCount":1
  },
  "luName":"CTN",
  "dataType":"SCHEMA",
  "schema":{
    "name":"CTN",
    "tables":[
     {
       "name":"CTNMAININFO",
       "columns":[
        {
          "name":"ACCOUNT_STATUS",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"MAIL",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"ACCOUNT_NUMBER",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"CTN",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"ACCOUNT_TYPE",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"IP_ADDRESS",
          "type":"TEXT",
          "tags":[
           ""
          ],
          "isPk":false
        }
       ]
     },
     {
       "name":"SUBSCRIBERADDRESS",
       "columns":[
        {
          "name":"ID_NUMBER",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"CITY",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"STREET_NO",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"APARTMENT_NO",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        },
        {
          "name":"POSTAL_CODE",
          "type":"TEXT",
          "tags":[
           ""
          ],
          "isPk":false
        },
        {
          "name":"MAIL",
          "type":"TEXT",
          "tags":[
           ""
          ],
          "isPk":false
        },
        {
          "name":"STREET",
          "type":"TEXT",
          "tags":[
           "data"
          ],
          "isPk":false
        }
       ]
     }
    ]
  }
 }

Appendix B: CDC Schema Update Message - Example

Schema Update

{
   "info":{
      "trxId":"f0626955-2aee-4e83-93d8-d657edc12283",
      "ts":1587370059952,
      "msgNo":1,
      "msgCount":1
   },
   "luName":"CTN",
   "dataType":"SCHEMA_UPDATE",
   "schemaUpdate":{
      "name":"CTN",
      "tables":[
         {
            "name":"CTNMAININFO",
            "updateType":"UPDATED",
            "columns":[
               {
                  "name":"IP_ADDRESS",
                  "type":"TEXT",
                  "tags":[
                     ""
                  ],
                  "updateType":"REMOVED"
               }
            ]
         },
         {
            "name":"T1",
            "updateType":"UPDATED",
            "columns":[
               {
                  "name":"F2",
                  "type":"TEXT",
                  "tags":[
                     ""
                  ],
                  "updateType":"REMOVED"
               },
               {
                  "name":"F3",
                  "type":"TEXT",
                  "tags":[
                     ""
                  ],
                  "updateType":"REMOVED"
               },
               {
                  "name":"F4",
                  "type":"TEXT",
                  "tags":[
                     ""
                  ],
                  "updateType":"REMOVED"
               }
            ]
         }
      ]
   }
}

Schema Drop

{
  "info":{
    "trxId":"5a96898d-a9df-42c0-be66-81bba9339746",
    "ts":1587369528918,
    "msgNo":1,
    "msgCount":1
  },
  "luName":"CTN",
  "dataType":"SCHEMA_UPDATE",
  "schemaUpdate":{
    "name":"CTN",
    "tables":[
      {
        "name":"CTNACCOUNTBALANCE",
        "updateType":"REMOVED",
        "columns":[

        ]
      },
      {
        "name":"CTNMAININFO",
        "updateType":"REMOVED",
        "columns":[

        ]
      },
      {
        "name":"SUBSCRIBERADDRESS",
        "updateType":"REMOVED",
        "columns":[

        ]
      },
      {
        "name":"T1",
        "updateType":"REMOVED",
        "columns":[

        ]
      }
    ]
  }
}

Appendix C: CDC Delete Tables Message - Example

{
   "info":{
      "trxId":"db7cea26-458d-4477-a2e7-6c67fd1d8781",
      "ts":1587369340824,
      "msgNo":23,
      "msgCount":24
   },
   "luName":"CTN",
   "iid":"1",
   "dataType":"DELETE_TABLES",
   "deleteTables":{
      "tables":[
         "T1", “T2”, “T3”
      ]
   }
}

Appendix D: CDC Table Change Info Message - Example

Data Change

{
   "info":{
      "trxId":"db198b31-736c-4566-aac7-a0891055bbf1",
      "ts":1587369152081,
      "msgNo":5,
      "msgCount":5
   },
   "luName":"CTN",
   "iid":"4",
   "dataType":"DATA_CHANGE",
   "tblChange":{
      "pk":[

      ],
      "dataChange":{
         "type":"INSERT",
         "table":"SUBSCRIBERADDRESS",
         "names":[
            "STREET",
            "STREET_NO",
            "APARTMENT_NO",
            "FIRST_NAME",
            "MAIL"
         ],
         "oldValues":[

         ],
         "newValues":[
            "Straco",
            "412",
            "69",
            "EINHORN",
            "ovAMthQr@gmail.com"
         ]
      }
   }
}

Data Republish

{
  "info":{
    "trxId":"db7cea26-458d-4477-a2e7-6c67fd1d8781",
    "ts":1587369340825,
    "msgNo":24,
    "msgCount":24
  },
  "luName":"CTN",
  "iid":"1",
  "dataType":"DATA_REPUBLISH",
  "tblChange":{
    "pk":[

    ],
    "dataChange":{
      "type":"INSERT",
      "table":"T1",
      "names":[
        "F1",
        "NOW"
      ],
      "oldValues":[

      ],
      "newValues":[
        "1",
        "2020/04/16 16:11:43:399"
      ]
    }
  }
}