Knowledge Base Academy Community My Tickets
Fabric 8.1
Fabric 8.1 Fabric 8.0 Fabric 7.2 Fabric 7.1 Fabric 7.0 Fabric 6.5 Fabric 6.4

Fabric System Database

Fabric System Database is used by Fabric internal processes to monitor, secure, control, configure, audit and operate the application.

System Database Types

Fabric supports several types of Databases as the System Database, as described below.

NoSQL distributed database, such as Cassandra DB

  • Pros:
    • Scalable
    • Distributed
    • Built-in TTL mechanism on row level
    • If Cassandra is used as a MicroDB storage, there is no need to introduce additional DBs
    • Managed services (such as AWS Keyspaces or Astra) are supported
    • Supported by the iidFinder solution
    • Built-in mechanism for managing parallel threads during bulk instance loading
  • Cons:
    • Consistency
    • Not easy to operate and maintain

Relational database, such as PostgreSQL

  • Pros:
    • Consistency
    • In case of the TDM solution, PostgreSQL is already introduced
    • Compliance with services such as Cloud Spanner, AlloyDB
    • Easy to maintain
  • Cons:
    • Single point of failure
    • Not supported by the iidFinder solution

SQLite

  • Pros:
    • Development and single-node environments

Fabric creates several keyspaces or schemas (in case of SQLite or PostgreSQL) for its operation, where each starts with the k2 prefix.

Each deployed LU creates an additional k2view_[LU Name] keyspace or schema, such as k2view_customer.

Notes:

  • When set in the node.id configuration file, the cluster_id is concatenated to each keyspace or schema name. For example, if the cluster_id is set to crm1, the created keyspace or schema is k2view_customer_crm1.
  • When a Fabric project opens in the Fabric Studio, it concatenates the project name to the schema name.
  • A k2view_k2_ws keyspace or schema is created for the deployed WS.
  • When Fabric is configured to run a System DB type other than Cassandra - an internal product job runs, scans the tables to be cleaned and cleans them, using the TTL concept. The definition of each table's TTL value is tracked in the k2_table_level_ttl table.

Fabric uses the Cassandra DB as its default system management database.

When working with a non-Cassandra System DB, like SQLite or PostgreSQL, the following should be set in config.ini:

  • Populate the parameter values included in [system_db] section, which was introduced in Fabric V7.2. If you use Cassandra as the System DB, then you can ignore this section.

  • Set the SERVER_AUTHENTICATOR config parameter's value to be 'fabric' (default value is 'cassandra'). For more information about SERVER_AUTHENTICATOR config options, read here. Note: When using Cassandra as the System DB, 'fabric' may also be used as an authenticator.

The DEFAULT_GLOBAL_STORAGE_TYPE parameter in the [fabric] section is set to SYSTEM_DB. This means that by default, the Fabric storage type is the same as the Fabric System DB. You can either update the [system_db] settings only, impacting both the Storage and System DB types together, or define each one of them to have a different DB type.

The credentials of the Fabric system database are configured in the config.ini and are being encrypted by Fabric. You can also use one of the integrated Secret Managers to store them.

The following table lists the keyspaces or schemas created by Fabric:

Keyspace/Schema Name and Description

Keyspace/Schema Tables

k2system - Fabric main system keyspace/schema.

 

Table Name

Table Description

k2_lut_info 

 

Holds the metadata of the LUs, Common (reference) tables and Web Services deployed to Fabric.

k2_jobs

 

Holds information on the execution of all Fabric jobs.

nodes

 

List of all Fabric nodes in the cluster.

global_settings

List of all Globals and Environments whose default value has been overidden using the SET_GLOBAL command. This table is used to identify the overridden value of Globals or Environments when restarting Fabric.

broadway_recovery_point

Holds information on Broadway flows with recovery points.

Click here for more information about Broadway Recovery.

k2auth - Fabric security and credentials keyspace

Table Name

Table Description

user_credentials

List of Fabric users and their assigned roles.

roles

List of role definitions.

credentials

List of tokens with assigned roles. The tokens are encrypted.

permissions

List of permissions for a given role.

stripe_key_storage

Holds encrypted information about the master key. This table contains the key description, index, and value fields. The Master Key is broken into bytes, where each byte is stored in a separate record. Click here for more information about Fabric Security.

keys_descriptions

Holds a description of the master key.

k2batchprocess - Fabric batch processes information

Table Name

Table Description

batchprocess_list

List of the entire history of the batch process commands.

batchprocess_node_info

 

Summary of handled entities per batch process per node.

batchprocess_entities_info

Detailed information about an execution of a given entity per batch process command.

batchprocess_entities_errors

Detailed information about failed entities per batch process command. This table simplifies the analysis of failed entities.

k2audit - Fabric auditing

Table Name

Table Description

k2_auditing

Saves all activities performed on Fabric, when AUDIT is set to ON in the config.ini file.

k2staging - staging tables for the IIDFinder mechanism: 

A different set of caching, delta, solo and orphans’ tables is created for each LU that is synchronized by the IIDFinder (proactive sync) mechanism.

 

    

K2view_<LU Name> - a new keyspace is created for each deployed LU.

Note that when deploying the LU to the Fabric debug server, Fabric also concatenates the Fabric version and the project name to the keyspace of each LU. For example: k2view_test_cust_6_2_kb_fabric_project.

Table Name

Table Description

entity

Stores the list of all LUIs and their MicroDB.

entity_chunks

Stores large LUIs. The MicroDB of a Large LUI is divided into chunks and is stored in entity_chunks.

Click here for more information about Fabric storage.

k2masking - tables that support Broadway masking mechanism.

The k2masking keyspace can be created using the installation SQL script provided as part of the Masking library.

Click here for more information about Broadway Masking.

Table Name

Table Description

masking_cache

Stores the cached masked values.

uniqueness

Supports the uniqueness of the masked value per execution and masking ID.

TDM_SEQ_MAPPING

Keeps the masked values per execution ID, including the additional information such as LU, table, IID to enable the creation of reports.

Click here for more information about Fabric Architecture overview.

Previous

Fabric System Database

Fabric System Database is used by Fabric internal processes to monitor, secure, control, configure, audit and operate the application.

System Database Types

Fabric supports several types of Databases as the System Database, as described below.

NoSQL distributed database, such as Cassandra DB

  • Pros:
    • Scalable
    • Distributed
    • Built-in TTL mechanism on row level
    • If Cassandra is used as a MicroDB storage, there is no need to introduce additional DBs
    • Managed services (such as AWS Keyspaces or Astra) are supported
    • Supported by the iidFinder solution
    • Built-in mechanism for managing parallel threads during bulk instance loading
  • Cons:
    • Consistency
    • Not easy to operate and maintain

Relational database, such as PostgreSQL

  • Pros:
    • Consistency
    • In case of the TDM solution, PostgreSQL is already introduced
    • Compliance with services such as Cloud Spanner, AlloyDB
    • Easy to maintain
  • Cons:
    • Single point of failure
    • Not supported by the iidFinder solution

SQLite

  • Pros:
    • Development and single-node environments

Fabric creates several keyspaces or schemas (in case of SQLite or PostgreSQL) for its operation, where each starts with the k2 prefix.

Each deployed LU creates an additional k2view_[LU Name] keyspace or schema, such as k2view_customer.

Notes:

  • When set in the node.id configuration file, the cluster_id is concatenated to each keyspace or schema name. For example, if the cluster_id is set to crm1, the created keyspace or schema is k2view_customer_crm1.
  • When a Fabric project opens in the Fabric Studio, it concatenates the project name to the schema name.
  • A k2view_k2_ws keyspace or schema is created for the deployed WS.
  • When Fabric is configured to run a System DB type other than Cassandra - an internal product job runs, scans the tables to be cleaned and cleans them, using the TTL concept. The definition of each table's TTL value is tracked in the k2_table_level_ttl table.

Fabric uses the Cassandra DB as its default system management database.

When working with a non-Cassandra System DB, like SQLite or PostgreSQL, the following should be set in config.ini:

  • Populate the parameter values included in [system_db] section, which was introduced in Fabric V7.2. If you use Cassandra as the System DB, then you can ignore this section.

  • Set the SERVER_AUTHENTICATOR config parameter's value to be 'fabric' (default value is 'cassandra'). For more information about SERVER_AUTHENTICATOR config options, read here. Note: When using Cassandra as the System DB, 'fabric' may also be used as an authenticator.

The DEFAULT_GLOBAL_STORAGE_TYPE parameter in the [fabric] section is set to SYSTEM_DB. This means that by default, the Fabric storage type is the same as the Fabric System DB. You can either update the [system_db] settings only, impacting both the Storage and System DB types together, or define each one of them to have a different DB type.

The credentials of the Fabric system database are configured in the config.ini and are being encrypted by Fabric. You can also use one of the integrated Secret Managers to store them.

The following table lists the keyspaces or schemas created by Fabric:

Keyspace/Schema Name and Description

Keyspace/Schema Tables

k2system - Fabric main system keyspace/schema.

 

Table Name

Table Description

k2_lut_info 

 

Holds the metadata of the LUs, Common (reference) tables and Web Services deployed to Fabric.

k2_jobs

 

Holds information on the execution of all Fabric jobs.

nodes

 

List of all Fabric nodes in the cluster.

global_settings

List of all Globals and Environments whose default value has been overidden using the SET_GLOBAL command. This table is used to identify the overridden value of Globals or Environments when restarting Fabric.

broadway_recovery_point

Holds information on Broadway flows with recovery points.

Click here for more information about Broadway Recovery.

k2auth - Fabric security and credentials keyspace

Table Name

Table Description

user_credentials

List of Fabric users and their assigned roles.

roles

List of role definitions.

credentials

List of tokens with assigned roles. The tokens are encrypted.

permissions

List of permissions for a given role.

stripe_key_storage

Holds encrypted information about the master key. This table contains the key description, index, and value fields. The Master Key is broken into bytes, where each byte is stored in a separate record. Click here for more information about Fabric Security.

keys_descriptions

Holds a description of the master key.

k2batchprocess - Fabric batch processes information

Table Name

Table Description

batchprocess_list

List of the entire history of the batch process commands.

batchprocess_node_info

 

Summary of handled entities per batch process per node.

batchprocess_entities_info

Detailed information about an execution of a given entity per batch process command.

batchprocess_entities_errors

Detailed information about failed entities per batch process command. This table simplifies the analysis of failed entities.

k2audit - Fabric auditing

Table Name

Table Description

k2_auditing

Saves all activities performed on Fabric, when AUDIT is set to ON in the config.ini file.

k2staging - staging tables for the IIDFinder mechanism: 

A different set of caching, delta, solo and orphans’ tables is created for each LU that is synchronized by the IIDFinder (proactive sync) mechanism.

 

    

K2view_<LU Name> - a new keyspace is created for each deployed LU.

Note that when deploying the LU to the Fabric debug server, Fabric also concatenates the Fabric version and the project name to the keyspace of each LU. For example: k2view_test_cust_6_2_kb_fabric_project.

Table Name

Table Description

entity

Stores the list of all LUIs and their MicroDB.

entity_chunks

Stores large LUIs. The MicroDB of a Large LUI is divided into chunks and is stored in entity_chunks.

Click here for more information about Fabric storage.

k2masking - tables that support Broadway masking mechanism.

The k2masking keyspace can be created using the installation SQL script provided as part of the Masking library.

Click here for more information about Broadway Masking.

Table Name

Table Description

masking_cache

Stores the cached masked values.

uniqueness

Supports the uniqueness of the masked value per execution and masking ID.

TDM_SEQ_MAPPING

Keeps the masked values per execution ID, including the additional information such as LU, table, IID to enable the creation of reports.

Click here for more information about Fabric Architecture overview.

Previous

© Copyright 2024 K2VIEW