Knowledge Base Academy Community My Tickets
Fabric 7.2
Fabric 8.0 Fabric 7.2 Fabric 7.1 Fabric 7.0 Fabric 6.5 Fabric 6.4

Fabric - Main Configuration Files

Configuration Directory

Important:

  • Fabric configuration files are located under the $K2_HOME/config directory.
  • Fabric configuration template files are located under the $K2_HOME/fabric/config.template.

Make sure to edit the configuration files under the $K2_HOME/config directory and not under the template directory. Configuration files should be edited on all Fabric nodes so it will become effective on the cluster level.

Main Configuration Files

Configuration File

Description

config.ini

 

Fabric's main configuration file holding different sections of parameters where each section has its own parameters. Default Fabric values are set for commented parameters.

iifConfig.ini

The IIDFinder mechanism's main configuration file.

node.id

List of Fabric node identifiers for the Affinity mechanism. Supports several Fabric clusters on one Cassandra cluster.

logback.xml, logback-iid_finder.xml, and logback-init_finder.xml

Fabric logs configuration files.

jvm.options

 Sets the flags used by Fabric to startup the JVM (Java Virtual Machine). For example:  To use the machine's local timezone, uncomment the                            -DFABRIC_LOCAL_TIMEZONE parameter and set it to true to use the local time-zone of the Fabric server.

jmxremote.access and jmxremote.password

Remote JMX API access to monitoring.

modules

List of internal Fabric modules. Each module depends on the previous modules in the file. You can comment some internal Fabric modules and restart the Fabric node to avoid starting the commented modules and have a lightweight start on Fabric. For example: comment the jobs module to avoid running jobs on the Fabric node. The following modules can be commented:

  • jobs,running Fabric jobs.
  • webserver,connecting Fabric via http or https. For example: invoke Fabric WS, deploy Fabric implementation from Fabric Studio.
  • commonarea, accessing common Reference Tables.
  • dbserve, connecting Fabric via remote JDBC connection. For example: login Fabric console using fabric command, defining Fabric node as a remote Fabric interface.
  • clustertimecheck, comparing the time between the Fabric nodes, included in the Fabric cluster and throwing an error if a difference is identified.
  • fips, fips certification mode, to turn it on use fips:mode=on. If it is not added fabric works by default in fips=off mode.

config.ini

Fabric's main configuration file which holds different sections of parameters where each section has its own parameters and default Fabric values for commented parameters.

Parameters Category

Section Names

Main Parameters

Cassandra Connection
  • cassandra
  • default_session

Configurations for creating a connection to the Cassandra cluster:

  • Replication options.
  • Consistency level.
  • Cassandra's connection details.
  • Set an SSL connection.

Fabric Settings
  • fabric
  • fabric_cluster
  • jdbc-server
  • audit
  • audit_kafka_producer
  • Date and time formats.
  • Storage and Export directories.
  • Fabric auditing settings.
  • Connection pools settings.
  • The maximum number of concurrent connections that are allowed for a single data source.
  • Batch process settings.
  • Default Sync mode.
  • SYNC_PROTECTION parameter.
  • Sync of CommonDB Reference tables.
  • Parallel Sync on Table Populations with the same execution order.
  • Enable running DELETE INSTANCES IF NOT EXIST Fabric command.
  • LUI compression types when storing the LUI in Cassandra.
  • Web service parameters.
  • Fabric interaction with Cassandra for adding, removing or editing user’s activities.
  • Fabric Heartbeat mechanism for Fabric jobs.

PubSub
  • default_pubsub

  • PubSub abstraction layer configuration which defines the Kafka settings across various Fabric processes.

Fabric Jobs
  • jobs
  • Thread pool size, set the maximum number of jobs to be executed in parallel in a single Fabric node.
  • Archiving time, the number of hours to delete the Job record from the k2_jobs table.

Batch Process
  • batch_process
  • MAX_WORKERS_PER_NODE parameter sets the maximum number of threads that are used in all batch process units (executions) together on this node.     
  • Supports MAX_WORKERS_PER_NODE=0 per node to avoid running a batch process on a specific node.    

Parsers
  • parsers

  • Writing method into Cassandra using Cassandra loader or JDBC driver for parsers. The PARSER_WRITING_TYPE parameter defines the method used to load the data into Cassandra by Parsers: Cassandra Loader or JDBC driver. The default value is JDBC.

IIDFinder
  • finder
  • Enable ORPHANAGE job, handles the Orphans record in the background.
  • Enable SWEEP job, sweeps the invalid cache entries.

Cassandra Loader configuration
  • default_loader
  • parser_loader
  • batch_process_loader
  • iid_finder_loader
  • Cassandra Loader configuration.
  • The default setting of default_loader for parsers, batch processes, or IIDFinder activities can be overridden.

Cassandra loader session configuration
  • loader_session         
  • Overrides the default_session for the sessions, created for Cassandra Loader operations (parsers, batch process, IIDFinder).

CommonDB Reference Tables
  • common_area_config
  • common_area_pubsub

 

LUI Storage
  • fabricdb
  • cassandra_entity_storage

 

Fabric Security Hardening
  • encryption

 

Change Data Capture (CDC) and Search
  • cdc
  • search_loader_pubsub
  • search_engine

 

Consistency Level
  • default_session
  • CONSISTENCY_LEVEL, default is LOCAL_QUORUM. Impacts all Cassandra actions.
  • SERIAL_CONSISTENCY_LEVEL, default is LOCAL_SERIAL. Used internally for Fabric Jobs.

Set Fabric Time Zone

Fabric is a multi-node, multi Datacenter (DC) system which interacts and exchanges data with target and source systems and exposes APIs. This may present a challenge when dealing with time zones.

By default, Fabric is configured to use the Coordinated Universal Time (UTC) zone, regardless of the time zone of the host server. UTC is roughly equivalent to GMT time but does not observe Day Light Saving.

When getting data from source systems and storing it in Fabric, the best practice is to normalize all timestamps (populated in Fabric) to UTC and store any additional time zone information in separate fields.

When exposing or exporting data to upstream systems that require a different time presentation, the conversion should be done during the data movement (not stored in Fabric) and planned per consumer. This behavior makes it easier to process logs and analyze data interchange of cross time zones datacenters, nodes and external systems.

Fabric does allow changing this default behavior and changing the process time zone from UTC to the server's local time:

  • Go to the jvm.options file in the config directory and uncomment the following line (remove the # character) from the following parameter: #-DFABRIC_LOCAL_TIMEZONE=true. This affects internal activities such as log file reporting or trace files and can also change the behavior of some JDBC drivers against date fields that contain time zone information.

  • To change how data is saved to Fabric, open the config.ini file in the config folder and change the DATETIME_FORMAT_LOCAL_TIMEZONE parameter as follows:

    DATETIME_FORMAT_LOCAL_TIMEZONE=true

    This affects the conversion of Date objects to a table field entry in the Fabric database and formats them according to the Wall Time of the local time zone.

Note that you must restart Fabric after making these updates.

Since these entries affect different stages of data intake and exposure, it is important to test the Date format after changing them to make sure the required format is achieved against all sources and data types. Changes will not affect data already in the database.

node.id

This file lists Fabric node identifiers for the Affinity mechanism. The following identifiers can be set in the node.id file:

  • uuid, if this parameter is not defined, Fabric automatically generates a value for the uuid during startup.

  • logical_id, used to define an Affinity for the Fabric jobs mechanism. The logical_id contains only letters and numbers. Several nodes can share the same logical_id. In addition, several logical IDs can be set for one node. The number of threads allocated to each logical_id can also be defined by concatenating it to the logical_id name separated with a colon sign. For example, the logical_id for a given node has the following values: A:2, B:3, and C:6. If there are 10 threads in the pool for this node, then the job using logical_id C as an Affinity will get 6 out of the 10 threads.

    • Affinity Pool Size (from 6.4.2 onwards)

    A Recommended Pool Size capability has been added to the affinity function to rebalance jobs and get the ability to dynamically split (in runtime) jobs executions between nodes. 2 new parameters can be defined:

    • recommended number of nodes
    • maximum number of nodes
    • logical_id:2 4 or logical_id:2-4 whereby 2 is the recommended number and 4 the maximum number of nodes to be allocated to jobs.
    • In cases where only one value is defined, it will be assumed that it refers to the maximum number of nodes.

  • cluster_id, cluster identifier. The cluster_id is set to support a configuration of several Fabric clusters on one Cassandra cluster. The cluster_id is concatenated to each keyspace name. For example- if the cluster_id is set to “fabric1”, then Fabric concatenates “_fabric1” to each keyspace.

    IID Finder

    New Parameter added - iidfinder_job:1 to configure the number of cloud jobs per node. The defaulat value is 1.

    Could be set to 0 if IidFinder is not used at all, or on some nodes, but not all of them.

Update Configuration Files

Update Config.ini, Node.id, and Logback Files

Fabric adds a notification to the k2fabric.log if the updates are loaded automatically to Fabric. If the changes require a restart of the Fabric node, Fabric adds a warning to the log file.

Examples:

  • INFO [FileChangeMonitor] 2020-06-07 18:30:40,207 c.k.f.c.i.Configurator - [LID1000000000496] Configurator changed fabric.DELETE_INSTANCES_IF_NOT_EXIST_COMMAND_ENABLED from false to true INFO [FileChangeMonitor] 2020-06-07 18:30:40,207 c.k.f.c.i.FileChangeMonitor - [LID1000000000496] config.ini was reloaded
  • WARN [FileChangeMonitor] 2020-06-07 18:38:33,270 c.k.f.c.i.Configurator - [LID1000000000496] Configurator will not update fabricdb.MDB_DEFAULT_SCHEMA_CACHE_STORAGE_TYPE from com.k2view.cdbms.dao.CassandraEntityStorage to NONE at runtime.

Update Jvm.option, Jmxremote, and Modules Files

Fabric must be restarted to apply the updates.

Update iifConfig.ini

When updating the iifConfig.ini, do the following:

  • When updating the parameters under orphanage_job or sweep_job sections, or ENABLE_KAFKA_DELTA parameters, restart the Fabric node.
  • Restart the IIDFinder process.

Click for more information about the IIDFinder Process.

Previous

Fabric - Main Configuration Files

Configuration Directory

Important:

  • Fabric configuration files are located under the $K2_HOME/config directory.
  • Fabric configuration template files are located under the $K2_HOME/fabric/config.template.

Make sure to edit the configuration files under the $K2_HOME/config directory and not under the template directory. Configuration files should be edited on all Fabric nodes so it will become effective on the cluster level.

Main Configuration Files

Configuration File

Description

config.ini

 

Fabric's main configuration file holding different sections of parameters where each section has its own parameters. Default Fabric values are set for commented parameters.

iifConfig.ini

The IIDFinder mechanism's main configuration file.

node.id

List of Fabric node identifiers for the Affinity mechanism. Supports several Fabric clusters on one Cassandra cluster.

logback.xml, logback-iid_finder.xml, and logback-init_finder.xml

Fabric logs configuration files.

jvm.options

 Sets the flags used by Fabric to startup the JVM (Java Virtual Machine). For example:  To use the machine's local timezone, uncomment the                            -DFABRIC_LOCAL_TIMEZONE parameter and set it to true to use the local time-zone of the Fabric server.

jmxremote.access and jmxremote.password

Remote JMX API access to monitoring.

modules

List of internal Fabric modules. Each module depends on the previous modules in the file. You can comment some internal Fabric modules and restart the Fabric node to avoid starting the commented modules and have a lightweight start on Fabric. For example: comment the jobs module to avoid running jobs on the Fabric node. The following modules can be commented:

  • jobs,running Fabric jobs.
  • webserver,connecting Fabric via http or https. For example: invoke Fabric WS, deploy Fabric implementation from Fabric Studio.
  • commonarea, accessing common Reference Tables.
  • dbserve, connecting Fabric via remote JDBC connection. For example: login Fabric console using fabric command, defining Fabric node as a remote Fabric interface.
  • clustertimecheck, comparing the time between the Fabric nodes, included in the Fabric cluster and throwing an error if a difference is identified.
  • fips, fips certification mode, to turn it on use fips:mode=on. If it is not added fabric works by default in fips=off mode.

config.ini

Fabric's main configuration file which holds different sections of parameters where each section has its own parameters and default Fabric values for commented parameters.

Parameters Category

Section Names

Main Parameters

Cassandra Connection
  • cassandra
  • default_session

Configurations for creating a connection to the Cassandra cluster:

  • Replication options.
  • Consistency level.
  • Cassandra's connection details.
  • Set an SSL connection.

Fabric Settings
  • fabric
  • fabric_cluster
  • jdbc-server
  • audit
  • audit_kafka_producer
  • Date and time formats.
  • Storage and Export directories.
  • Fabric auditing settings.
  • Connection pools settings.
  • The maximum number of concurrent connections that are allowed for a single data source.
  • Batch process settings.
  • Default Sync mode.
  • SYNC_PROTECTION parameter.
  • Sync of CommonDB Reference tables.
  • Parallel Sync on Table Populations with the same execution order.
  • Enable running DELETE INSTANCES IF NOT EXIST Fabric command.
  • LUI compression types when storing the LUI in Cassandra.
  • Web service parameters.
  • Fabric interaction with Cassandra for adding, removing or editing user’s activities.
  • Fabric Heartbeat mechanism for Fabric jobs.

PubSub
  • default_pubsub

  • PubSub abstraction layer configuration which defines the Kafka settings across various Fabric processes.

Fabric Jobs
  • jobs
  • Thread pool size, set the maximum number of jobs to be executed in parallel in a single Fabric node.
  • Archiving time, the number of hours to delete the Job record from the k2_jobs table.

Batch Process
  • batch_process
  • MAX_WORKERS_PER_NODE parameter sets the maximum number of threads that are used in all batch process units (executions) together on this node.     
  • Supports MAX_WORKERS_PER_NODE=0 per node to avoid running a batch process on a specific node.    

Parsers
  • parsers

  • Writing method into Cassandra using Cassandra loader or JDBC driver for parsers. The PARSER_WRITING_TYPE parameter defines the method used to load the data into Cassandra by Parsers: Cassandra Loader or JDBC driver. The default value is JDBC.

IIDFinder
  • finder
  • Enable ORPHANAGE job, handles the Orphans record in the background.
  • Enable SWEEP job, sweeps the invalid cache entries.

Cassandra Loader configuration
  • default_loader
  • parser_loader
  • batch_process_loader
  • iid_finder_loader
  • Cassandra Loader configuration.
  • The default setting of default_loader for parsers, batch processes, or IIDFinder activities can be overridden.

Cassandra loader session configuration
  • loader_session         
  • Overrides the default_session for the sessions, created for Cassandra Loader operations (parsers, batch process, IIDFinder).

CommonDB Reference Tables
  • common_area_config
  • common_area_pubsub

 

LUI Storage
  • fabricdb
  • cassandra_entity_storage

 

Fabric Security Hardening
  • encryption

 

Change Data Capture (CDC) and Search
  • cdc
  • search_loader_pubsub
  • search_engine

 

Consistency Level
  • default_session
  • CONSISTENCY_LEVEL, default is LOCAL_QUORUM. Impacts all Cassandra actions.
  • SERIAL_CONSISTENCY_LEVEL, default is LOCAL_SERIAL. Used internally for Fabric Jobs.

Set Fabric Time Zone

Fabric is a multi-node, multi Datacenter (DC) system which interacts and exchanges data with target and source systems and exposes APIs. This may present a challenge when dealing with time zones.

By default, Fabric is configured to use the Coordinated Universal Time (UTC) zone, regardless of the time zone of the host server. UTC is roughly equivalent to GMT time but does not observe Day Light Saving.

When getting data from source systems and storing it in Fabric, the best practice is to normalize all timestamps (populated in Fabric) to UTC and store any additional time zone information in separate fields.

When exposing or exporting data to upstream systems that require a different time presentation, the conversion should be done during the data movement (not stored in Fabric) and planned per consumer. This behavior makes it easier to process logs and analyze data interchange of cross time zones datacenters, nodes and external systems.

Fabric does allow changing this default behavior and changing the process time zone from UTC to the server's local time:

  • Go to the jvm.options file in the config directory and uncomment the following line (remove the # character) from the following parameter: #-DFABRIC_LOCAL_TIMEZONE=true. This affects internal activities such as log file reporting or trace files and can also change the behavior of some JDBC drivers against date fields that contain time zone information.

  • To change how data is saved to Fabric, open the config.ini file in the config folder and change the DATETIME_FORMAT_LOCAL_TIMEZONE parameter as follows:

    DATETIME_FORMAT_LOCAL_TIMEZONE=true

    This affects the conversion of Date objects to a table field entry in the Fabric database and formats them according to the Wall Time of the local time zone.

Note that you must restart Fabric after making these updates.

Since these entries affect different stages of data intake and exposure, it is important to test the Date format after changing them to make sure the required format is achieved against all sources and data types. Changes will not affect data already in the database.

node.id

This file lists Fabric node identifiers for the Affinity mechanism. The following identifiers can be set in the node.id file:

  • uuid, if this parameter is not defined, Fabric automatically generates a value for the uuid during startup.

  • logical_id, used to define an Affinity for the Fabric jobs mechanism. The logical_id contains only letters and numbers. Several nodes can share the same logical_id. In addition, several logical IDs can be set for one node. The number of threads allocated to each logical_id can also be defined by concatenating it to the logical_id name separated with a colon sign. For example, the logical_id for a given node has the following values: A:2, B:3, and C:6. If there are 10 threads in the pool for this node, then the job using logical_id C as an Affinity will get 6 out of the 10 threads.

    • Affinity Pool Size (from 6.4.2 onwards)

    A Recommended Pool Size capability has been added to the affinity function to rebalance jobs and get the ability to dynamically split (in runtime) jobs executions between nodes. 2 new parameters can be defined:

    • recommended number of nodes
    • maximum number of nodes
    • logical_id:2 4 or logical_id:2-4 whereby 2 is the recommended number and 4 the maximum number of nodes to be allocated to jobs.
    • In cases where only one value is defined, it will be assumed that it refers to the maximum number of nodes.

  • cluster_id, cluster identifier. The cluster_id is set to support a configuration of several Fabric clusters on one Cassandra cluster. The cluster_id is concatenated to each keyspace name. For example- if the cluster_id is set to “fabric1”, then Fabric concatenates “_fabric1” to each keyspace.

    IID Finder

    New Parameter added - iidfinder_job:1 to configure the number of cloud jobs per node. The defaulat value is 1.

    Could be set to 0 if IidFinder is not used at all, or on some nodes, but not all of them.

Update Configuration Files

Update Config.ini, Node.id, and Logback Files

Fabric adds a notification to the k2fabric.log if the updates are loaded automatically to Fabric. If the changes require a restart of the Fabric node, Fabric adds a warning to the log file.

Examples:

  • INFO [FileChangeMonitor] 2020-06-07 18:30:40,207 c.k.f.c.i.Configurator - [LID1000000000496] Configurator changed fabric.DELETE_INSTANCES_IF_NOT_EXIST_COMMAND_ENABLED from false to true INFO [FileChangeMonitor] 2020-06-07 18:30:40,207 c.k.f.c.i.FileChangeMonitor - [LID1000000000496] config.ini was reloaded
  • WARN [FileChangeMonitor] 2020-06-07 18:38:33,270 c.k.f.c.i.Configurator - [LID1000000000496] Configurator will not update fabricdb.MDB_DEFAULT_SCHEMA_CACHE_STORAGE_TYPE from com.k2view.cdbms.dao.CassandraEntityStorage to NONE at runtime.

Update Jvm.option, Jmxremote, and Modules Files

Fabric must be restarted to apply the updates.

Update iifConfig.ini

When updating the iifConfig.ini, do the following:

  • When updating the parameters under orphanage_job or sweep_job sections, or ENABLE_KAFKA_DELTA parameters, restart the Fabric node.
  • Restart the IIDFinder process.

Click for more information about the IIDFinder Process.

Previous

© Copyright 2024 K2VIEW