Fabric - Main Configuration Files

Configuration Directory

Important:

• Fabric configuration files are located under the $K2_HOME/config directory.
• Fabric configuration template files are located under the $K2_HOME/fabric/config.template.

Make sure to edit the configuration files under the $K2_HOME/config directory and not under the template directory. Configuration files should be edited on all Fabric nodes so it will become effective on the cluster level.

Main Configuration Files

config.ini

Fabric's main configuration file which holds different sections of parameters where each section has its own parameters and default Fabric values for commented parameters.

Set Fabric Time Zone

Fabric is a multi-node, multi Datacenter (DC) system which interacts and exchanges data with target and source systems and exposes APIs. This may present a challenge when dealing with time zones.

By default, Fabric is configured to use the Coordinated Universal Time (UTC) zone, regardless of the time zone of the host server. UTC is roughly equivalent to GMT time but does not observe Day Light Saving.

When getting data from source systems and storing it in Fabric, the best practice is to normalize all timestamps (populated in Fabric) to UTC and store any additional time zone information in separate fields.

When exposing or exporting data to upstream systems that require a different time presentation, the conversion should be done during the data movement (not stored in Fabric) and planned per consumer. This behavior makes it easier to process logs and analyze data interchange of cross time zones datacenters, nodes and external systems.

Fabric does allow changing this default behavior and changing the process time zone from UTC to the server's local time:

• Go to the jvm.options file in the config directory and uncomment the following line (remove the # character) from the following parameter: #-DFABRIC_LOCAL_TIMEZONE=true. This affects internal activities such as log file reporting or trace files and can also change the behavior of some JDBC drivers against date fields that contain time zone information.

• To change how data is saved to Fabric, open the config.ini file in the config folder and change the DATETIME_FORMAT_LOCAL_TIMEZONE parameter as follows:

  DATETIME_FORMAT_LOCAL_TIMEZONE=true

  This affects the conversion of Date objects to a table field entry in the Fabric database and formats them according to the Wall Time of the local time zone.

Note that you must restart Fabric after making these updates.

Since these entries affect different stages of data intake and exposure, it is important to test the Date format after changing them to make sure the required format is achieved against all sources and data types. Changes will not affect data already in the database.

node.id

This file lists Fabric node identifiers for the Affinity mechanism. The following identifiers can be set in the node.id file:

• uuid, if this parameter is not defined, Fabric automatically generates a value for the uuid during startup.

• logical_id, used to define an Affinity for the Fabric jobs mechanism. The logical_id contains only letters and numbers. Several nodes can share the same logical_id. In addition, several logical IDs can be set for one node. The number of threads allocated to each logical_id can also be defined by concatenating it to the logical_id name separated with a colon sign. For example, the logical_id for a given node has the following values: A:2, B:3, and C:6. If there are 10 threads in the pool for this node, then the job using logical_id C as an Affinity will get 6 out of the 10 threads.

  • Affinity Pool Size (from 6.4.2 onwards)

  A Recommended Pool Size capability has been added to the affinity function to rebalance jobs and get the ability to dynamically split (in runtime) jobs executions between nodes. 2 new parameters can be defined:

  • recommended number of nodes
  • maximum number of nodes
  • logical_id:2 4 or logical_id:2-4 whereby 2 is the recommended number and 4 the maximum number of nodes to be allocated to jobs.
  • In cases where only one value is defined, it will be assumed that it refers to the maximum number of nodes.

• cluster_id, cluster identifier. The cluster_id is set to support a configuration of several Fabric clusters on one Cassandra cluster. The cluster_id is concatenated to each keyspace name. For example- if the cluster_id is set to “fabric1”, then Fabric concatenates “_fabric1” to each keyspace.

  IID Finder

  New Parameter added - iidfinder_job:1 to configure the number of cloud jobs per node. The defaulat value is 1.

  Could be set to 0 if IidFinder is not used at all, or on some nodes, but not all of them.

Update Configuration Files

Update Config.ini, Node.id, and Logback Files

Fabric adds a notification to the k2fabric.log if the updates are loaded automatically to Fabric. If the changes require a restart of the Fabric node, Fabric adds a warning to the log file.

Examples:

• INFO [FileChangeMonitor] 2020-06-07 18:30:40,207 c.k.f.c.i.Configurator - [LID1000000000496] Configurator changed fabric.DELETE_INSTANCES_IF_NOT_EXIST_COMMAND_ENABLED from false to true INFO [FileChangeMonitor] 2020-06-07 18:30:40,207 c.k.f.c.i.FileChangeMonitor - [LID1000000000496] config.ini was reloaded
• WARN [FileChangeMonitor] 2020-06-07 18:38:33,270 c.k.f.c.i.Configurator - [LID1000000000496] Configurator will not update fabricdb.MDB_DEFAULT_SCHEMA_CACHE_STORAGE_TYPE from com.k2view.cdbms.dao.CassandraEntityStorage to NONE at runtime.

Update Jvm.option, Jmxremote, and Modules Files

Fabric must be restarted to apply the updates.

Update iifConfig.ini

When updating the iifConfig.ini, do the following:

• When updating the parameters under orphanage_job or sweep_job sections, or ENABLE_KAFKA_DELTA parameters, restart the Fabric node.
• Restart the IIDFinder process.

Click for more information about the IIDFinder Process.