Fabric - Main Configuration Files

Configuration Directory

Important:

• Fabric configuration files are located under the $K2_HOME/config directory.
• Fabric configuration template files are located under the $K2_HOME/fabric/config.template.

Make sure to edit the configuration files under the $K2_HOME/config directory and not under the template directory. Configuration files should be edited on all Fabric nodes so it will become effective on the cluster level.

Main Configuration Files

config.ini

Fabric's main configuration file which holds different sections of parameters where each section has its own parameters and default Fabric values for commented parameters.

Set Fabric Time Zone

Fabric is a multi-node, multi Datacenter (DC) system which interacts and exchanges data with target and source systems and exposes APIs. This may present a challenge when dealing with time zones.

By default, Fabric is configured to use the Coordinated Universal Time (UTC) zone, regardless of the time zone of the host server. UTC is roughly equivalent to GMT time but does not observe Day Light Saving.

When getting data from source systems and storing it in Fabric, the best practice is to normalize all timestamps (populated in Fabric) to UTC and store any additional time zone information in separate fields.

When exposing or exporting data to upstream systems that require a different time presentation, the conversion should be done during the data movement (not stored in Fabric) and planned per consumer. This behavior makes it easier to process logs and analyze data interchange of cross time zones datacenters, nodes and external systems.

Fabric does allow changing this default behavior and changing the process time zone from UTC to the server's local time:

• Go to the jvm.options file in the config directory and uncomment the following line (remove the # character) from the following parameter: #-DFABRIC_LOCAL_TIMEZONE=true. This affects internal activities such as log file reporting or trace files and can also change the behavior of some JDBC drivers against date fields that contain time zone information.

• To change how data is saved to Fabric, open the config.ini file in the config folder and change the DATETIME_FORMAT_LOCAL_TIMEZONE parameter as follows:

  DATETIME_FORMAT_LOCAL_TIMEZONE=true

  This affects the conversion of Date objects to a table field entry in the Fabric database and formats them according to the Wall Time of the local time zone.

Note that you must restart Fabric after making these updates.

Since these entries affect different stages of data intake and exposure, it is important to test the Date format after changing them to make sure the required format is achieved against all sources and data types. Changes will not affect data already in the database.

node.id

This file lists Fabric node identifiers for the Affinity mechanism. The following identifiers can be set in the node.id file:

• uuid, if this parameter is not defined, Fabric automatically generates a value for the uuid during startup.

• logical_id, used to define an Affinity for the Fabric jobs mechanism. The logical_id contains only letters and numbers. Several nodes can share the same logical_id. In addition, several logical IDs can be set for one node. The number of threads allocated to each logical_id can also be defined by concatenating it to the logical_id name separated with a colon sign. For example, the logical_id for a given node has the following values: A:2, B:3, and C:6. If there are 10 threads in the pool for this node, then the job using logical_id C as an Affinity will get 6 out of the 10 threads.

  • Affinity Pool Size (from 6.4.2 onwards)

  A Recommended Pool Size capability has been added to the affinity function to rebalance jobs and get the ability to dynamically split (in runtime) jobs executions between nodes. 2 new parameters can be defined:

  • recommended number of nodes
  • maximum number of nodes
  • logical_id:2 4 or logical_id:2-4 whereby 2 is the recommended number and 4 the maximum number of nodes to be allocated to jobs.
  • In cases where only one value is defined, it will be assumed that it refers to the maximum number of nodes.

• cluster_id, cluster identifier. The cluster_id is set to support a configuration of several Fabric clusters on one Cassandra cluster. The cluster_id is concatenated to each keyspace name. For example- if the cluster_id is set to “fabric1”, then Fabric concatenates “_fabric1” to each keyspace.

  IID Finder

  New Parameter added - iidfinder_job:1 to configure the number of cloud jobs per node. The defaulat value is 1.

  Could be set to 0 if IidFinder is not used at all, or on some nodes, but not all of them.

Update Configuration Files

Update Config.ini, Node.id, and Logback Files

Fabric adds a notification to the k2fabric.log if the updates are loaded automatically to Fabric. If the changes require a restart of the Fabric node, Fabric adds a warning to the log file.

Examples:

• INFO [FileChangeMonitor] 2020-06-07 18:30:40,207 c.k.f.c.i.Configurator - [LID1000000000496] Configurator changed fabric.DELETE_INSTANCES_IF_NOT_EXIST_COMMAND_ENABLED from false to true INFO [FileChangeMonitor] 2020-06-07 18:30:40,207 c.k.f.c.i.FileChangeMonitor - [LID1000000000496] config.ini was reloaded
• WARN [FileChangeMonitor] 2020-06-07 18:38:33,270 c.k.f.c.i.Configurator - [LID1000000000496] Configurator will not update fabricdb.MDB_DEFAULT_SCHEMA_CACHE_STORAGE_TYPE from com.k2view.cdbms.dao.CassandraEntityStorage to NONE at runtime.

Update Jvm.option, Jmxremote, and Modules Files

Fabric must be restarted to apply the updates.

Update iifConfig.ini

When updating the iifConfig.ini, do the following:

• When updating the parameters under orphanage_job or sweep_job sections, or ENABLE_KAFKA_DELTA parameters, restart the Fabric node.
• Restart the IIDFinder process.

Configuration Changes' UI Interface

Objective

The interface enables a user with suitable permissions to access the Admin page. This is done in order to change values in the config.ini file, for the specific node, and to save these changes (overrides) in the system DB.

Configuration filtering options

• Show All - all configuration parameters are viewable in the table.
• Overrides Only (default view) - only parameters, whose values were changed, are viewable.
• Once the 'Hidden' checkbox is checked, the hidden parameters are also viewable.
• All table columns can be filtered, sorted and grouped.

Update a parameter

1. To update a parameter value, go to the value column and edit (use the edit icon or simply edit) the value of the specific parameter that you wish to change.
2. Once changed, the entire row will automatically be highlighted to indicate that the change has been applied.
3. Once saved, the highlighted indication will be turned off.
4. In case the changed parameter requires a restart to be applicable, a corresponding icon in the relevant key column will indicate it, along with a tooltip indication.
5. Once a change in a parameter, which requires a restart, occurs and is saved to a system DB, the icon indication will blink and a pop-up indication on the save button will be highlighted, indicating that this change will be applicable only after a restart.
6. Revert to default - once a value was changed, but not saved to a system DB, the revert option will enable you to go back to the original default value. Upon saving the change, the revert option will no longer be available, until another change will be applied.
7. When switching to an overrides-only view, after a change has been applied but not saved, once reverting to the default value, the parameter will be removed from this view.
8. Reset Drops will delete the overrides - if you have settings for a specific machine, it is advisable to save them in the config.ini manually/hardcoded, and to use the IGNORE_CONFIG_OVERRIDES_ON_START setting for recovery purposes in case of a faulty value, which will ignore all applied overrides.

Add a parameter

1. Clicking on the Add Parameter button will open a new editable row highlighted in green.
2. The section column will provide a drop-down list of existing sections to choose from, or alternatively, a new section can be entered.
3. Use the edit icons on the 'KEY' and 'VALUE' columns to enter a new value.
4. Note: The 'Description' field is not editable.
5. A delete button in the revert column will enable deleting the new added parameter's row.
6. Adding an existing parameter, which is configured as hidden, will override its value once saved.

Some sections of config.ini are not shown in the configuration table UI.

Known Issue

Some parameters of the config.ini that are supposed to be unhidden - are hidden in the configuration table. To mitigate this issue, if there is a parameter that needs to be used and is not shown in the table, tick on the hidden filter, which will enable you to see it.