Fabric Commands

Fabric includes a number of commands for viewing Fabric configurations, updating Fabric settings and running Fabric processes. Fabric commands can be executed from either the Fabric console or via the user code (project implementation) that invokes Fabric commands in the execute() and fetch() methods.

Fabric commands are not case sensitive. For example, a Get, get, or GET command are all equivalent.

In this document, we write commands in ALL CAPS when describing the commands in general, and in small type when quoting a command line example. This is done for clarity purposes only. You can type the commands in whatever type is convenient for your purposes, but it is a good practice to keep the manner of typing commands consistent.

Fabric Help

After logging into Fabric, type HELP to view a list of available Fabric commands. To view the description and syntax of a specific command, type help [command name].

For example:

Fabric Commands - Main Groups

Fabric commands can be divided into the following groups:

Get Commands

Get Instance

The GET command is used to get information for a given LUI and to synchronize information from data sources if needed. Note that multiple LUs can be received using a GET command. However, multiple LUIs cannot be received from the same LU using a GET command.

The following message is displayed when attempting to get multiple LUIs from the same LU using a GET command:

Only single instance per LUT can be used on the same GET command.

The consistency level of the GET LUI command can be set to ONE. If it fails to achieve a QUORUM consistency level, the sync mode is set to OFF. To do so, run the following Fabric command on the session:

SET LUI_READ_ONE_WHEN_FAIL = true

Note that this command sets the consistency level on the session level. The default value of this parameter is false.

The following table lists the GET commands:

Remote GET and GETF Commands

GET and GETF commands can be executed from a datacenter (DC) that is not connected to data sources if other DCs are connected to the source interfaces. To carry out a GET or GETF command in such a manner, populate the DC parameter name of the GET and GETF commands. This will invoke the remote DC connected to the data source via the JDBC. The remote GET and GETF commands return the instances after executing the commands on the remote Fabric node. Cassandra then replicates the data between the nodes of the Cassandra cluster.

The remote GET and GETF commands run on a random Fabric node on the remote DC. Therefore, always verify the permissions for the GET and GETF commands’ execution on Fabric’s local and remote nodes.

Note that users are responsible for identifying if a sync on an LUI is required, and to only then run the remote GET or GETF commands. This prevents unnecessary calls to the remote Fabric node and getting the local LUI version instead.

Delete LUI Command

The DELETE INSTANCE command deletes an LUI or multiple LUIs from Fabric. Unlike the GET command, several LUI from the same LU can be deleted using one DELETE command. The consistency level of the Delete Instance is set in the LU_INSTANCE_DELETE parameter of the config.ini file. The default value is LOCAL_QUOROM.

The following table lists the DELETE commands:

Release LU

The Fabric RELEASE command is used to detach the LUI from the session on a list of LUs or all LUs.

Fabric View

Fabric has commands that display a Fabric configuration and its settings. For example:

• Fabric cluster information:

  • CLUSTERID, returns the cluster identifier defined on the node.id.
  • CLUSTERSTATUS, returns the status of all Fabric nodes. Also includes: node_id, logical IDs, DC name, IP addresses.
  • TIME, gets the node system time.
  • VERSION INFO, the version of the installed Fabric. Note that to get the Fabric version when logged out of Fabric, use the k2fabric -version command.

• Information about the deployed implementation:

  • DESCRIBE, to query Fabric's metadata structure.
  • LIST, a list of deployed objects and Fabric credentials (ROLES, USERS, TOKENS, ROLE_PERMISSIONS, ENVIRONMENTS and METHODS).

• General information:

  • SET, displays the current session’s settings: Sync Mode, the LUI in the scope (the latest LUI, get on each LU), the deployed project name, Globals' values and the active environment.

Fabric Settings

SET Command

The Fabric SET command displays the values Fabric session's settings such as project name, sync mode, scope, and displays the session variables values. Note that the settings with default value are not displayed. They become part of SET command's output only after being set to another value.

Fabric Setting - Session Level

The Fabric SET command enables updating Fabric settings on a session level.

• Set global variables.

• Sync settings:

  • Set sync mode.
  • Set sync timeout.
  • Set ignore source exception.
  • Set always_sync.
  • Set sync_on_demand.

• Set the active environment.

• SET ATTACH_POLICY command, to set the MicroDB attachment policy to the Fabric session. The valid values are:

  • LATEST (default) - Check the Storage for the latest version of the MicroDB. Use the one in cache only if it is the latest.
  • ANY - Use the MicroDB in cache if one exists, bring the MicroDB from Storage if not.
  • TRY_LATEST - Try to bring the latest MicroDB from Storage. If the operation fails, use the one in cache, if exists.
  • The default can be changed in config.ini.

• SET OUTPUT command, set the output format of the query's results.

• SET INSTANCE_TTL command, set the Time To Live (TTL) in seconds for each LUI; the LUI is deleted automatically from Fabric after the set TTL ends. Fabric 7.1 adds the option to avoid saving the LUI in Fabric (instead of save and delete) if the TTL is set to zero. Note that when the TTL is set to zero, Fabric does not clean an older version of the LUI (if it exists). The TTL must be greater than zero in order to clean the LUI from Fabric.

• SET LUI_READ_ONE_WHEN_FAIL command, set the consistency level for the GET LUI command to ONE. If it fails to achieve a QUORUM consistency level, the sync mode is set to OFF.

• SET FROM command, update several settings in one compound command using JSON structure with the following syntax: "set from '{["scope" : {\}],]["attached" : {"\": "\"[ , "": "<LUI>"} , …]}' " . The command is built from two optional parts, each can be omitted:

  • Scope, holds one or more set commands like "sync" or "environment".

  • Attached, the LUs to load into the session. Several LUI can be specified if they are from different LUTs.

  • Examples:

  fabric>set from '{ "attached" : {"Customer": "1", "ORDERS": "4"}}';
  (1 row affected)
  
  fabric>set from '{ "scope" : {"sync": "on", "environment" : "_dev"}, "attached" : {"Customer": "1", "ORDERS": "3"}}';
  (1 row affected)
  
  fabric>set from '{ "scope" : {"sync": "force", "environment" : "UAT1"}}';
  (1 row affected)
  

• SET USER_ROLES command, returns the list of roles of the connected user.

• SET AUTO_MDB_SCOPE command, provides an ability to query the Logical Unit without performing the GET command explicitly ("No Get") when an SQL statement includes a WHERE clause with the filter by IID. The filter must include the field name defined as Instance ID Column of the LU Root Table, otherwise the error message is displayed.

  The following logic is performed on each SQL statement run:

  • Sync the LUI based on the defined sync mode.
  • Execute the query.
  • Release the instance.

  The queries without a filter by IID cannot be executed in this mode. To deactivate it, set AUTO_MDB_SCOPE to false.

  fabric>set auto_mdb_scope=true;
  (1 row affected)

  fabric>select * from CRM.customer where customer_id = 123;
  |CUSTOMER_ID|SSN       |FIRST_NAME|LAST_NAME|HAS_OPEN_CASES|VALIDATIONS_NOT_PASSED|
  +-----------+----------+----------+---------+--------------+----------------------+
  |123.0      |7416713403|Gaynelle  |Gill     |0             |null                  |

  (1 rows)

  fabric>select customer.customer_id, subscriber.contract_id, subscriber.contract_description 
  from CRM.customer, CRM.subscriber where customer.customer_id = 123;
  |CUSTOMER_ID|CONTRACT_ID|CONTRACT_DESCRIPTION|
  +-----------+-----------+--------------------+
  |123.0      |314.0      |5G tether           |
  |123.0      |315.0      |10G LTE             |
  |123.0      |316.0      |450 min             |
  |123.0      |317.0      |Unlimited call      |
  |123.0      |318.0      |Unlimited text      |

  (5 rows)

  fabric>select * from CRM.address where entity_id = 123;
  Cannot execute the query due to missing WHERE clause on the IID column.

Note that this feature enables querying Fabric by various external systems (such as BI) that are not familiar with the Fabric syntax. They can use standard SQL language rather than the Fabric GET command. For external connection to Fabric, AUTO_MDB_SCOPE=true should be set via the Fabric Connection URL.

• SET DEFAULT command, can be used to reset all the related parameters set on a session level to their default value.

• SET DB_PROXY command, can be used to activate an operations' scope toward the specified DB interface, so that until it is turned off, all operations are done against this interface.

  • Syntax: SET DB_PROXY [= <interface name>]

  • Description: Activates an operations' scope toward the specified DB interface, so that until it is turned off, all operations are done against this interface.

  • If interface name is not specified, the command will show the current interface name.

  • To turn it off use: set db_proxy=off.

  • A new parameter was added to config.ini called ENABLE_DB_INTERFACE_PROXY, it is set by default to FALSE. Set it to TRUE to enable using this new command.

• SET BUFFER_RESULT_SET command, enables uploading all the following SELECT statements from Fabric to memory.

  • Syntax: SET BUFFER_RESULT_SET = true
  • To inactivate it, set it back to false.
  • The purpose is to avoid the problem of locked MicroDB SQLite file which can happen in a use case when a GET & SELECT from LU1 is followed by loop on GET & SELECT from LU2.

• SET CLUSTER_DISTRIBUTE_AFFINITY command, to distribute the subsequent Fabric command to the specified affinity.

  • Syntax: SET CLUSTER_DISTRIBUTE_AFFINITY = <AFFINITY>
  • Use ALL to distribute the following Fabric command to all live nodes.

Fabric Setting via JDBC Connection URL

Fabric supports the ability to set the session variables via the Fabric Connection URL by concatenating them to the connection string using the following syntax:

jdbc:fabric://[server:port]?user=[user_name]&password=[password]&[key1]=[value1]&[key2]=[value2]

For example:

jdbc:fabric://[localhost:5124]?user=admin&password=admin&timeout=0&sync=off&auto_mdb_scope=true

This can be used when external systems (such as BI) that are not familiar with the Fabric syntax, are connecting to Fabric. The delimiter can be either & or ;.

Fabric Setting - Cluster Level

Use the SET_GLOBAL command to set an active environment or a global value on a Fabric cluster.

The values are kept in the System DB global_settings table under k2system schema.

Fabric Security and Credentials

Fabric Security Commands

• Master key generation commands used to encrypt LUI data and to encrypt an interface’s details. Click to open the Fabric Devops Security articles and to read more about Fabric Security Hardening.
• Fabric Credentials Commands, a list of commands for setting Fabric credentials like, users, roles, tokens or permissions.

Deploy and Drop Commands

Fabric commands to deploy Fabric implementation and Fabric Environments on the Fabric console.

Drop LU Command

The DROP LUTYPE command deletes LU metadata (LU schema) and its LUIs from Fabric. The DROP command also deletes the the LU from Storage and the related LU entry from the k2_lut_info in the System DB. Once the LU is dropped it should be redeployed to the Fabric server.

Click for more information about Fabric System DB.

Note that this command is used mainly in a Testing environment to restart deployment configurations. In Production, the DROP LUTYPE command and reset.sh script are rarely used. A possible scenario is to clean the environment after a soft launch prior to starting an actual Production run. A Drop is followed by an initial load / migration of the data for the dropped LU.

Drop LU Syntax

DROP LUTYPE [LU Name];

Example:

fabric>DROP LUTYPE Customer;

Fabric Environments and Interfaces

Fabric enables the deployment of Fabric environments and setting active environments on a session or cluster levels.

The interfaces of an active environment can be tested using the TEST_CONNECTION command. To do so, run the TEST_CONNECTION command without parameters to test the connection of all interfaces (DB and none DB) in the active environment.

Run Queries on System DB

CQL queries can be run on Cassandra System DB in the Fabric server using the CQL command only for the selected statement.

Example of CQL command on Cassandra:

fabric>cql select * from k2view_customer.entity;

Click for more information about Cassandra Basic Commands.

Jobs Commands

Get the Fabric jobs' list and status, start, stop, update and resume jobs.

Click for more information about Fabric jobs.

Batch Process Commands

The Batch Process mechanism enables executing different types of Fabric commands in a batch mode on remote Fabric nodes.

Fabric has commands that start an execution, retry and cancel the execution of a batch process and commands that monitor an execution on batch processes.

Note that MIGRATE commands are used as aliases to BATCH commands.

Click for more information about the Fabric Batch Process mechanism.

Process Control

PS and Kill Commands

• The PS command displays the current tasks running on the Fabric cluster, i.e. you can run this command on node1 and view tasks running on node2. The PS command displays different types of tasks like Fabric commands, Fabric Jobs, Web Service and Graphit, Sync processes, Broadway Actor, parser, or user logic. When a task/thread is specified as a parameter it shows its stack trace.

• The KILL command is used to kill any running task displayed by the PS command. Note that you can kill a task that runs on a different node on the Fabric cluster.

Execution Monitoring

The TRACE command enables tracing internal Fabric operations by request and writing them into Tracing files.

Click for more information about the Tracing mechanism.

CommonDB & Reference Tables

Fabric enables creating Reference tables which can be used by all LUs or Web Services. A Reference table typically contains metadata. For example, a Postal Code table that identifies the postal code of customer addresses.

Reference table commands enable synchronizing, getting the sync status and waiting for a sync's processing of Reference tables to be completed before continuing the workflow.

Click for more information about Reference Tables.

Fabric Transactions

Fabric enables running a single transaction on a specific LU table of the Instance ID or on a Reference table. When this functionality is used, Fabric becomes the master of the data rather than syncing data from external systems. This way, Fabric can get transaction feeds and update a related Instance ID or Reference table accordingly. Always start a transaction with a BEGIN command before running INSERT, UPDATE or DELETE commands, and use COMMIT or ROLLBACK commands to commit or rollback the updates.

Fabric has a set of commands that support transactions:

• BEGIN, start a transaction.
• SELECT, UPDATE, INSERT, and DELETE, run Select, Insert, Update and Delete transactions on the LUI or Reference table data.
• COMMIT and ROLLBACK, commit or rollback the updates.

Fabric also enables writing the transaction into a delta table using the SET ASYNC_TRX=true command.

Click for more information about Fabric Transactions.

CDC and Search

The Fabric Change Data Capture (CDC) solution notifies external systems about data changes and has built-in integration with Elasticsearch to enable a cross LUI search.

For example: search all customers called “John Doe” that live in “New-York”.

Fabric has a SEARCH command that initiates a search on Elasticsearch. In addition, the Fabric CDC_REPUBLISH_INSTANCE command can be used to republish CDC data on LUI.

Broadway Command

The Fabric BROADWAY command enables running a Broadway flow and providing the LU name and execution parameters using param=value syntax. The flow can be invoked by a command after it has been deployed. The command syntax is:

broadway <LUT>.<FLOW_NAME> [param1=value1, param2=value2...] RESULT_STRUCTURE=<ROW/COLUMN/CURSOR>

Below are the types of execution parameters:

1. External input arguments of a flow, if they exist.

2. Result Structure enables defining the format of the flow output. The default mode is configurable via config.ini. Three modes exist:

   • COLUMN (default) – The outputs are returned as each output in a column.

   |result |date         |                                                 
   |+-----+--------------+                                                 
   |15     |2022-07-19   |
   

   • ROW – The Broadway flow outputs are returned as each output in a row.

   |column |value       |                                                     
   |+-------+-----------+                                                   
   |result   |15        |                                                   
   |date     |2022-07-19|
   

   • CURSOR – The first flow's output is transformed into a table. Other outputs are being disregarded.

   Case 1: the following output:

   |column   |value     |
   +---------+----------+
   |result   |1,2,3     |
   |date     |2022-07-19|
   

   will be transformed to:

   |result |
   +-------+
   |1      |
   |2      |
   |3      |
   

   Case 2: the following output:

   |column   |value                                                                 |
   +---------+----------------------------------------------------------------------+
   |result   | [{ val1: 1,val2: 2,val3: 3},{ val1: 4,val2: 5,val3: 6}               |
   |date     | 2022-07-19                                                           |
   

   will be transformed to:

   |val1 |  val2  | val3
   +-----+--------+-----
   |1    |  2     |  3
   |4    |  5     |  6
   

3. Recovery parameters:

   • recoveryId, unique ID for running the flow with a recovery point. Flow recovery is enabled only if the recovery ID is supplied.

   • recoveryTtl (optional), time to live in seconds for the recovery point to be kept in the Cassandra broadway_recovery_point table under the k2system keyspace. The default value is defined in the Broadway config.ini section with the RECOVERY_TTL_SEC key.

   • recoveryMaxTries (optional), maximum number of retries until the flow is deleted from the Cassandra broadway_recovery_point table. The default value is defined in the Broadway config.ini section with the RECOVERY_MAX_RETRIES key.

   Click for more information about Broadway Recovery Points.

   • Example of running in a recovery mode:

   fabric>broadway AT_MPI.files_test recoveryId=rl1, recoveryTtl=3000, recoveryMaxTries=5, param1=test1, param2=test2;
   

4. Profiler Telemetry:

   • To invoke the Broadway profiler, set profilerTelemetry to true. This will add the profiler results to the command results, under the profilerTelemetry key.

   Click for more information about Broadway Profiler.

   • Example of running with a profiler:

   fabric>broadway P2.getLUVariable luName='P2' variableName='TDM_TAR_ENV_NAME' profilerTelemetry=true;
   

Queries Helpers

An SQL statement can be preceded by the EXPLAIN keyword or by the EXPLAIN QUERY PLAN phrase. The SQL statement then behaves like a query and returns information about the SQL statement’s operations if the EXPLAIN keyword or phrase is omitted.

EXPLAIN and EXPLAIN QUERY PLAN are intended for interactive analysis and troubleshooting only.

Example:

MDB Export / Import

Fabric supports exporting or importing the MicroDB data from SQLite to another DB type (PostgreSQL). This solution allows performing the Fabric data backup, share it with others or import data from external data sources into Fabric.

The following Fabric commands have been introduced for this purpose:

• MDB_EXPORT without IID

  • In this mode, the schema of the specified Logical Unit (LU) is dropped (if exists) and created again.
  • All pre-defined constraints (such as PKs, FKs, and indexes), are included in the exported schema.
  • The command returns the number of exported tables.
  • Notes:
    • The FK creation is configurable and depends on the command's FK parameter (true/false).
    • A new column named __iid is added to each table, as well as to each table’s primary key (PK).

• MDB_EXPORT with IID

  • In this mode, data is retrieved from the Fabric MDB and written to the pre-created PG schema using the driver provided by the specified interface.
  • The data will be added to the PG tables based on their PKs and FKs, ensuring that the table constraints are respected.
  • The command returns the number of exported rows.

• MDB_IMPORT with IID

  • The command enables the data import from the external PG storage into the Fabric MDB, replacing the existing MDB data.

Both import and export commands can optionally receive a list of tables to be excluded from the import or export process.

MDB Size

Fabric provides the MDB_SIZE command to return the size of one or several MDB. The syntax is:

• Return the size of the specified instance:

MDB_SIZE <LUT_NAME>.'<INSTANCE_ID>'

• Return the size of the listed instances:

MDB_SIZE <LUT_NAME>.(<instance 1,instance 2,etc...>)