Big Logical Units
Fabric enables storing big LUIs without size limitations by splitting data into chunks. The SQLite file's chunks are written into the System DB entity_chunks table in parallel.
When the System DB is Cassandra, the Cassandra Loader is used. The Loader configuration for the parallel save can be done using the config.ini by adding a section named [LU type]_ cassandra_entity_storage per each LU. The parameters under this section are the same as the Cassandra Loader definition parameters (for example, Loader execution mode).
The LUI data is first written into the entity_chunks table, and then, after all chunks were successfully written, the entity table is populated.
The entity table includes the following:
- batch_id - a unique ID that represents the relationship between an entity and entity_chunks tables.
- chunks_count - the number of chunks.
- data - holds the LUI SQLite file after compression. For performance optimization during parallel data inserts, when the number of chunks > 1, the SQLite file size is set to zero.
The entity_chunks table includes the following data:
- id - holds the instance ID.
- sync_version - holds the same version that is populated in the entity table.
- batch_id - holds the same ID as in the entity table.
- chunk_index - holds the chunk number.
- data - holds the split SQLite file after compression, by the chunk index.
The chunk size is set using the config.ini file parameters, defined per node:
- INSTANCE_CHUNK_SIZE - impacts the read from System DB (pagination) and write into System DB chunk size, whose default is 10MB (10,485,760 bytes).
Improve LUI Partitioning
When dealing with a large amount of entity chunks, LUI Partitioning can be enabled by setting the following parameter in the config.ini file: ENABLE_PARTITIONED_MDB=true
Note that there is no upgrade path for existing projects. You must clean all data in Fabric and bring Fabric back up. It is recommended to turn this feature on when dealing with big LUIs that are split into multiple chunks.
Parallel Load of Big LUI from System DB
When loading the chunks of big LUIs from System DB to Fabric, as part of the GET command, there is a trade-off between the performance of the load and the memory allocated to this process. To improve the performance of the load, you can define the number of threads that will be executed in parallel. When setting the number of threads, you must also define the maximum memory allowed to be used for the parallel load.
The config.ini parameters for configuring the above are:
- ASYNC_LOAD_MAX_THREADS - the maximum number of threads (over all of the Fabric nodes) to be allocated. The default setting is zero, which means that the parallel load is disabled.
- ASYNC_LOAD_MAX_MEMORY_IN_MB - the maximum memory to be allocated for the parallel load process. The default setting is 2000 MB.
These parameters are applicable when the LUI Partitioning is enabled.
Big Logical Units
Fabric enables storing big LUIs without size limitations by splitting data into chunks. The SQLite file's chunks are written into the System DB entity_chunks table in parallel.
When the System DB is Cassandra, the Cassandra Loader is used. The Loader configuration for the parallel save can be done using the config.ini by adding a section named [LU type]_ cassandra_entity_storage per each LU. The parameters under this section are the same as the Cassandra Loader definition parameters (for example, Loader execution mode).
The LUI data is first written into the entity_chunks table, and then, after all chunks were successfully written, the entity table is populated.
The entity table includes the following:
- batch_id - a unique ID that represents the relationship between an entity and entity_chunks tables.
- chunks_count - the number of chunks.
- data - holds the LUI SQLite file after compression. For performance optimization during parallel data inserts, when the number of chunks > 1, the SQLite file size is set to zero.
The entity_chunks table includes the following data:
- id - holds the instance ID.
- sync_version - holds the same version that is populated in the entity table.
- batch_id - holds the same ID as in the entity table.
- chunk_index - holds the chunk number.
- data - holds the split SQLite file after compression, by the chunk index.
The chunk size is set using the config.ini file parameters, defined per node:
- INSTANCE_CHUNK_SIZE - impacts the read from System DB (pagination) and write into System DB chunk size, whose default is 10MB (10,485,760 bytes).
Improve LUI Partitioning
When dealing with a large amount of entity chunks, LUI Partitioning can be enabled by setting the following parameter in the config.ini file: ENABLE_PARTITIONED_MDB=true
Note that there is no upgrade path for existing projects. You must clean all data in Fabric and bring Fabric back up. It is recommended to turn this feature on when dealing with big LUIs that are split into multiple chunks.
Parallel Load of Big LUI from System DB
When loading the chunks of big LUIs from System DB to Fabric, as part of the GET command, there is a trade-off between the performance of the load and the memory allocated to this process. To improve the performance of the load, you can define the number of threads that will be executed in parallel. When setting the number of threads, you must also define the maximum memory allowed to be used for the parallel load.
The config.ini parameters for configuring the above are:
- ASYNC_LOAD_MAX_THREADS - the maximum number of threads (over all of the Fabric nodes) to be allocated. The default setting is zero, which means that the parallel load is disabled.
- ASYNC_LOAD_MAX_MEMORY_IN_MB - the maximum memory to be allocated for the parallel load process. The default setting is 2000 MB.
These parameters are applicable when the LUI Partitioning is enabled.
