Fabric Troubleshooting Overview
Fabric provides the following methods for troubleshooting executed processes.
Log files. All activities performed in Fabric are written into log files in the server. In addition, the activities run on the Fabric debug server started by the Studio, are written to the Log screen in the Fabric Studio. The log messages display the failed LU and Table Population names in case of failure.
Monitoring stuck processes, using the following tools:
PS command – a Fabric command like User Jobs, Web Service, Graphit or a Sync process which displays tasks running on the Fabric cluster. The PS command can be used to identify stuck processes and their running duration and when needed, to kill stuck processes using the kill command.
jjstack.sh script – a Fabric script that collects Java stack traces for a given process, stores the stacks and analyzes the results. The script can be applied to the Fabric server or the IID Finder.
Monitoring memory leaks or intensive memory consumption using a Heap Dump file which is created either automatically during a Fabric crash when memory usage exceeds the definition, or manually on demand. Heap Dump files can be investigated to analyze the source of a memory leak.
How Do I Run jjstack?
Use cases for running jjstack.sh are:
- An explicit request from R&D to enable the investigation of a problem in Fabric.
- When you suspect that a job or a Migrate command is stuck.
- To investigate a performance issue in the implementation layer.
When running jjstack, first run the script and then analyze the output either on your own or with the help of R&D.
The following table describes the syntax and parameters for calling the jjstack.sh script. The script is located under $K2_HOME/fabric/scripts in the Fabric server.
Click to view the Example of the jjstack Output File.
How Is a Heap Dump File Created?
A Heap Dump file is automatically created when Fabric crashes due to memory usage exceeding the definition. When required, a Heap Dump file can also be created manually using the jmap command.
Default memory usage is defined in the $K2_HOME/config/ jvm.options configuration file and is equal to 2G. The location of a Heap Dump file is defined in the jvm.options configuration file. If it is not defined there, it is created in the folder where Fabric is started, which is usually $K2_HOME or $K2_HOME/fabric/scripts.
Note that Heap Dump files can take up a lot of disk space, therefore it is recommended to delete them after an investigation has been completed. Head Dump files should be uploaded to K2view sftp dedicated servers in order to allow their analysis by the R&D team.
The following table describes the syntax and parameters for creating the Heap Dump file using the jmap command.
Fabric Troubleshooting Overview
Fabric provides the following methods for troubleshooting executed processes.
Log files. All activities performed in Fabric are written into log files in the server. In addition, the activities run on the Fabric debug server started by the Studio, are written to the Log screen in the Fabric Studio. The log messages display the failed LU and Table Population names in case of failure.
Monitoring stuck processes, using the following tools:
PS command – a Fabric command like User Jobs, Web Service, Graphit or a Sync process which displays tasks running on the Fabric cluster. The PS command can be used to identify stuck processes and their running duration and when needed, to kill stuck processes using the kill command.
jjstack.sh script – a Fabric script that collects Java stack traces for a given process, stores the stacks and analyzes the results. The script can be applied to the Fabric server or the IID Finder.
Monitoring memory leaks or intensive memory consumption using a Heap Dump file which is created either automatically during a Fabric crash when memory usage exceeds the definition, or manually on demand. Heap Dump files can be investigated to analyze the source of a memory leak.
How Do I Run jjstack?
Use cases for running jjstack.sh are:
- An explicit request from R&D to enable the investigation of a problem in Fabric.
- When you suspect that a job or a Migrate command is stuck.
- To investigate a performance issue in the implementation layer.
When running jjstack, first run the script and then analyze the output either on your own or with the help of R&D.
The following table describes the syntax and parameters for calling the jjstack.sh script. The script is located under $K2_HOME/fabric/scripts in the Fabric server.
Click to view the Example of the jjstack Output File.
How Is a Heap Dump File Created?
A Heap Dump file is automatically created when Fabric crashes due to memory usage exceeding the definition. When required, a Heap Dump file can also be created manually using the jmap command.
Default memory usage is defined in the $K2_HOME/config/ jvm.options configuration file and is equal to 2G. The location of a Heap Dump file is defined in the jvm.options configuration file. If it is not defined there, it is created in the folder where Fabric is started, which is usually $K2_HOME or $K2_HOME/fabric/scripts.
Note that Heap Dump files can take up a lot of disk space, therefore it is recommended to delete them after an investigation has been completed. Head Dump files should be uploaded to K2view sftp dedicated servers in order to allow their analysis by the R&D team.
The following table describes the syntax and parameters for creating the Heap Dump file using the jmap command.
