Using jcmd
with CrateDB on Docker¶
Most the guides about Java diagnostic utilities explain how to use them on
applications running directly as process on the operating system. Fewer of
them cover how to apply jcmd
commands inside a Docker container.
Introduction
When it comes to troubleshooting a CrateDB instance running inside a container
like Docker, things work a bit differently.
This document explains why the standard way to run jcmd
does not work, and
how to solve it.
Note
The document does not explain how to analyze the output, because that is identical to non-containerized applications.
Table of contents
Run jcmd
inside container¶
The commands below use docker
. In the same spirit, you can also use
podman
.
Connect with user crate
When aiming to invoke jcmd
inside the container, please make sure to
impersonate as the crate
user.
docker exec -it --user=crate cratedb /bin/bash
/crate/jdk/bin/jcmd 1 VM.version
Otherwise, the command will fail with an error like AttachNotSupportedException:
Unable to open socket file /proc/1/root/tmp/.java_pid1
.
Root Cause
The entrypoint of the crate
Docker image ensures that the CrateDB Java
process runs as user crate
, since CrateDB must be run as a non-root
user.
This is done by chroot
ing with user crate
(chroot --userspec=1000 /
"$@"
), because this does not spawn an additional process for changing the
user - unlike su crate -c "$@"
, where su
would result in the process
with PID 1
and the crate command would be a child-process with a different
PID. This is not what one wants in a Docker container, where the application
must (?) run as PID 1.
Examples
Let’s start by starting a cratedb
container.
docker run --rm -it --name=cratedb \
--publish=4200:4200 --publish=5432:5432 \
--env=CRATE_HEAP_SIZE=2g crate/crate:nightly \
-Cdiscovery.type=single-node
docker exec -it --user=crate cratedb /bin/bash
$ /crate/jdk/bin/jcmd -l
1 io.crate.bootstrap.CrateDB -Cpath.home=/crate -Cdiscovery.type=single-node
301 jdk.jcmd/sun.tools.jcmd.JCmd -l
[crate@cratedb data]# /crate/jdk/bin/jcmd 1 VM.version
1:
OpenJDK 64-Bit Server VM version 13.0.1+9
JDK 13.0.1
jcmd <PID> help
lists all available commands that you can now start using
for troubleshooting CrateDB inside the Docker container.
[crate@cratedb data]# /crate/jdk/bin/jcmd 1 help
1:
The following commands are available:
Compiler.CodeHeap_Analytics
Compiler.codecache
Compiler.codelist
Compiler.directives_add
Compiler.directives_clear
Compiler.directives_print
Compiler.directives_remove
Compiler.queue
GC.class_histogram
GC.class_stats
GC.finalizer_info
GC.heap_dump
GC.heap_info
GC.run
GC.run_finalization
JFR.check
JFR.configure
JFR.dump
JFR.start
JFR.stop
JVMTI.agent_load
JVMTI.data_dump
ManagementAgent.start
ManagementAgent.start_local
ManagementAgent.status
ManagementAgent.stop
Thread.print
VM.class_hierarchy
VM.classloader_stats
VM.classloaders
VM.command_line
VM.dynlibs
VM.events
VM.flags
VM.info
VM.log
VM.metaspace
VM.native_memory
VM.print_touched_methods
VM.set_flag
VM.stringtable
VM.symboltable
VM.system_properties
VM.systemdictionary
VM.uptime
VM.version
help
For more information about a specific command use 'help <command>'.
To execute one of these commands from outside of the Docker container without
explicitly attaching to it, you can combine the docker exec
command with the
jcmd
command.
$ docker exec -it --user=crate cratedb /crate/jdk/bin/jcmd 1 <CMD>
For example, running GC.heap_info
on Docker container with ID
cratedb
.
$ docker exec -it --user=crate cratedb /crate/jdk/bin/jcmd 1 GC.heap_info
1:
garbage-first heap total 524288K, used 129716K [0x00000000e0000000, 0x0000000100000000)
region size 1024K, 126 young (129024K), 22 survivors (22528K)
Metaspace used 57165K, capacity 59755K, committed 60080K, reserved 1099776K
class space used 7721K, capacity 8941K, committed 8960K, reserved 1048576K
Troubleshooting Commands¶
These are the most common troubleshooting tasks, but of course there are many
more possibilities to get diagnostic information using the jcmd
command.
You can find more information about the utility at the jcmd documentation.
Heap Info¶
- Command:
jcmd <PID> GC.heap_info
Example
$ docker exec -it --user=crate cratedb /crate/jdk/bin/jcmd 1 GC.heap_info
1:
...
Heap Dump¶
- Command:
jcmd <PID> GC.heap_dump <PATH>
Example
$ docker exec -it --user=crate cratedb /crate/jdk/bin/jcmd 1 GC.heap_dump /data/crate.hprof
1:
Heap dump file created
Note
The <PATH>
should be a path that resides on a mounted volume, so you can
access the created heap dump from ouside of the container and the container
is not “blown up”.
Thread Dump¶
- Command:
jcmd <PID> Thread.print
Example
$ docker exec -it --user=crate cratedb /crate/jdk/bin/jcmd 1 Thread.print
1:
...
Java Flight Recorder (JFR)¶
- Command:
jcmd <PID> JFR.start name=<NAME> duration=<DURATION> filename=<PATH> settings=profile
Example
$ docker exec -it --user=crate cratedb /crate/jdk/bin/jcmd 1 JFR.start name=recording1 duration=60s filename=/data/recording1.jfr
1:
Started recording 1. The result will be written to:
/data/recording1.jfr
Note
The <PATH>
should be a path that resides on a mounted volume, so you can
access the created jfr dump from ouside of the container and the container
is not “blown up”.
Notes¶
Previously, you needed to use a chroot
command inside the container, in
order to invoke the jcmd
command properly. While this procedure still works,
it is more convenient to use the --user
option as outlined above.
In case you need the old variant, we are displaying it below.
[user@host ~]# docker exec -it cratedb /bin/bash
[crate@cratedb data]# chroot --userspec=1000 / /crate/jdk/bin/jcmd 1 VM.version
1:
OpenJDK 64-Bit Server VM version 13.0.1+9
JDK 13.0.1
docker exec -it cratedb chroot --userspec=1000 / /bin/bash
/crate/jdk/bin/jcmd 1 VM.version