Troubleshooting
Diagnostics & Incident Playbook Root Cause Analysis (RCA)

Apache Tomcat Troubleshooting & Root-Cause Playbook

Instant resolution steps for JVM memory exhaustion, hung threads, port conflicts, socket starvation, and crash dump analysis.

1. Diagnosing java.lang.OutOfMemoryError Variants
java.lang.OutOfMemoryError: Java heap space High Urgency

Root Cause: The application instantiated objects exceeding -Xmx, or retained references preventing Garbage Collection (heap memory leak).

Remedy:

  1. Ensure -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/tomcat/dumps/ is active in setenv.sh.
  2. Analyze the resulting .hprof file with Eclipse MAT (Memory Analyzer Tool) or VisualVM. Check the Dominator Tree for leaked static collections, unclosed sessions, or unindexed database query result sets.
  3. Increase -Xmx using our Sizing Engine if valid payload load exceeds capacity.
java.lang.OutOfMemoryError: Metaspace Classloader Leak

Root Cause: Dynamic class loading, cglib/byte-buddy proxies, or application redeployments without restarting Tomcat causing ClassLoader pinning.

Remedy:

  1. Avoid repeated hot-redeployments in production without container restarts.
  2. Increase -XX:MaxMetaspaceSize (e.g. from 512M to 768M/1024M for large enterprise frameworks like Spring Boot/Hibernate).
  3. Inspect jcmd <PID> VM.classloaders to inspect lingering classloaders.
java.lang.OutOfMemoryError: Direct buffer memory Off-Heap NIO

Root Cause: Heavy off-heap direct allocations (Netty, NIO channel transfers, unclosed ByteBuffers) exceeding -XX:MaxDirectMemorySize.

Remedy: Explicitly size -XX:MaxDirectMemorySize=512m or higher, and verify ByteBuffer lifecycle management.

2. Essential JVM Diagnostic Commands
Real-time runtime inspection using standard JDK tools
# 1. Find the Tomcat process PID
pgrep -f "org.apache.catalina.startup.Bootstrap"

# 2. Capture a live Thread Dump (3 iterations separated by 5 seconds to spot hung threads)
for i in 1 2 3; do
    jcmd <PID> Thread.print > thread_dump_$i.txt
    sleep 5
done

# 3. Trigger manual Heap Dump for memory leak analysis
jcmd <PID> GC.heap_dump /var/log/tomcat/dumps/manual_dump_$(date +%s).hprof

# 4. View real-time Metaspace & GC statistics
jstat -gcutil <PID> 1000 10

# 5. Inspect Native Memory Tracking (if -XX:NativeMemoryTracking=summary is enabled)
jcmd <PID> VM.native_memory summary
3. Port Binding & Thread Starvation Triage

java.net.BindException: Address already in use

Occurs when port 8080, 8443, or 8005 is occupied by a lingering prior instance or another service.

# Linux: Identify and kill the lingering socket holder
sudo ss -tulpn | grep ':8080'
# Or using lsof:
sudo lsof -i :8080

# Windows PowerShell:
Get-NetTCPConnection -LocalPort 8080 | Select-Object OwningProcess, State
Stop-Process -Id <PID> -Force

Thread Starvation / Request Latency Spikes

If requests hang or timeout with HTTP 504 from the load balancer, Tomcat's connector thread pool (maxThreads="200") may be saturated waiting on slow database queries, external REST calls without timeouts, or Java deadlocks.

Paste your thread dump into our built-in Thread Dump Analyzer Tool → for immediate automated detection of thread contention, blocked states, and monitor deadlocks.