Transcript Slides - Javaday

THE ART AND
METHODOLOGY OF
TROUBLESHOOTING
Alexey Diomin, [email protected]
Production
Agenda



Logging
Offline analysis
Online analysis
Monitoring

System monitoring
 Zabbix
 Nagios

Application Monitoring
 New
Relic
 Logstash
 Kibana
Zabbix. CPU load example
Zabbix. Memory example
Zabbix. CMS Old Gen
Logstash. Kibana
“We need to go deeper”
“We need to go deeper”

jstack (thread dump)
 jstack
<pid>
 kill -QUIT <pid>
jstack


Very simple profiler
Lock detection
jstack example
“We need to go deeper”

jstack (thread dump)
 jstack
<pid>
 kill -QUIT <pid>

jmap
 jmap
-dump <pid>
 jmap -histo <pid|file>
jmap

Eclipse Memory Analyzer (MAT)

http://www.eclipse.org/mat/

jvisualvm

http://visualvm.java.net/
MAT
MAT
Leak Detection
RingBuffer
MAT
“We need to go deeper”

jstack (thread dump)
 jstack
<pid>
 kill -QUIT <pid>

jmap
 jmap
-dump <pid>
 jmap -histo <pid|file>

JFR / JMC
JFR
JFR
Event Type
JFR
Basic features:
 Code Profile
 Allocation profile
JFR
Basic features:
 Code Profile
 Allocation profile
Killer features:
 Contentions
 Exceptions rate
Code profile
Allocation profile
Netty’s bug example
Contention
Hbase latency
Contentions example
Exceptions rate
Context Switch
Exception example

1.
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException
“You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax
to use near ….”
SQL syntax for prepared statements does not support multi-statements (that is, multiple
statements within a single string separated by “;” characters).
2.
“Can't create more than max_prepared_stmt_count statements”
useServerPrepStmts
Use server-side prepared statements if the server supports them
useServerPrepStmts
Summary



Logging
Offline analysis
Online analysis
Links

Java Performance: The Definitive Guide

http://shop.oreilly.com/product/0636920028499.do

Troubleshooting Guide for Java SE 6 with HotSpot VM

http://www.oracle.com/technetwork/java/javase/tsg-vm149989.pdf
Black boxes?
Not in troubleshooting!!!