Transcript Template

Mobilab
Dependability analysis of the Java Virtual Machine
Salvatore Orlando
Ph.D. student
University of Naples Federico II
[email protected]
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
1
::. Presentation
Salvatore Orlando
Ph.D. Student at the University of Naples “Federico II”
MobiLab Research Group member
Advisor: Prof. Stefano Russo
Ph.D. Research Programme:
Dependability attributes of Virtual Execution Environments
Current work:
Java Virtual Machine monitoring for dependability assessment
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
2
::. Outline
• Rationale
• Research work about the Java Virtual Machine
• Research work about dependability issues of the JVM
• Why focus on Sun Hotspot JVM
• Internal architecture of the Sun Hotspot® 5.0 JVM
• JVM Monitoring for dependability assessment
• Failures and sources of errors in the JVM
• Our Approach
• Monitoring and management technologies (JSR-174 & JSR-163)
• JMX (Java Management Extensions) and the platform MBeans
• JVMTI (JVM Tool Interface)
• The distributed monitoring system
• Architecture of the monitoring system
• Details of the JVM monitoring subsystem
• Data Analysis methodology
• Future work
• Assessing JVM dependability
• Augmenting the JVM with Fault Tolerance
• Moving to the «mobile side»
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
3
Advices,
Hints,
Questions and
Criticisms
are welcome!
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
4
::. Just a bit of history… (of the evolution of the Java Platform)
Java is born, but it
is only used for web
applets and small
digitally controlled
consumer devices.
Java is still
at an
embryonal
state. Only
after 4 years
Java is
unveiled to
the
community
1991
1993
www.mobilab.unina.it
The Java Platform
begins to be
widely used,
especially in
embedded
devices
1995
1997
Always more
enterprises behave on
the Java Platform for
their corporate
applications
Java is starting to be
used in missionJava becomes
critical and
one of the most
dependable
important
applications
platforms for
cellular phones
Java is now
widely employed
also in web
applications
1999
2001
2003
The dawning of Java History
Yesterday’s History
Nowadays history
[email protected]
2005
JVM Dependability Analysis
5
::. Just a bit of history… (Last Milestones and near future)
October 1, 2004: J2SE 5.0 Released
Waiting for J2SE 6.0 release and the MVM
(Multitasking Virtual Machine)
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
6
::. Internal architecture of the Sun Hotspot® 5.0 JVM
VM Runtime
•The VM Compilation
Runtime
includes the core components of the JVM.
Unit
It is in charge of:
JVMTI
Compilation Policy Manager
Deoptimizer
VM
core
JNI
Class Loader
•Dispatching andServer
executing
Bytecode instructions
Client
Interpreter
Compiler
Compiler
runtime
•Managing threads
Runtime and
Runtimeobjects’ locks
•…
Server
Client
(Opto)
(C1)
Interpreter and profiling
•Collecting dataCompiler
for debugging
Compiler
Shared Runtime
OS Interface
•Handling the following components
OS MMU
Assembler
Timer
Synchronizer
Management Services
OS Thread
I/O
Time
•The Memory Management Unit is in charge of:
•Allocating oops (i.e.: a Class, a method, an object…)
•Managing objects finalization
Concurrent
•Managing objects generations
Serial
Low-Pause
•Performing Garbage Collection
Garbage Collectors
•…
Throughput
Permanent Generation
•The Compilation unit is in charge of:
Young Generation
Tenured
Generation
Method Area
Thread-specific
data
(handleArea,
ResourceArea)
Memory
Management
Unit
Native
method
stack
Method
stack
•Discovering «Hotspots» in Java Applications
frame
SharedHeap
•Compiling or interpreting a Java
method
PC
Registers
•Transforming intermediate bytecode
representation
into
native
assembly code
JVM Heap
•…
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
7
::. Rationale
• Why focus on the Sun Hotspot® JVM
Many implementations of the Java Virtual Machine
specification available:
Jikes RVM
Kaffe
MacOS RT for Java
Blackdown
JRockit
…
But…
The Sun’s JVM represents the most
widespread implementation
The Sun’s JVM is the most complete (and complex)
JVM implementation
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
8
::. A little snapshot of the research about the JVM
Period: 2000-2005
Data from 114 articles
published in IEEE and
ACM conferences and
reviews
6%
23%
14%
17%
9%
11%
VM optimizations and technological improvements
Performance measurements and improvements
Real-time
Fault tolerance
www.mobilab.unina.it
20%
Security Issues
JVM-tools
Hardware JVM/Java-based OS
[email protected]
JVM Dependability Analysis
9
::. Research about fault tolerance features of the JVM
Only a little part of the research work on the Java Virtual
Machine addresses dependability issues
The greater part of these works are focused on specific
components of the JVM:
… techniques for protecting unsafe operations in the
garbage collector
… techniques for protecting objects against memory
errors
The only works that address fault tolerance in the entire
virtual machine make use of state machine replication
(Hypervisor-based Fault tolerance) to build a fault-tolerant
JVM and it has been carried out at University of Texas in Austin
and Technion Israel Institute of Technology
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
10
::. Some clues about sources of errors in the JVM
2,4%
0,8%
1,6%
3,2%
4,8%
7,9%
11,1%
52,7%
11,1%
22,3%
11,1%
19,0%
15,1%
11,9%
VM C ore
Synchronizer
Deoptimizer
www.mobilab.unina.it
Shared Runtime
C lassloader
Interpreter
Runtime Unit
Garbage C ollector
JNI
JIT C ompiler
Memory Management Unit
25,0%
OS Interface
Shared Heap
Optimizing JIT C ompiler
C ompilation Unit
Data taken from Sun
and Jikes Bug
Databases.
99 VM-related
segnalations selected
• Among over 500 bug
reports related to the
virtual
machine,
we
selected only the ones
addressing non alwaysreproducibile failures or
failures reproducible only
with specific workloads.
• Then
we
furtherly
refined
the
selection
deleting reports with few
details or clearly related
to bugs in source code.
[email protected]
JVM Dependability Analysis
11
::. At the present date…
It
is recognized
that hardware
faults
longer ainto
major
Most
of the research
work related
to are
faultno
tolerance
thesource
JVM of
faults.
greater
part ofcomponents
faults in modern
are concerned with
focusedThe
only
on singular
of thesystems
Virtual Machine,
software.
especially the memory management unit
The
JVM is becoming
more and
more
complex,itbeing
nowadays
By analyzing
failure reports
in Bug
Databases
is possible
to argue
almost
to a real operating
system
that thecomparable
memory management
unit is not
the only (and not the major)
source
errors inwidely
the JVM
Java
is of
nowadays
used in enterprise software systems
Dependability
issues to
of the
Virtual Machine
have never been
Java is starting
be Java
employed
in mission-critical
systems.
completely
fleshed
out,been
sinceused
a thorough
study
its dependability
For instance,
Java has
to develop
theof
ROver
Sequence Editor
has
never
been
done
(ROSE), a
component of the Rover Sequencing on Visualization
Program
(RSVP),
to control
the
Spirit robot
the exploration
of
Summarizing,
weused
argue
that it’s
necessary
to in
focus
on
Mars
the virtual machine as a whole, taking into account the
role played by each of its components
yet…
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
12
::. Failures,sources of errors and error propagation in the JVM (1/5)
Java
Application
Objects
Accumulated
errors
Compilers
Garbage
Collector
Java
Exception
Handler
VM-related
errors
Error
manifestation
Errors in
Virtual Machine
components
Accumulated Errors
www.mobilab.unina.it
When an error occurs in an Internal
Component of the Java Virtual Machine:
Application
Failures
VM Core
JVM layer
Errors in JVM Components
Applicationrelated
errors
Memory
Subsystem
Synchronizer
Application Service Interface
JNI
Virtual Machine Service Interface
Operating System Service Interface
Class Loader
Errors in
Virtual Machine
components
Application Layer
Sources of
errors
• If it is intercepted by the
Java Exception Handler, The
error is thrown as a Java
Exception. We can observe
Java exception messages such
as OutOfMemory or
IllegalMonitorState
• If the Java Exception
handler could not intercept it,
the Error leads directly to an
Application failure, so we get
OS-level error messages (or
Internal VM error messages)
such as SIGSEGV or SIGBUS
[email protected]
JVM Dependability Analysis
13
::. Failures,sources of errors and error propagation in the JVM (2/5)
31,7%
A Java exception is thrown
18,3%
The JVM dies with no
messages or hangs
11,0%
39,0%
JAVA_EXC EPTION
Data taken from Sun
and Jikes Bug
Databases.
99 VM-related
segnalations selected
www.mobilab.unina.it
OS_MESSAGE
VM_ERROR
HANG_SILENT_C RASH
The JVM dies with an
internal error message
i.e.: InternalError,
AssertionFailure
The JVM dies with an
Operating System error
message
i.e.: SIGSEGV, SIGBUS,
Access Violation
[email protected]
JVM Dependability Analysis
14
::. Failures,sources of errors and error propagation in the JVM (3/5)
Java
Application
Objects
Accumulated
errors
Garbage
Collector
Applicationrelated
errors
Java
Exception
Handler
Application
Failures
VM-related
errors
Memory
Subsystem
Error
manifestation
VM Core
Synchronizer
Errors in
Virtual Machine
components
Accumulated Errors
JVM layer
www.mobilab.unina.it
• Developer’s mistakes (we assume that
FileNotFoundException
programs are written
by skilled
developers…)
• Unexpected conditions in the
InternalError
Application
Sources of
errors
VM related Ambiguous
Compilers
ArrayIndexOutOfBoundEx
ception
Exceptions are raised
due to:
Application
related
Errors in
Virtual Machine
components
Application Service Interface
JNI
Virtual Machine Service Interface
Operating System Service Interface
Class Loader
OutOfMemoryError
Some of these exceptions are clearly
application-related, others are VM (or
OS) related and
others ambiguous: it is
IllegalMonitorStateExc
not clear if the eption
exception is due to errors
generated at the application or at the
ConcurrentModification
VM layer.
Exception
Application Layer
[email protected]
JVM Dependability Analysis
15
::. Failures, sources of errors and error propagation in the JVM (4/5)
• When an ambiguous exception is raised, discriminating whether
it is due to a VM-related error or an application-related one it is an
hard task.
• Indeed,
Moreover,
propagation
phenomena
be to:
observed in the Virtual
anerror
ambiguous
exception
could becould
related
Machine, since errors in JVM could lead to error in other components and/or
• An unexpected
condition
in the
application
(i.e.: a date has a
application
failures (as
shown in
previous
example)
format different from the expected one)
• Errors in file system (i.e.: a required file does not exist)
• Errors in hardware resources (i.e: a device cannot be opened)
• Errors in the Virtual Machine
(i.e.: a ConcurrentModificationException is thrown on an object
even if no thread is operating on that object)
•The data analysis methodology will have to deal with this phenomenon, so
•We
going toalgorithms
work with “trusted''
i.e. applications that are
errorare
coalescing
have to beapplications,
implemented
known to be bug-free and tested to work fine under stress condition.
• In this way we remove ambiguous and application-related exceptions.
Such exceptions are always due to an error in the VM or in the underlying
OS. By analyzing OS logs we should be able to distinguish VM-related
exceptions from OS-related ones.
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
16
::. Failures,sources of errors and error propagation in the JVM (5/5)
1
Compilers
Garbage
Collector
Errors in
Virtual Machine
components
Java
Application
Objects
Accumulated
errors
2
5
4
Error
manifestation
Errors in
Virtual Machine
components
Accumulated Errors
www.mobilab.unina.it
2. An application thread calls this
method. An exception is raised since it
contains bytecode for objects belonging
to another class
3. The error propagates to the VM core
components since the wrong VM
operations are scheduled
VM-related
errors
VM Core
JVM layer
1. A fault in the compiler cauase the
wrong version of a compiled method to
be loaded
Application
Failures
3
Synchronizer
A sufficiently realistic example
Applicationrelated
errors
Java
Exception
Handler
Memory
Subsystem
Application Service Interface
JNI
Virtual Machine Service Interface
Operating System Service Interface
Class Loader
Sources of
errors
4. Other exceptions (i.e.:
InternalError) are raised
5. The user observes an application
failure (the VM has crashed)
Application Layer
[email protected]
JVM Dependability Analysis
17
::. Approach (1/2)
Monitoring the behavior of JVM components in order to gather fieldbased data to be able to
….performing dependability analisys
….identifying dependability bottlenecks
….analyzing pathological behaviors
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
18
::. Approach (2/2)
Our first goal:
Monitor Internal JVM Behavior
Use built-in reporting
and logging features
(e.g.: -verboseGC,
failure reports)
Use standardized
monitoring
technologies
(JSR-3, JSR-163)
• Not enough information available
• Data format strictly dependent
from the particular implementation
www.mobilab.unina.it
Manually instrument
JVM source code
• Very hard task (21.336.425
bytes in 1.455 files)
• High risk of bug injection
• The work has to be entirely
repeated to monitor another
implementation of the JVM
[email protected]
JVM Dependability Analysis
19
::. Monitoring and management technologies (2/4)
JMX (Java Management Extensions)
MBean
server
Management
applications
It isto
a registry
for MBeans.
They connect
the managed
JVM
It
provides
the
services
for
through its connectors.
manipulating
MBeans
In this way
these application
can
use Agent Services MBeans and
resource MBeans
to retrieve
Agent services
information
or manage
Applications
Control
the resources
and the JVM
anditself
make them available
to remote management
applications. (i.e:
sending
notification
Connectors
when
the value
of the
Listeners
for requests
managed
resource
from remote
changes)
Management
MBeans
(Management
applications.
Beans)
They instrumentation
use RMI
Provide
for technology
Managed resources
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
20
::. Monitoring and management technologies (3/4)
JPPA (Java Platform Profiling Architecture)
Platform MBeans (MXBeans)
•
•
•
•
MBeans registered in the MBean Server at bootstrap time
They provide data relative to internal VM state (aggregate and detailed values)
They provide Functions for managing the virtual machine
They are accessible through a JMX-compliant management application (i.e.:Jconsole)
JVMTI Functions
www.mobilab.unina.it
Monitored
JPPA-compliant
JVM
JVMTI
events
[email protected]
JVMTI
Agent
• A JVMTI agent is a shared library that
starts along with the JVM
• In the JVMTI agent there are defined
Callback functions for internal JVM Events
• Events can be dinamycally enabled/disabled
• The greater part or callbacks can use
JVMTI functions for:
•examining internal JVM state
•Managing the JVM
Callbacks
JVMTI
(Java Virtual Machine Tool Interface)
JVM Dependability Analysis
21
::. Distributed Monitoring System Architecture (1/2)
JMX Server
Connector
RMI Connection
JMX Client
Connector
Monitor
MBeans
Platform
MBeans
Notify
MBeans
Monitored JVM
JVMTI Agent
Benchmark App
Used to send notification to the Data
Collector and to query JVM state from
the Monitored JVM
XML state
description
processing
Retrieve data
JMX
management
App
Notifications
Data Collector
Data Collection
• State updated when a
notification is received
• Event notification is
written to a Database
• Interested JVM
component(s) state is
written to an XML file
Collected data
JVM Monitoring
Subsystem
• Event filtering
• State change
notifications are
sent
www.mobilab.unina.it
Data Analyzer
Event Database
XML state
descriptors
• Each entry in the database
represent an event notified by a
JVM
• Each entry is linked to an XML file
describing the state of the
component affected by the above
mentioned event
[email protected]
JVM Dependability Analysis
22
::. Distributed Monitoring System Architecture (2/2)
Monitored JVM & Data Collector: Interaction diagram
JVMTI Agent
Monitor MBean
2. Component state updated
locally
1. Event raised
3. Periodically checks for changes
in component state
Notify MBean
5. Query
Component
state
(performs event filtering)
Data Collector
4. Send notification to
the data collector
6. Update
DataBase
www.mobilab.unina.it
[email protected]
JVM Dependability
23
::. JVM monitoring subsystem details
NotifyMBeans
(JMX
– agent
services
level) to
observe
MonitorMBeans
and
send
TheJVMTI
Collector
queries
the
Monitor
MBeans
update
theare
descriptors
ofevents
the
The
Callback
MonitorMBeans
functions
agent
(JMX
MonitorAgent
use
–
JVMTI
resource
Function
starts
instrumentation
along
to
gather
with
informations
the
level)
JVM
and
updated
handles
about
by
the
notification
to
the Collector
when there are relevant changes in the statestate
of
state
of
the
virtual
Machine
raised
of
MonitorAgent
the
virtual
by
the
machine
JVMTI
through
back-end
global
references
in
the
virtual
in
JNI
machine
environment.
A
Monitor
the MonitorMBeans, which in turns reflect the state of the various
MBean
is defined
for
each component of the JVM.
components
of the
JVM
2) State data gathered
by Callbacks
in Monitor Agent
5
JVMTI Functions
Notifications
to Collector
4
Notify
MBeans
3) MonitorMBeans
updated
4) NotifyMBeans send
notifications to the
Collector
5) Collector Queries
MonitorMBeans
www.mobilab.unina.it
2
Monitored
JPPA-compliant JVM
Callback
Functions
1) Events handled
by Callbacks
in Monitor Agent
Queries
from
Collector
Running
Application(s)
Monitor
MBeans
3
JNI Environment
1
JVMTI
events
MonitorAgent
Global
references to
MonitorMBeans
[email protected]
JVM Dependability Analysis
24
::. Distributed Monitoring System: collected data details (1/3)
Class Loader
• JVMTI Events intercepted
ClassLoad (Class File read) – ClassPrepare (Verification process completed)
• State descriptor
State of each class handled by the Class Loader:
Error, Loaded, Verified, Prepared, Unloaded
Compiler Unit
• JVMTI Events intercepted
Compiled method load (method compiled and loaded), Compiled method unload
(method unloaded or moved), Method entry, Method exit
• State descriptor
Map of compiled/interpreted methods including: method name and, for each compiled
form of the method: compiled code size, code starting address, VM –specific
compiler information.
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
25
::. Distributed Monitoring System: collected data details (2/3)
Runtime core
• JVMTI Events intercepted
Thread start - Thread end - Monitor enter (attempt to acquire a lock owned by
another thread) - Monitor entered (lock acquired)- Monitor wait (waiting for an
object) - Monitor waited (finished waiting) – Exception (Exception thrown) –
Exception catch (Previously thrown exception has been catched) – Method Entry
(Java or native method entered) – Method Exit (generated upon exit from a Java or
native method)
• State descriptor
State of each Java thread including:
• its stack trace
• Its current lock status (locks acquired, locks which the thread is waiting for)
• Its possible pending exception
• Its method invocation graph (reports also the timestamp of each method entry
and method exit event)
• Its timeline (a timestamp is applied each time the thread changes state)
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
26
::. Distributed Monitoring System: collected data details (3/3)
Memory
Unit
MemoryManagement
Management
Unit
• JVMTI Events intercepted
Garbage collection start (a serial garbage collection starts) – Garbage collection
• State Descriptor
finish (a serial garbage collection finishes) – VM Object Allocation (new object
• Garbage
collector
timeline
allocation)
– Object
Free (the
Garbage collector frees an object)
• Object allocation and deallocation schedule
• Heap evolution
history
(for to
both
the
young
the tenured
 Huge-sized
heap – It
is possibile
take
a dump
ofand
the entire
shared generation)
heap, but it
will take too
long (especially
if graph
the dump has to be transferred over the network)
• Reachable
objects
 Data
about oops
in heap
area
are gathered
• location
and size
of each
reachable
object iterating over reachable
objects and sweeping the entire heap (like the GC does), obtaining in this
• Access/Modification flags for each reachable object
way syntethic information about objects (location, size and checksum)
• Hash value
describing
the content
of each
reachable
object and JNI are
 Stop-the-world
collector
– when
this collector
works
Java thread
blocked.
 Defer notification of events related to garbage collection until the
garbage collection finishes
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
27
::. Benchmark applications and Workload definition
 Web applications: the JVM is executed on the server
side to dispatch requests to a back-end architecture
• Jakarta Tomcat web server with J2SE 5.0 (Tiger)
• JSP+Servlets applications
• Web-stress tools to impose very high workloads to the VM
 Client and server applications, such control or scientific
ones. The JVM is deeply used for GUI design or mathematical
computations
• Javolution library benchmarks
• Other benchmarks…
 In order to make the field data measurement campaign more
effective, JVM is employed with two families of workloads:
• Workloads designed to stress VM components separately
• Workloads designed to stress the entire virtual machine as much
as possible following the behavior of the above mentioned
categories of applications
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
28
::. Data Analisys (1/2)
1. Detect failures at the application layer
2. Distinguish failures that are due to failures in the
underlying layers from failure that are due to the JVM
3. Given the instant tn where the failure has been detected,
analyze the evolution of the state of the JVM in the interval
[tn-D;tn], in order to identify the component where the
source of the failure was located. D is a thresold that we
assume to be the longest time that a fault in the JVM takes to
manifest as a failure at the application layer.
4. By analyzing data related to several failures identify
patterns of pathological behavior in the JVM.
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
29
::. Future Work: Assessing JVM dependability
We aim to give an answer to the following question:
«Is Java ready to support dependable applications?»
• The field-measurement campaign started aims to identify the
most critical components in the virtual machine and evaluate
their dependability attributes and bottlenecks
• Our monitoring system will be applied to Sun Hotspot® and
IBM JRockit® on different platforms (Solaris, Linux, win32)
At the present date only Sun and IBM implementations are
compliant to JSR-163 specification
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
30
::. Future Work: Augmenting the JVM with fault-tolerance
JRE provides no direct support for fault tolerance.
Hence Java applications:
Completely ignore failures
or…
Provide application-dependent fault detection and
recovery strategies or…
Delegate fault tolerance to external systems such as
transactional databases
In previous works (Alvisi,Friedman) fault tolerant JVMs were
developed using state replication
We aim at building a fault-tolerant JVM applying fault
tolerance techinques to internal JVM components
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
31
::. Future Work: Moving to the «mobile side»
•
Java is nowadays widely used in embedded devices such as
PDAs and cellular phones.
•
Such devices are often shipped with «embedded JVMs»
•
Such JVMs are a «compact» edition of the JVMs for desktop
and server systems, they include only a limited subset of the
features included in those JVMs.
•
At present time we don’t know any JVM for embedded devices
compliant to JSR-3 or JSR-163
•
Our goal is to perform a dependability analysis of these JVMs
To this aim we have to address the following issues:
• Investigate about the architeture of these JVMs
• Develop methodologies for their monitoring
www.mobilab.unina.it
[email protected]
JVM Dependability Analysis
32
And now it’s
time for
questions !
Contact info:
[email protected]
Mobilab Research Group
http://www.mobilab.unina.it/
www.mobilab.unina.it
[email protected]