Transcript J2EE: Black Box in the Oracle World

J2EE: Black Box in the Oracle World
Don Bergal,
Confio Software
August 17, 2006
1
Objective: Answer a Question
 “Why does a DBA need visibility
upstream in the application in order
to manage the database?”
An alternate title for today:
 “How to be a better DBA by
watching something other than the
Oracle database”
Agenda
J2EE Black Box in an Oracle World
a) The Black Box Application Problem
b) The DBA Hot Seat
c) System Statistics versus Wait Time Diagnostics
d) A Better J2EE End-To-End Performance Solution
e) Best Practices Response Time Diagnostics Methodology
f) Q & A
The Black Box Application Problem
J2EE: Black Box in Oracle World
How can you
manage here?
If you cannot
see here!
Web store
Black Box:Billing
Applications and End Users
Financial
System
Web
customer
Internal
user
Batch job
See the applications that impact database
performance
Web store
Billing
Financial
System
Database
App Servers
Web
customer
Internal
user
Batch job
Web Users
Result: Disconnect Between IT Groups?
Java Development
DBA’s
J2EE Web & App Tiers
Databases
•
Client Web/Applet/Application
•
Database Management
•
JSP/Servlet Development
•
Database Development
•
EJB Container Development
•
SQL Development
•
JDBC Development
•
SQL Tuning & Optimization Tools
•
Uses Desktop J2EE Profilers
•
Ratio Based Monitoring & Tuning
No Visibility into the
Database Tier
No Visibility into the
J2EE Tier
Effect: Black Boxes leave DBA defenseless

‘Silo-based, Single System’ tools only shows database
system perspective (even though they are blamed for
problem in any system). Can’t tell when to hand-off
problem to another group (Dev/SysAdmin/etc…)
Black Box
Black Box
Blamed for all layers, but can only see the database
The DBA Hot Seat
Finger Pointing Happens When You Can’t See
the Problem
What’s wrong with
your *&!@$#
database?
Management
It’s a database issue!
DBA
Java/Web Developer
Guilty until proven innocent
Network Engineers
It’s a database issue!
DBA Hot Seat
Software Vendor
It’s a configuration
issue!
User Wait Time
Do Customers See the Problem Before You?
Warning!
SLA Violation
Not my code…
Java Developer
DBA
System Admin
What
performance
problems ?!?
Not my Network,
Servers, Load
Balancer,
Firewall…
Hot Seat comes from UNCLEAR Ownership
It’s an Application issue!
DBA
It’s a Database issue!
Java App Owner
Need evidence to show where the problem exists, avoid “Finger
Pointing” to the database
Performance Suspects Line Up…
Developer
D
B
A
System
Admin
Networking
14
Lack of Visibility is Source of Problem
Typical Multi-Tiered J2EE System Leaves the DBA Blind
Service Levels
Measured Here…
Web page request
App Layer
Apache
WebLogic
WebSphere
IIS
Oracle eBusiness
Oracle 9iAS
Peoplesoft
Siebel
JD Edwards
SAP
Custom apps
Web Server
J2EE layer
…Blame assigned
Here
Oracle database
Why Does the DBA Need J2EE Visibility?
You can’t control what you can’t see
You can’t show/demonstrate it’s not a DB Issue
You can’t manage your database if you are blind to
the source of the requests
Traditional Performance Tools versus Wait-Time Diagnostics
Current Tools – Never Cross the Silo Boundaries
Web Server
CPU
J2EE Server
CPU
Sessions
Memory
Database
CPU
Threads
Heap Size
Processes
Reads
 Tune inside each “Silo” to improve end-end performance
 “System Health” or ratio-based approach to


performance…
Statistics…% Utilization…Event Counts…
No direct relationship to End-User experience or SLA
Weak linkage between tiers limits collaboration
Standard J2EE App Performance Monitoring:
How Does This Help Identify User Response?
Idle threads
Number of
requests
Throughput
Waiting
Requests
Memory
(Heap) Usage
J2EE Profilers: High Load, Test Data Only
OS Level System Monitoring (rstatd/perfmon)
Network System Monitoring
Traditional Tools vs. Wait-Time Methodology
 Ratio-Base traditional tools
show “Oracle system health”
but no link to end user service
 Use Wait Time to correlate user
requests to SQLs and performance
impact
A Better J2EE End-To-End Performance Solution
Requirements for Oracle & J2EE Monitoring







Use Oracle Wait-Time Based Monitoring
Visibility across Java and Oracle layers
Lightweight JVM Method Tracker
Agnostic JVM server support
Fast installation and data collection techniques
Cost Effective
EASE OF USE – MAKE IT OBVIOUS!
Wait-Time Based Performance Analysis
 Tune by determining where client request time is spent
 Determine where Oracle resources are spent during the execution of
SQL execution
 Emerging best-practice for database tuning
• “You can’t tell how long something took by counting how many times it
happened. … If you’re only measuring event counts, then you’re not
measuring what the users care about.”
—
Optimizing Oracle Performance, O’Reilly Press
 Oracle is building up Wait-based tuning capabilities with each
database release…10g has over 800 wait-time based performance
counters
Oracle Wait Interface Book
Free book at www.confio.com
A Wait-Time J2EE Performance Methodology: 3
Key Requirements
1. Follow every transaction individually
2. Identify every “step” along the path
3. Measure the “wait-time” at each step
Result: Minimize time to resolution and division of work by
clearly identifying which step is performing poorly.
Alternative Monitoring Architectures
Best Practices Wait Time Diagnostics Methodology
(Day in the life on the performance team)
4 Steps for J2EE to Oracle Visibility
1.
2.
3.
4.
Capture Every User Request (end-to-end transaction)
Capture Every Step Inside Each Tier
Measure Response Time of Each Step
Identify Possible Causes Based on Wait-Times
Since every step is visible, you can
always see the problem
4 Steps for J2EE to Oracle Visibility
1.
2.
3.
4.
Capture Every User Request (end-to-end transaction)
Capture Every Step Inside Each Tier
Measure Response Time of Each Step
Identify Possible Causes Based on Wait-Times
Since every step is visible, you can
always see the problem
1. Capture Every User Request
Identify the slowest performing end-to-end Requests
Slowest performing can be:
• Longest average request-response time
• Largest accumulated Wait-Time in application (across
all sessions)
1. Capture Every User Request
PlaceOrder.jsp Request has
most serious delays
2. Capture Every Step Inside Each Layer
Identify the slowest steps within the slowest Request
Key Requirement: Correlate Application
Response to SQL Response Times
Question: Which SQL statements are causing Wait Time for
my application?
2. Capture Every Step, Identify Slowest
UpdateOrder Total (SQL) is
slowest in PlaceOrder.jsp
(Most Wait-Time
accumulated)
3. Measure Response Time of Each Step
Focus on steps that consume the most time
Detailed Response Time for the Critical
Step
Update Order Total
• ‘Undo Segment Extension’
Wait-Event is bottleneck
4. Possible Causes Based on Wait Times
Diagnose the step with the greatest Wait Time
4. Possible Causes Based on Wait Times
Execution path and time spent on each ‘Hot Spot”
Identify Chain of Calls
• URL Request -> Method Call –> SQL
• Allocate Wait Time to each Method or SQL Call
• Profiler data Without the System Impact
Recall 3 Key Requirements to Provide the
Analysis
1. Identify individual
Request
2. Identify Every
step – Java method
or Wait-Event
3. Measure Wait time
for Every Step
Customer Example




Retail transaction system
Online media fulfillment
Customer requests running slowly, unknown
reason
Ignite for Oracle and Ignite for Java used to
examine situation
Wait Time for “Authorize Transaction”
Identifies Problem Steps
Associates Request Wait-Time with Single
Procedure



Same query multiple times in single procedure
Same row retrieved 7 times
Change to select once, cut Wait-Time to less than half
Summary: Look For…




Continuous monitoring 24/7 – not
session/period specific
Production data – not synthetic “test”
transactions
Correlate Application & Oracle – where most
of the time is spent
Individual transactions, not system-wide
statistics
Visibility for the DBA




DBA has a need to see the J2EE Application
Solve database problems by understanding the other
systems
Look for Wait Time – that’s where the action is
Achieve collaboration between Developers, Architects
and DBA’s
 DBA can be the leader, not the one to
take the blame!
About Confio Software

Wait-Time based performance tools for continuous, low
impact monitoring

Ignite for Oracle – Deep, detailed visibility into SQLs and
Wait-Events

Ignite for Java – End-to-End Application through Oracle
visibility, correlates Java and SQL Wait-Time

Ignite for SQL Server – Coming soon, customer testing
Free Trial at www.confio.com
About the Author



Don Bergal, Confio Software
Responsible for product management, among
other things…
18 years software/telecom industry
experience [email protected]
Q & A