Transcript Correlating Instrumentation data to system states: A building block

Correlating Instrumentation data
to system states: A building block
for automated diagnosis and
control
Ira Cohen, Jeffrey S. Chase et al.
Introduction

Networked systems continue to grow in scale

Complex behavior stemming from interaction of






Workload
Software structure
Hardware
Traffic conditions
System goals
Pervasive System needed to manage such a system

Examples?



HP’s Openview
IBM’s Tivoli
(Aggregates + displays graphically)
Introduction

Two approaches to build self managing
systems

A priori models



Event-condition-action rules
Not based on real systems
(Disadvantages?)


Difficult and costly
Unreliable, does not take account of all
Introduction

Statistical learning techniques

Assumes little to no domain knowledge


Hence “general”
Problem!

Still have to identify techniques that are
powerful enough to induce effective models
that are:



Efficient
Accurate
Robust
Goals

Automatically analyze instrumentation data from
network services in order to




Forecast
Diagnose
Repair failure conditions
We use the Tree-Augmented Naïve Bayesian
Networks (TANs) as the basis for


Diagnosis
Forcasting


System-level instrumentations in a 3-tier network service.
Widely used in various fields, but TANs are not used
in the context of computer systems.
Goals

Analyzed data from 124 metrics gathered
from

3 tiered e-commerce site under synthetic load




Httperf
Java PetStore as platform
TAN model select combination of metrics and
threshold values that complies with Service
Level Objectives for average response time.
Results later
What is a TAN?


Bayesian network is an annotated directed
acyclic graph encoding a joint probability
distribution
Naïve Bayesian Network



State var S is only parent of all other vertices
Assumes all metrics are fully independent given S
TANs consider relationships among metrics
themselves, with constraint that each metric has
only one other parent than S
Why Use a TAN?


Based on premise that a relatively small
subset of metrics and threshold values
is sufficient to approximate the
distribution accurately
Outperforms generalized Bayesian
networks and other alternatives in both


Cost
Accuracy
Why use a TAN?



Useful for forecasting failures and violations
Possible to induce models that predict SLO violations
in near future, even when system is stable
Automated controller can invoke directly


Identify impending violation
Respond





Loading
Adding resources
Cheap model to induce
Possible to maintain multiple models
Periodic refresh
Setup

System is 3-tier webservice






Apache
Middleware (BEA WebLogic)
Oracle db
3 Servers with HP Openview to collect
statistics
Load Generator is httperf
SLO indicator processes the logs to determine
compliance
Interpretability and
Modifiability

TANs offer other advantages



Interpretability
Modifiability
Influence of each metric can be quantified in a
probabilistic model



Analysis catalogs each type of violation according to the
metrics and values that correlate with observed instances
Strength is given from prob value occurring in different
states
Gives insight to causes of violations and how to repair
Workloads

Varies several characteristics




Aggregate req rate
Number of concurrent connections
Fraction of data-intensive vs app-intensive
requests
This is to exercise the model-induction
methodology by providing it with a wide
range of M,P pairs


Where M = sample of values for system metrics
P = vector of app-level performance
measurements
Workloads


RAMP: Increasing concurrency
STEP: Background + Step function



Background constant traffic
Bursty, hour long bursts
BUGGY: Increasing aggregate req. rate
Results




Varied SLO thresholds to explore effect
on induced models
To eval accuracy of models under
varying conditions
Trained and evaled TAN classifier for
each of 31 different SLO definitions
Baseline: accuracy of 60-pctile SLO
classifier (MOD) and CPU as metric.
Results






Overall BA of TAN is 87-94%
90+% for all experiments
6% False alarm for 2 experiments, 17% for BUGGY
Single metric is not sufficient to capture pattern of
SLO violations (CPU)
Small number of metrics is sufficient to capture
pattern (3-8)
Sensitive to workload and SLO definition (MOD
always has high detection rate, but generate false
alarms at increasing rate as SLO thresh increases)
Conclusion

TANs are attractive for self-managing
systems





Build system models automatically
No a priori knowledge required
Generalizes to wide range of conditions
Zeroes in on most relevant metrics
Practical
Conclusion



Possible work to adapt this to changing
conditions
Close the loop for automated diagnosis
and control
Ultimately most successful model is a
hybrid of


Automatically induced models
A priori models
Questions?