Transcript physical measurement and IT project

Projects Overview
CS533 Performance Evaluation
Projects Overview
1
Your Projects This Semester
Performance Measurement Warm-ups
• A series of small tasks to get you in the mode for measuring and
reporting
• Follow closely the material in the text – “Network Performance”
The Main Performance
• A large project following your own interests.
• Contains a series of steps to ensure success.
• These steps include:
Project Pre-Proposal
Project Proposal
Design Proposal
Project Report
Projects Overview
2
Warm-Ups
Students taking this course have usually had a strong interest in network
performance. As such, these warm-up exercises give you a chance to
practice measurement on networks.
There are three warm-ups to be accomplished in the next few weeks:
1.
2.
3.
Using a network snoop device; Wireshark or tcpdump
Warning – these
Measuring network performance
warm-ups are
Measuring application performance
Ground rules for all warm-up reports.
undefined and
open ended.
Note – the measurement and the report both matter.
1. The report should state clearly what’s been done.
2. It should state the environment used. “Wireshark was used to measure the
round-trip ping time as a function of physical distance between sites.”
3. The report should include measurements where one thing depends on
another – there MUST BE a graph in your report.
4. There should be an indication of the accuracy of your measurements.
What’s the uncertain in your values? Are they repeatable? If someone else
were to measure the same thing, would they get the same result?
5. Plan on delivering your report to the class; should be no more than 5
minutes in length. I will critique your results on the fly. Everyone will learn.
Overview or it
3
6. Plan also on handing in your report – it can be a few slidesProjects
for instance
can be on paper.
Warm-Ups
Does Correlation
Imply Causation?
Examples:
There's a strong correlation between foot size and ability to do arithmetic.
(Of course, it's not causation; both measured variables are strongly correlated with
the age of the children. Older ones have bigger feet and better math ability.)
People who drink diet soft drinks don't lose weight. In fact, they gain weight, a
new study shows. (Best Theory: When we offer our bodies the sweet taste of
diet drinks, but give them no calories, it makes us crave more calories.)
The more firemen fighting a fire, the bigger the fire is observed to be.
(Therefore firemen cause an increase in the size of a fire? )
As ice cream sales increase, the rate of drowning deaths increases sharply.
(Therefore, ice cream consumption causes drowning.)
Projects Overview
4
Warm-Ups
Use of Performance Nomenclature:
It is expected that by the time you finish each of these warm-ups, you will have
made use of Performance concepts.
It is SOOO easy to talk only about throughput, for instance, and ignore all the
other nice concepts.
But I’m expecting that you will talk about:
Response Time
Utilization
Throughput
Capacity/Bandwidth
Projects Overview
5
Warm-Ups
Ground rules for all warm-up reports.
The WPI Acceptable Use Policy http://www.wpi.edu/+AUP clearly prohibits the
use of sniffers on the network.
This rule is in the Acceptable Use Policy to prevent denial of service conditions
resulting from 'active' sniffing, which uses gratuitous ARP traffic to redirect
packet flow, and to preserve privacy and uphold Federal law by not allowing
students to accidentally wiretap a communications channel.
There is a way to perform the measurement tests as described in these projects
that does not violate the AUP. Students need to run their sniffers in nonpromiscuous mode. This allows for collection of data in the sniffer that they
would be processing anyways, without seeing any other traffic on the
network.
This solution avoids both potential problems outlined above. Here is how to use
non-promiscuous mode for your tests. In Wireshark, that setting is a
checkbox on the capture configuration screen. In tcpdump, that setting is the
'-p' flag.
Projects Overview
6
Environments and Tools
In doing these warmups, we’re exploring three different “levels” of environments.
Warmup #1 focusses on very LOW level environments – where we look at individual actions
– for instance, individual network packets, individual disk accesses, individual process
scheduling. We can study only ONE action – and get as much possible information as
we can. Or we can look at multiple instances of this simple action, with a goal of getting
more information about its performance.
Warmup #2 looks at the next “higher” level of environment. An aggregate action – a stream
of data made up of many packets, disk accesses over a span of time, and so on.
Warmup #3 in intended to look at a complex environment. For instance, the disk activities
that occur to a “user” environment, the network traffic coming from multiple sources, the
car and truck traffic associated with an intersection, etc. These do not lend themselves
to simple analysis, and we get only aggregate results.
Once we’ve identified the environment, then we can select the tool.
Low level environments require tools that can measure individual actions – Wireshark to
watch an individual network packet, timing of individual left turns at an intersection.
Medium level experiments probably won’t measure individual actions, but require tools that
can measure throughput, response time, queueing time for a series of actions.
High level, complex environments will perhaps need tools that understand the environment.
For instance, a database in the cloud will support transactions made up of a number of
smaller actions – it may involve network, disk, and other resources. Projects Overview
7
Warm-Up #1
Use a software snoop device to measure network traffic on a machine.
What you need to know:
1. Chapters 1 – 2 in “Network Performance”
2. Class notes from the first lecture on measurement.
In this project, you will install a sniffer on a machine of your choice and use it to measure low
level network performance.
A sniffer is a program running on a machine that is able to capture packets and packet
information on that machine. In considering what machine to use, be aware that you
will need generally administrative access to that hardware. Chapter 2 of “Network
Performance” talks about tcpdump, windump, and Wireshark (formerly ethereal). I’m a
Wireshark fan myself, but it doesn’t really matter which you use.
Here’s your assignment:
Measure some aspect of network traffic where you measure the same thing but change
some one or two variables. Organize your results so they are easily understandable
(presentation is everything!) including a plot or graph.
Here are several possible examples – but I encourage you to think up your own:
1.
2.
While Wireshark is running, connect to a website with your browser. How long does it take to
complete the handshaking required to establish the connection. Now vary the distance to the site
you’re connecting to. Does this change the connection time? Does the connection time depend on
other things – like time of day, number of hops, etc. ( Remember that correlation does not imply
correlation! What is the real cause of a dependency you find?)
While Wireshark is running, ping a number of sites and determine the exact time it takes for that ping
to complete (rather than the very general number reported by ping.) The ping time depends on what
Projects Overview
8
– distance, hops, etc???
Warm-Up #1
Name______________________
Use a software snoop device to measure network traffic as seen by a node.
1.
Does the measurement accomplish the assignment?
A.
B.
C.
2.
Measurement Criteria:
A.
B.
C.
D.
E.
3.
Uses a sniffer of some kind.
Measures network traffic at a low level (and not via some mega-very-complicated-application
high level interface).
Uses a simple tool (browser, ping, or traceroute, for instance) to drive the traffic.
Is the experiment conducted such that there’s only one independent variable at a time? And
that one independent variable leads to a change in the dependent variable – i.e., the
dependent variable is really dependent?
Are there several measurements under the same conditions – do we have the ability to assess
the uncertainty in the results obtained? Is there discussion of significant figures, standard
deviation and variation so we know how much we trust the numbers?
Is there an analysis of why the variations seen in the numbers might occur? Is there a sanity
check that the information presented makes sense?
Is there an analysis that the dependence is due to causation and not just correlation?
Is there a physical model presented that explains why the observed behavior occurs?
The presentation:
A. Is well organized.
B.
C.
D.
There’s a plot or graph presented. The visual information accurately represented what was
measured.
There was an in-class delivery and material given to me for further evaluation.
Completed within the time limit.
9
Mid-Level Tools
In Warm-Up #1, there was a low-level tool you used; Wireshark is an excellent example – it
gives lots of very detailed information.
Warm-Up #2 will use tools at the next higher level. On Windows these might be the Task
Manager (fairly simple statistics) or the Performance Monitor that gives you detailed
information. Neither of these give packet level details, but they provide information
about aggregate network behavior.
On LINUX, there are not such nice tools that come with the OS (at least, not that I’ve found.)
There are numerous tools out there that you can download. Netstat and traceroute
come with LINUX, but the command line interface isn’t much – you can get free GUI
interfaces for them. Also take a look at: http://www.uperf.org/
http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html and many others.
Note that there are two flavors of measurement here – and either is appropriate for your
experiment:
1) The measuring device is independent of the generator of traffic; it’s essentially standing
“outside” of the generator and gathers what’s happening.
2) The tool that generates the traffic also incorporates a means of measuring that traffic
The network generation tool is usually designed to max out the connection between A and B.
It’s a measurement of capacity, and will find a bottleneck somewhere along the path. This is
more of a network measuring tool (fine for this Warmup) than a system or application
tool.
Projects Overview 10
Warm-Up #2
Measure Network Performance - throughput
What you need to know:
1. Chapters 4 – 12 in “Network Performance”
2. Class notes from the first lecture on measurement.
In this project, you will measure the performance of a network. This network might be a local
LAN consisting of machines on the same switch, it could be a metropolitan network of
machines near us, or it might span the globe – your choice.
Here’s your assignment:
Use several of the tools described on the previous slide to measure network traffic. As in
Warm-Up #1, vary the properties of what you are measuring in order to get a variety of
measurements, and again display and explain your results.
Here is one possible example – but I encourage you to think up your own:
1. Determine the throughput between your machine and an off-site location as a function
of the time of day. When is the best and worst time to use your machine as a source
across the WPI network? Are your numbers consistent from day to day? What
matters in getting good throughput?
Your report should include the data analysis as you did in Warm-Up #1, (see Ground Rules)
but in addition you should report on the relative advantages and disadvantages of the
several tools you used..
Projects Overview
11
Warm-Up #2
Name______________________
Measuring a Mid-Level Environment
1.
Does the measurement accomplish the assignment?
A.
B.
C.
2.
Measurement Criteria:
A.
B.
C.
D.
E.
3.
Uses a tool to measure traffic.
Measures network traffic at a medium level (and not via some mega-very-complicatedapplication high level interface).
Uses a simple tool to generate the traffic – perhaps same as measuring tool.
Is the experiment conducted such that there’s only one independent variable at a time? And
that one independent variable leads to a change in the dependent variable – i.e., the
dependent variable is really dependent?
Are there several measurements under the same conditions – do we have the ability to assess
the uncertainty in the results obtained? Is there discussion of significant figures, standard
deviation and variation so we know how much we trust the numbers?
Is there an analysis of why the variations seen in the numbers might occur? Is there a sanity
check that the information presented makes sense?
Is there an analysis that the dependence is due to causation and not just correlation?
Is there a physical model presented that explains why the observed behavior occurs?
The presentation:
A. Is at the right level; just the right amount of detail – enough to explain what you did, but not so
B.
C.
D.
much you are giving TOO much data.
There’s a plot or graph presented. The visual information accurately represented what was
measured.
There was an in-class delivery and material given to me for further evaluation.
Completed within the time limit.
12
Warm-Up #3
Measure Application Performance
What you need to know:
1. Chapters 14 – 18 in “Network Performance”
2. Class notes on simulation.
In this project, you will determine the performance of an application. You can do that in one
of two ways – with a traffic generator/benchmark/workload (Chpt. 16) or with a simulator
(the other chapters)
Here’s your assignment:
Use a tool of your choosing to emulate the behavior a system would have. This “system” is
more complex than the simple traffic patterns that you would get from the network
measurements. This tool can be one you build, it can be one you’ve previously used
(but “enhanced” to emulate a whole system) or could be a new tool.
You are free to come up with an environment of interest to you.
measurement and reporting are essential.
But as before,
Please try to DESIGN YOUR EXPERIMENT: There are a series of steps in Section 12 of the
“Measurement and Statistics” Slides. Use the steps from there that are applicable. That
section asks questions that will help you break down your project into do-able steps. Too
many of us do an experiment and then say “that’s what I was planning on measuring all
along.” This design process is extremely important.
Your report should include the data analysis as you did in Warm-Up #2, (see Ground Rules)
but in addition you should report on the relative advantages and disadvantages
of the 13
Projects Overview
tool you used.
Warm-Up #3
Name______________________
Measuring an Application Environment
1.
Does the Design and Measurement accomplish the assignment?
A.
B.
C.
2.
Measurement Criteria:
A.
B.
C.
D.
E.
3.
Shows a use of the 12-step design process.
Uses a tool to measure traffic.
Measures at an application level – a fairly complex environment. Show that the benchmark
you are using is representative of a real environment.
Is the experiment conducted such that there’s only one independent variable at a time? And
that one independent variable leads to a change in the dependent variable – i.e., the
dependent variable is really dependent?
Are there several measurements under the same conditions – do we have the ability to assess
the uncertainty in the results obtained? Is there discussion of significant figures, standard
deviation and variation so we know how much we trust the numbers?
Is there an analysis of why the variations seen in the numbers might occur? Is there a sanity
check that the information presented makes sense?
Is there an analysis that the dependence is due to causation and not just correlation?
Is there a physical model presented that explains why the observed behavior occurs?
The presentation:
A.
B.
C.
D.
E.
F.
The presentation is mailed to me by noon on the date it is due.
Includes a discussion of the environment being studied.
Is at the right level; Not too much or too little detail.
There’s a plot or graph presented. The visual information represents what was measured.
There was an in-class delivery and material given to me for further evaluation.
Presentation Completed within the time limit.
14
The Main Performance
Choosing a Project.
There are of course many great ideas for projects. Here’s my overview of
these ideas.
Simulations:
I am delighted if you want to do a simulation. This can be a simulation of computer behavior (of
course), or it can be of any other phenomenon. There are two conditions for the simulations I will
accept:
a) A simulation based on some REAL data of a physical phenomenon. You can use some alreadyavailable simulation tool for this. BUT the tool must be able to predict a result that you didn't measure
and you need to be able to verify this prediction. It is not acceptable to use an existing model with data
you get from the web or anywhere else; you must measure it.
EXAMPLE: You measure a traffic intersection. You know the timing of all the lights, you know
traffic patterns; you know for each minute of a two-hour period how many cars are queued waiting for
a left turn, etc. You know EVERYTHING about that intersection. You put that data into an existing
simulation package and tune it to macth your intersection. THEN you give it the light patterns and
basic properties for a second intersection - now the model is able to successfully predict the traffic
patterns of this second intersection. That's how you know the model is correct.
Projects Overview
15
The Main Performance
Choosing a Project.
Simulations (Continued):
a) You build the simulation model. You need to collect some data for this, but I’m much less
concerned about the extend of that data – you’ve doing a lot of hard work building the model, so that
makes up for the lesser time spent measuring.
EXAMPLE: On the following pages, I give two examples of models built by former students; one
build a monopoly game – it had a mechanism for moving the pieces around the board, included buying
and selling of properties, for Chance and Community Chest, etc.
Another student built a network model including several routers and nodes and determined what
happens as traffic increases in this environment.
Projects Overview
16
The Main Performance
Choosing a Project.
Measurements:
People almost always measure something that’s computer or network related; it’s so much easier to
determine the performance characteristics of such things. You get to use all the things you’ve learned
this semester. Talking about the “performance of weather” or the “performanceof responses to a
questionnaire” while theoretically possible, I suppose, are pretty hard to do in practice.
Usually in this case your project would be a bigger example of the kind of thing you did for your
warm-ups. It would follow the twelve-step process, and would be of a bigger scope and in more detail
than you did for those warm-ups.
Here’s a list of measurement tools that people have used in the past:
Tracing and Tools Profiling Project in Eclipse, Iometer, Perfmon, Tcpdump, Wireshark, Iperf
Simgrid  simgrid.gforge.inria.fr, NetworkTrafficGenerator (ntg), Network Simulator (ns)
Projects Overview
17
The Main Performance
Choosing a Project.
Projects From Last Year
Ray Tracing Performance: A study of the factors affecting the performance of a program rendering a
three dimensional object.
Evaluating Virtual Machine Software Performance: Compares the performance of two virtual
machines as well as native OS running a number of benchmarks on Linux and Windows.
Performance Analysis of Common Security Tools: Looks at the hardware and network performance
of two products.
Slingbox: Analyzing the performance of the video streaming of SlingBox under different network
settings.
Traffic Simulation: Wrote a simulation model (from scratch) and used it to understand automobile
traffic patterns.
Monopoly Simulation: Wrote a simulation model (from scratch) for the game of monopoly. This did
not use much real data, but presented a major challenge in modeling the various aspects of the game.
Projects Overview
18
The Main Performance
Choosing a Project.
Projects From Last Year
Web Server Performance: What factors affect the performance of Apache Tomcat.
Home WLAN Performance: Studies factors affecting the performance of home networks.
Mesh Networks: Measured and analyzed the performance of networks of computers.
Hadoop: A study of the measured performance of a “big data” database.
Data Base Behavior: Performance factors affecting the measured performance of data base updates.
Mobile Hard Drive: Measured the performance of a hard drive under a large number of conditions.
Projects Overview
19
The Main Performance
It is assumed that you will choose a topic an aspect of performance that is of
interest to you or that you can use in your job.
The project contains several components as shown below. The due dates for
these components are given on the main syllabus page.
Project Pre-Proposal
Goal:
Deliverable:
I Owe You:
What To Look For:
To get agreement on a topic in an informal way.
Here you and I have a conversation/e-mail/brief write-up that contains a
very short statement of what you want to do for a project.
Very rapid feedback that the direction you have chosen is OK.
Major issues I will check for include:
• Scope/size - does the project you are proposing contain the
appropriate amount of work?
• Diversity – does the project contain several components such as
measurement, simulation, etc. from the topics we are discussing in
class.
Projects Overview
20
The Main Performance
Project Proposal
Goal:
Deliverable:
What’s Included:
•
•
•
•
To allow you to write up your project and present it to the class.
You will prepare and deliver a 10 minute class presentation. You will also
have a written proposal. It’s the written proposal that will be evaluated
for a grade.
There are a number of items that should be here. They include:
A description of the performance project you will do.
What performance components are included in this project.
An explanation of how this fits into some larger context.
A justification of why this is important and/or useful.
Design Proposal
Goal:
To give a formal progress report on your project. By this time, you have
gone down a number of dead-ends, figured out a number of details, and
now have a project that’s quite a bit different from when you made your
proposal. We want to hear where you’re at.
Deliverable:
You will prepare and deliver a 10 minute class presentation. You will
also have a written proposal. It’s the written proposal that will be
evaluated for a grade.
What’s Included: There are a number of items that should be here. They include:
•
A brief reminder of the performance project you are doing.
•
What performance components you have used thus far.
•
How your project has changed, and why (ways we have gone wrong can
be very educational.)
Projects Overview 21
•
Difficulties ahead.
Design Proposal
What you should report during this presentation:
Goal: To give a status report on your project. We want to hear what you’ve done so far.
.
Deliverable: You will prepare and deliver a 15 (FIFTEEN) minute class presentation.
What’s Included:
There are a number of items that should be here. They include:
1.
A description of the performance project you are doing. Focus on the PERFORMANCE aspects
and less on the network, etc. aspects. Please avoid “ratholes.”
2.
A justification that this is a PERFORMANCE project and not some warmed over project from
another course.
3.
4.
The Performance Engineering techniques/practices you are using in this project.
Current progress:
a) Portion of the goal achieved so far.
b) Portion still to be done.
5. A description of:
a) The performance environment (what was happening on the machine in
terms of disk accesses, CPU usage, etc.)
b) Performance tools used.
c) Performance models employed.
Projects Overview
22
Project Report
Goal:
To give a formal report on your project. We want to hear what you’ve done
and how you’ve done it.
Deliverable:
What’s Included:
You will prepare and deliver a 30 (thirty) minute class presentation. You
will also have a written report. It’s the written report that will
be evaluated for a grade.
There are a number of items that should be here:
1.
A brief reminder of the performance project you are doing. Functionality (how your product
works) ONLY to the extent absolutely necessary to explain the performance work you've done.
Try to avoid rat-holes please.
2.
Items from previous reports you’ve done that help us understand the context of your project.
3.
The Performance Engineering techniques/practices you used in completing your project. The
best way to do this is to fit your methodology into the 12-step process we’ve discussed.
4.
Show the performance measurements or performance practices that you used. This includes
models, tools and techniques.
5.
A description of how much you trust your results. Show IN DETAIL the statistical analysis you
perform to show your trust.
6.
What did you learn from this project?
7.
Your report may well have additional topics.
Projects Overview
23