Transcript DBMS Benchmarks

An Intro. to Database Benchmarks
June 2001
Prof. Sang Ho Lee
Soongsil University
[email protected]
1
Benchmarks: What and Why
 To measure the performance of a hardware and software
system in a controlled environment using a standard
methodology
 Performance tests by which different database
hardware/software platforms are compared in terms of
price and performance
 Motivated and driven by database customer demand
(usually, to make acquisition decisions easy)
2
Performance Assurance
 Once you build the system you must do
 Quality assurance: Functionality validation
 Performance assurance: benchmark
• find the performance bugs
• looking for bottlenecks (queues)
 When performance assurance is complete, model can be
used to predict performance on higher loads, new hardware.
This is called “capacity planning”
3
Custom Benchmark vs. Generic Benchmark
 Custom benchmark
 Created by a particular customer based on a specific application
 Can measure precisely capabilities that the customer wants
 Very cost and hard to maintain
 Generic benchmark
 Created to be generally useful for a large group of potential
customers with relatively standard applications across an
application domain
 Also known as domain-specific benchmark
 Hybrid benchmark
 generic benchmark + custom benchmark
4
Popular Benchmarks














Wisconsin benchmark (1983)
TP1/DebitCredit benchmark (1985)
Transaction Processing Council formed (Aug. 1988)
TPC-A (Nov. 1989)
TPC-B (Aug. 1990)
AS3AP (1991)
SUN benchmark (001, object operations version benchmark) (1992)
TPC-C (Jul. 1992)
007 Benchmark (1993)
Sequoia 2000 Storage Benchmark (1993)
TPC-D (Apr. 1995)
BUCKY (1997)
TPC-W (1998)
BORD (2000)
5
How to Normalize Results
 Difficult to compare the performance of the wide variety of
database software/hardware platforms (from IBM
mainframe to PC)
 Current practice
 Requires domain-specific qualifications that must be met
• Example: ACID properties of transaction for OLTP
benchmarks
 Reports two specific measures
• a measure of peak performance, say transactions per second
(tps)
• a measure of price/performance, say dollars per tps
6
Scalar measures vs. Vector measures
 Vector measures allow users to distinguish different
performance effects in terms of functional origins, namely
separability
 Think about pros and cons !
7
Key criteria for domain-specific benchmarks
 Relevance: performs typical operations within the problem
domain
 Portability: easy to implement on many different systems
and architectures
 Scalability: applicable small and large computer systems
 Simplicity: easily understandable otherwise it would lack
credibility
 Vendor neutrality: must not be biased in favor of a
particular implementation
8
Brief history on OLTP benchmarks
 DebitCredit benchmark (1985)
 Anon. et al., “A Measure of Transaction Processing Power”, Datamation
 TP1 benchmark
 an implementation by vendors that typically departs from the DebitCredit
specifications one way or another
 Aug. 1988: Transaction Processing Council formed
 Originally eight vendors, currently 35 vendors joined





Nov. 1989: TPC-A, OLTP with LAN or WAN
Aug. 1990: TPC-B, OLTP with no network
Jul. 1992: TPC-C, On-Line Business Transaction Processing
Jun. 1995: TPC-A/B are frozen
Feb. 2001: TPC-C version 5.0 release
9
Decision support system (DSS) benchmarks
 Decision support system
 relatively small number of users
 mostly read-only queries, seldom update operation
 Queries involve a great many data and indexes, thus use
significantly more resources than OLTP transactions
 data warehouse application
 Wisconsin
 AS3AP
 Set Query
 TPC-D benchmark – obsolete as of June. 1999.
 Separated into TPC-R benchmark and TPC-H benchmark
10
Engineering database benchmarks
 Sun benchmark (001 benchmark)
 W. Rubenstein, M. Kubicar and R. Cattle (ACM SIGMOD 1987)
 R. Cattle, J. Skeen (ACM TODS 1992)
 HyperModel Benchmark
 T. Anderson, et al. (EDBT 1990)
 007 Benchmark
 M. Carley, D. DeWitt, J. Naughton (ACM SIGMOD 1993)
 A Complex Object Benchmark (ACOB)
 D. DeWitt, P. Futtersack, D. Maier and F. Velez
 VLDB conf. 1990
11
How benchmarks are used: Good stories
 Benchmark defines the fields
 Accelerates progress
• Designer can compare approaches
• Users can compare approaches
 Give performance agenda
• Measures release-to-release progress
• Set goals
• Something managers can understand
 Benchmark definition forms a community/vocabulary
 Cross fertilization of people and ideas
 Sharing of knowledge
12
Research issues
 What are good criteria for domain-specific benchmarks
 Currently, relevance, portability, scalability, simplicity
 What about scalar versus vector measures
 How to incorporate actual costs of system ownership
 Multi-user benchmark
 How to achieve saparability of performance effects
 How to develop an analytic framework
 New applications or models
 BLOBs, GIS, IR, Object-relational data model, etc.
13
Comments on benchmarks (1)
 Benchmarking is hard
 Tools are a big help, but each experiment takes a day
 Always do an atomic model first, check reality against the
model
 Keep careful notes
 Change only one thing at a time (controlled experiments)
 Use all the tools you can have: Good luck!
14
Comments on benchmarks (2)
 Watch out (unauthorized) vendor’s claims
 Watch out benchmarketing
 Benchmark result is certainly one thing, but not everything
 Understanding what results say EXACTLY is very important
 Must be aware of underlying platforms, networks and hardware, even
though we are talking about DBMS (or OLTP) performance
 Remember, vendor’s technologies and benchmark specifications
continue to evolve over time
15
References (1)
 T. L. Anderson, A. J. Berre, M. Mallision, H. Poster, and B. Schneider,
The HyperModel Benchmark, Proc. of the Int. Conference on
Extending Database Technology, 1990.
 Anon, et al., A Measure of Transaction Processing Power, Datamation,
1985.
 D. Bitton, D. J. DeWitt, and C. Turbyfill, Benchmarking Database
Systems: A Systematic Approach, Proc. of the Ninth Int. Conference
on Very Large Data Bases, 1983.
 D. Bitton and C. Turbyfill, A Retrospective on The Wisconsin
Benchmark, In: Readings in Database Systems, M. Stonebraker ed.,
Morgan Kaufman, 1988.
 M. J. Carey, D. J. DeWitt, and J. F. Naughton, The OO7 Benchmark,
Proc. of the ACM SIGMOD Conference, 1993.
16
References (2)
 M. J. Carey, D. J. DeWitt, J. F. Naughton, M. Asgarian, P. Brown, J. E.
gehrke, and D. N. Shah, The BUCKY Object-Relational Benchmark,
Proc. of the ACM SIGMOD Conference, 1997.
 M. J. Carey, D. J. DeWitt, C. Kant, and J. F. Naughton, A Status Report
on the OO7 OODBMS Benchmarking Effort, Proc. of the ACM
OOPSLA Conference, 1994.
 R. G. G. Cattell, An Engineering Database Benchmark, In: The
Benchmark Handbook for Database and Transaction Processing
Systems 2nd ed., J. Gray ed., Morgan Kaufmann, 1993.
 R. G. G. Cattell and J. Skeen, Object Operations benchmark, ACM
Transactions on Database Systems, 1992.
 D. J. DeWitt, The Wisconsin Benchmark: Past, Present, and Future, In:
The Benchmark Handbook for Database and Transaction Processing
Systems 2nd ed., J. Gray ed., Morgan Kaufmann, 1993.
17
References (3)
 J. M. Hellerstein and M. Stonebraker, Predicate Migration: Optimizing
Queries with Expensive Predicates, Proc. of the ACM SIGMOD
Conference, 1993.
 W. Kim, Completeness Criteria for Object-Relational Database
Systems, UniSQL Inc, 1996.
 S. H. Lee, S. J. Kim, and W. Kim, The BORD Benchmark for ObjectRelational Databases, Springer-Verlag Lecture Notes in Computer
Science, 2000.
 P. O’Neil, The Set Query Benchmark, In: The Benchmark Handbook
for Database Transaction Processing Systems 2nd ed., J. Gray ed.,
Morgan Kaufmann, 1993.
 P. O’Neil, Database Performance Measurement, In: The Computer
Science and Engineering Handbook, A Tucker ed., CRC Press, 1997.
18
References (4)
 W. B. Rubenstein, M. S. Kubicar, and R. G. G. Cattell, Benchmarking
Simple Database Operations, Proc. of the ACM SIGMOD Conference,
1987.
 H.Schreiber, Justitia – A Generic Benchmark for the OODBMS
Selection, Proc. of the Fourth International Conference of Data and
Knowledge Systems for Manufacturing and Engineering, 1994.
 K. Shanley, History and Overview of the TPC, Tranasaction
Processing Performance Council, www.tpc.org, 1998.
 M. Stonebraker, J. Frew, K. Gardels, and J. Meredith, The SEQUOIA
2000 Storage Benchmark, Proc. of the ACM SIGMOD Conference,
1993.
 M. Stonebraker, Object-Relational DBMSs, Morgan Kaufmann, 1996.
19
References (5)
 Transaction Processing Performance Council, TPC Benchmark C
standard specification version 5.0, www.tpc.org, 2001.
 Transaction Processing Performance Council, TPC Benchmark H
standard specification version 1.3.0, www.tpc.org, 1999.
 Transaction Processing Performance Council, TPC Benchmark R
standard specification version 1.2.0, www.tpc.org, 1999.
 Transaction Processing Performance Council, TPC Benchmark W
version 1.4, www.tpc.org, 2001.
 C. Turbyfill, C. Orji, and D. Bitton, AS3AP: An ANSI SQL Standard
Scaleable and Portable Benchmark for Relational Database Systems,
In: The Benchmark Handbook second edition, J. Gray ed., Morgan
Kaufmann, 1993.
20