Transcript Automatic Space Reclamation

#IDUG
Monitoring the Essentials in DB2 10.5
Philip K. Gunning
Gunning Technology Solutions, LLC
Session Code: C03
May 13, 2014 3:15 PM – 4:15 PM| Platform: DB2 LUW
#IDUG
2
DISCLAIMER
• DB2™ is a registered trademark of IBM Corporation.
• DB2 BLU, BLU Acceleration are registered trademarks or
copyrights of IBM Corporation
• Other logos and product/trade names are registered
trademarks or copyrights of their respective companies
#IDUG
3
The Warp Speed Achieved with BLU Acceleration
“For data analytics the volumes involved are usually
dramatically larger than cache sizes, which highly limits
exploitation of CPU caches. The secret to the warp speed-up
achieved in BLU Acceleration is that IBM has designed
every algorithm to minimize access to RAM, and maximize
processing time in L3 and L2 caches. For the
processor shown that provides
a whopping 15x-41x speed-up for memory
access” (Sam Lightstone, IBM,
www.Softwaretradecraft.com)
#IDUG
4
Overview
• Overview of BLU Acceleration
• Concepts
• Terminology
• DB2 in the New Galaxy
• When RAM is too slow
• Learning to speak and think “Column organized” in BLU
Acceleration
• Rules of Thumb for column organized data in terms workload,
concurrency, bufferpools, sort heap threshold, sortheap
#IDUG
5
Overview
• Important role of UTIL_HEAP_SZ
• New “Column Organized” monitoring metrics
• How efficient are my column-organized tables
• Are you getting what you think?
• Using MONREPORT reporting module
• Tuning Recommendations
• Best Practices
#IDUG
6
Quick Look at Prerequisites
• DB2 Workgroup Advanced Enterprise Server Edition
• DB2 Advanced Enterprise Server Edtion
• OS and Hardware prerequisites
#IDUG
7
New Registry Variable Setting for Analytical Workload
• Before creating a new database, set the DB2_WORKLOAD
registry variable to ANAYLTICS
• This will result in the following:
• Makes column organized tables the default table type
• DFT_TABLE_ORG=COLUMN
• Page and extent size configured for analytical processing
• PAGESIZE=32
• DFT_EXTENT_SZ=4
• DFT_DEGREE=ANY
• Memory for sorting, caching and hashing will be set based on the
amount of RAM and server size
• Page and extent size configured for analytical processing
• Enables automatic space reclamation
• Compression is always on
#IDUG
8
Setting the DB2_WORKLOAD Registry Variable
Continued
• Memory for sorting, caching and hashing will be set based on
the amount of RAM and server size
• CATALOGCACHE_SZ, SHEAPTHRES_SHR and SORTHEAP set to a value
that is optimized for the hardware you are using
• Intraquery Parallelism enabled for any workload even if DBM
CFG parameter INTRA_PARALLEL=OFF
• The default concurrency threshold on the
SYSDEFAULTMANAGEDSUBCLASS service subclass is enabled to
ensure maximum efficiency and utilization of the server
• Automatic Space Reclamation is enabled
• AUTO_MAINT=ON
• AUTON_REORG=ON
#IDUG
9
Setting the DB2_WORKLOAD Registry Variable
Continued
• Automatic Space Reclamation is enabled
• AUTO_MAINT=ON
• AUTO_REORG=ON
• Space reclamation performed for column-organized tables by
default
#IDUG
10
Use of AUTOCONFIGURE
• For an existing database that you want to configure for BLU
Acceleration
• Set the DB2_WORKLOAD=ANALYTICS registry variable and restart the
instance
• Connect to the database you want to run autoconfigure on
• Run “db2 autoconfigure apply dbname”
• After running autoconfigure, review the bufferpool.
SHEAPTHRES_SHR, SORTHEAP , UTIL_HEAP_SZ and adjust as
necessary
#IDUG
11
Use of AUTOCONFIGURE
• In an instance with multiple databases some of which are not
using BLU, set the registry variable before you create the
databases that will be using BLU, then unset the registry
variable after creating the databases that will use BLU
#IDUG
12
Setting SHEAPTHRES_SHR and SORTHEAP Parameters
• When BLU Acceleration is enabled, the self-tuning memory
manager (STMM) does not adjust the values of the
SHEAPTHRES_SHR and SORTHEAP database configuration
parameters automatically.
• Set with static values by the AUTOCONFIGURE command when
DB2_WORKLOAD=ANALYTICS registry setting is enabled
#IDUG
13
Setting SHEAPTHRES_SHR and SORTHEAP
• Analytical workloads require larger sort memory areas due to
the nature of the workload
• Optimal setting provides degree of concurrency required and
associated SORTHEAP sizing
• Setting SHEAPTHRES_SHR to AUTOMATIC not supported for
databases using column organized tables
• Tuning SHEAPTHRES_SHR and SORTHEAP as important as
tuning BUFFERPOOLS
• Aim to prevent sort overflows to disk which can significantly slow down
queries
• SET SEHAPTHRES_SHR to 40-50% of the memory available to
the database
• DATABASE_MEMORY DB CFG Parameter
#IDUG
14
Setting SHEAPTHRES_SHR and SORTHEAP
• The SORTHEAP DB CFG parameter controls the amount of
memory an application can use for SORTS
• SORTHEAP memory comes out of total SHEAPTHRES_SHR
memory
• Size the SHEAPTHRES_SHR DB CFG parameter and SORTHEAP
DB CFG parameter to provide the amount of concurrency
desired and to prevent sort overflows
• Rule of thumb for sizing SORTHEAP is set it to 5-20% of
SHEAPTHRES_SHR
• Monitor for sort overflows and post threshold sorts via
MON_GET_DATABASE and MON_GET_UNIT_OF_WORK table
functions
#IDUG
15
Monitoring SHEAPTHRES_SHR and SORTHEAP
• Monitor for sort overflows and post threshold sorts via
MON_GET_DATABASE and MON_GET_UNIT_OF_WORK table
functions
• Adjust SHEAPTHRES_SHR upward and SORTHEAP upward
gradually to eliminate sort overflows
• Don’t increase SORTHEAP without concomitant adjustment to
SORTHEAP
• Can decrease SORTHEAP independently
• Mine the DYNAMIC SQL and target queries with high number
of overflows
• Good way to train a DBA in tuning SQL or identifying proper indexes to
eliminate SORTS
#IDUG
16
Setting the STMTHEAP DB CFG Parameter
• Since analytic queries are more complex than OLTP type
queries, they can benefit from an increased STMTHEAP DB CFG
size
• The STMTHEAP DB CFG parameter specifies the size of the
workspace used by the SQL compiler during compilation of an
SQL statement
#IDUG
17
Importance of Setting UTIL_HEAP_SZ Correctly
• For LOAD operations and for building compression dictionaries
UTIL_HEAP_SZ must be sufficiently sized
• Set to AUTOMATIC if DB2_WORKLOAD=ANALYTICS
• If DATABASE_MEMORY set to AUTOMATIC should be sufficient
as it will grow as necessary
• If not sized sufficiently, can result in less than optimal
compression
#IDUG
18
Monitoring Adequacy of UTIL_HEAP_SZ setting
• Check the new PCTENCODED column in the SYSCAT.COLUMNS
catalog view which is the the percentage of values that are
encoded as a result of compression for a column in a columnorganized table.
• If the overall compression ratio for you’re the table in is too
low, check this statistic to see if values in specific columns were
left uncompressed.
• Many columns with a very low value (or even 0) can be an indicator that
utility heap might have been too small when the column compression
dictionaries were created
• You might also see very low values for columns that were
incrementally loaded with data that was outside of the scope of the
column compression dictionary
#IDUG
19
So, How are your Column-organized Tables
Performing?
• Use monitor elements returned by the MON_GET_TABLE table
function to determine the efficiency of column-organized
tables and to identify candidate tables suitable to be organized
by column
#IDUG
20
New Column-organized Metrics from MON_GET_TABLE
table function
• Use monitor elements returned by the MON_GET_TABLE table
function to determine the efficiency of column-organized
tables and to identify candidate tables suitable to be organized
by column
• By dividing the NUM_COLUMNS_REFERENCED monitor
element by the SECTION_EXEC_WITH_COL_REFERENCES
monitor element, you can determine the average number of
columns that a particular query accesses. If this average is
much less than the number of columns in the table, the
workload favors column organization.
• Controlled by the mon_obj_metrics DB CFG parameter setting
• Extended is the default and is required to return column-organized
metrics
#IDUG
Monitor Collection DB CFG Parameters (Review)
•
•
•
•
•
•
Mon_act_metrics – controls collection of activity level monitor elements on the entire
database (DEFAULT – BASE)
• MON_GET_ACTIVITY_DETAILS
• MON_GET_PKG_CACHE_STMT
• Activity event monitor (DETAILS_XML monitor element in the event_activity logical data
groups)
Mon_deadlock – controls generation of deadlock events on the entire database (DEFAULTWITHOUT_HIST)
Mon_locktimeout – controls generation of lock timeout events on the entire database
(DEFAULT – NONE)
Mon_lockwait – controls generation of lock wait events for the lock event monitor (DEFAULT –
NONE)
Mon_lw_thresh – the amount of time spent in lock wait before an event for mon_lockwait is
generated (DEFAULT - 5000000)
Mon_obj_metrics – controls collection of data object monitor elements on the entire database
(DEFAULT- EXTENDED *formerly BASE)
• MON_GET_TABLE
#IDUG
22
Monitor Collection DB CFG Parameters (Review)
•
•
Mon_req_metrics – controls the collection of request monitor elements on the
entire database (DEFAULT – BASE)
• MON_GET_UNIT_OF_WORK
• MON_GET_UNIT_OF_WORK_DETAILS
• MON_GET_CONNECTION
• MON_GET_CONNECTION_DETAILS
• MON_GET_SERVICE_SUBCLASS
• MON_GET_SERVICE_SUBCLASS_DETAILS
• MON_GET_WORKLOAD
• MON_GET_WORKLOAD_DETAILS
• Statistics event monitor (DETAILS_XML monitor element in the event_wlstats
and event_scstats logical data groups)
• Unit of work event monitor
Mon_uow_data – controls the generation of UOW events at the database level for
the UOW event monitor (DEFAULT – NONE)
#IDUG
23
Monitoring and MONREPORT Reporting Enhancements
•
•
•
•
•
MONREPORT.DBSUMMARY and MONREPORT.CONNECTION reporting modules
enhanced to support on column processing
New monitor elements to assess bufferpool efficiency
Monitor elements contain COL in the monitoring element name
New monitor elements to monitor prefetch performance for data in column
organized tables
New monitor elements to measure column data size
•
•
•
•
•
•
•
ADMIN_GET_TAB_INFO
New monitor elements to measure time spent
New monitor elements for the HASHED GROUP BY operator
New monitor interfaces to get request metrics that are aggregated along different
dimensions
New monitor interfaces to get activity metrics
Monitor interfaces to get database object metrics
Refer to Database Monitoring Guide and Reference for complete details
#IDUG
24
New CTQ Optimization Operator
• Use to denote operations on column organized tables
• Most processing should be below the CTQ operator in the
access path (db2exfmt, db2explain)
• Optim Performance Manager
• Optim Query Tuner
• If many operations above this operator indicator of inefficient
use of column organized table
• Re-look how table is being used
#IDUG
25
Reviewing time spent in column processing versus row
processing with new monitor elements
• New monitoring elements returned from MON_GET_ACTIVITY_DETAILS
tabe function to show where time is being spent
• Total_app_section_excutions
• Total_col_executions
• Total_col_proc_time
• Total_col_time
• Total_cpu_time
#IDUG
Understanding Automatic Space Reclamation
• Space reclamation for column-organized tables
• Space reclamation after deletions
• After a delete operation the row is logically deleted
• Space that was used by the deleted row is not available to subsequent
transactions and remains unusable until space reclamation occurs
• REORG TABLE is required to reclaim the space
#IDUG
Understanding Automatic Space Reclamation
• Space reclamation after updates
• After a delete operation the row is first deleted and a new copy of the
row inserted at the end of the table
• Updated row uses space in proportion to the number of times that the
row is updated until space reclamation occurs
• All rows in the extent where the update took place must be deleted
before any space reclamation can occur
• REORG TABLE is required to reclaim the space
#IDUG
Understanding Automatic Space Reclamation
• If DB2_WORKLOAD=ANALYTICS then AUTO_REORG enabled
• Define an automatic reorg online and/or offline policy using
the AUTOMAINT_SET_POLICY or AUTOMAINT_SET_POLICYFILE
stored procedures with the appropriate parameters
• Offline reorg considerations
• Need appropriate window
• Reclaims space
• Faster than online REORG
• Sample POLICYFILE in
SAMPLE DB2 directory
#IDUG
Automatic Space Reclamation
• Online REORG considerations
• Still need appropriate window
• Schedule ONLINE REORG window during period of low activity
• Reclaims space by specifying
• Specify RECLAIM EXTENTS option of REORG command to
reclaim space
• To determine when a table REORG is necessary, use the
ADMIN_GET_TAB_INFO table function to review an estimate of
the amount of reclaimable space to reclaim
• Monitor REORG status with db2pd or SNAPTAB_REORG
administrative view
#IDUG
Summary
•
•
•
•
•
•
•
•
•
•
•
DB2 BLU New Column-organized capabilities
Use of AUTOCONFIGURE
Review of DB2_WORKLOAD=ANALYTICS registry setting
SHEAPTHRES_SHR and SORTHEAP setting discussion and
recommendations
Table monitoring metrics
Monitoring Enhancements and some review
Activity detail monitoring elements
New Monitoring Elements
Importance of adequate UTIL_HEAP_SZ
REORG Considerations
Space Usage and Reclamation
#IDUG
References
• Tuning and Monitoring System Performance, located at
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!
/wiki/Wc9a068d7f6a6_4434_aece_0d297ea80ab1/page/Tuning%20and%2
0Monitoring%20Database%20System%20Performance
• DB2 10.5 Database Monitor Guide and Reference,
http://public.dhe.ibm.com/ps/products/db2/info/vr105/pdf/en_US/DB2M
onitoring-db2f0e1050.pdf
• Best Practices, Optimizing Analytical Workloads using DB2 10.5 with BLU
Acceleration,
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!
/wiki/Wc9a068d7f6a6_4434_aece_0d297ea80ab1/page/Optimizing%20ana
lytic%20workloads%20using%20DB2%2010.5%20with%20BLU%20Accelerat
ion/comment/1dc59ee4-8c70-43f9-9d36-fba334b95891
#IDUG
References
• DB2 BLU Acceleration, Rapid Adoption Guide,
http://www.ibm.com/developerworks/data/library/techarticle/dm1309db2bluaccel/
• When RAM is too Slow, http://softwaretradecraft.com/when-ram-is-tooslow-how-dynamic-in-memory-processing-changes-the-game-for-dataanalytics/
• DB2 10.5, What’s New,
http://public.dhe.ibm.com/ps/products/db2/info/vr105/pdf/en_US/DB2W
hatsNew-db2q0e1050.pdf
#IDUG
Philip K. Gunning
Gunning Technology
Solutions, LLC
[email protected]
Monitoring the Essentials in DB2 10.5
DB2 LUW
Please fill out your session
evaluation before leaving!