Transcript Exadata Software Maintenance

Exadata Software
Maintenance
IOUG 2013 Exadata Virtual Conference
Doug Utzig
Consulting Member of Technical Staff
Oracle MAA Development
1
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Agenda
 Software Architecture Overview
 Planning
 Updating Exadata Software
 Updating Grid Infrastructure and Database Software
 Security Updates
 Best Practices
2
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Planned Maintenance Improvements
Continual Improvements
• Reduce Downtime
• Simplify Install
•
•
•
•
•
•
•
•
Auto Cell firmware update
Exadata bundle patch (BP)
Out-of-place patching step-by-step
Minimal Pack firmware update
Cell rolling patch apply
BP EM install
DB+GI BP merge
BP OPatch Auto install
2009-2010
3
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
•
•
•
•
•
•
•
•
•
• Reduce Risk
• Transfer Knowledge
BP DG Standby-First install
Minimal Pack Linux kernel update
BP RAC rolling install
BP OPlan
IB Switch update w/ ILOM
Linux upgrade step-by-step
11.2.0.2 upgrade step-by-step
Oracle Advisor webcasts
IOUG Exadata SIG webcast
2011
• 11.2.0.3 upgrade step-by-step
• 11.2.0.3 Exachk upgrade readiness
• 11.2.0.3 QDPE
• 11.2.0.3 QFSDP
• EM 12c Out-of-place patching
• Linux dbserver ULN updates
• Exadata channel ISO
• Flash ESM removal
2012-2013
Exadata Database Machine
Software Review
Database Grid
– Exadata Database Server operating system
(Linux or Solaris)
– Grid Infrastructure and Database
Storage Grid
– Exadata Storage Server Software
Networking
– InfiniBand switch software
Other: Ethernet switch, PDU, KVM
4
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Planning
5
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Planning
Latest Releases
 MOS 888828.1
– “The Patching Note”
– Critical Issues and additional required patches
– Guidelines for related products (OEM, DBFS, RM, GG, SAP)
6
Software Component
Latest Release
Recommended Minimum
Exadata
11.2.3.2.1
11.2.2.4.2
Grid Infrastructure and
Database
11.2.0.3: 11.2.0.3.14
11.2.0.2: BP19
11.2.0.3: 11.2.0.3.8 (Jul 2012 QDPE)
11.2.0.2: BP17
InfiniBand Switch
1.3.3-2
1.3.3-2
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Software Delivery
Quarterly Full Stack Download Patch (QFSDP)
 Follow quarterly updates for
planned maintenance
 Contains latest recommended
– Infrastructure
 Exadata Storage Server
 InfiniBand Switch
– Database
– SystemsManagement (EM)
|- Infrastructure/
|
--ExadataStorageServer/
|
-11.2.3.2.1/
|
--InfiniBandSwitchSoftware/
|
-1.3.3-2/
|
--SunRackIIPDUMeteringUnitFirmware/
|
-1.04/
|- Database/
|
--11.2.0.3_QDPE_Jan2013/
|
--OPatch/
|
--OPlan/
|- SystemsManagement/
|
--CloudControl12/
|
--Agent/
|
--WLS/
|
--OMS/
 Release aligned on PSU / CPU schedule
7
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Planned Maintenance
Suggested Schedule
Frequency
Planned Maintenance – Software Updates
Grid Infrastructure / Database quarterly patch (QDPE)
3-6 months
Exadata software
Grid Infrastructure / Database patch set
1-2 years
InfiniBand switch software
2-4 years
8
Grid Infrastructure / Database release (e.g. 11.1 to 11.2, or 11 to 12)
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Software Maintenance
Patching Tools
Component
Software
Patching Tool
Storage Server
Exadata
patchmgr
Exadata
Linux – YUM
Solaris – PKG
GI/DB QuarterlyPatch
OPatch
GI/DB PatchSet
OUI and DBUA
All
ILOM
Database Server
InfiniBand Switch
System readiness - Exachk (MOS 1070954.1)
9
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Updating Exadata Software
Exadata Storage Server
Exadata Database Server
10
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Exadata Storage Server
 Release contains all storage software / firmware
 patchmgr controls install and rollback
– One invocation patches all cells
Install nothing but this patch on cells
– Rollback to most recent successfully
No unsupported configuration changes
installed version
11
Rules
•Will be lost next update
•May cause next update to FAIL
Method
Downtime
Patch install time
Rolling
None
Up to 2 hours per cell
Non-Rolling
Up to 2 hours total
Up to 2 hours total
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Exadata Database Server Operating System
Oracle Linux
Two types of packages
1. Exadata Core Packages (update required)
 Coincides with Exadata Storage Server release
 YUM update from Exadata ULN channel only (or the ISO)
 Includes firmware
Exadata Core Packages
kernel*
kexec-tools
<more in Owner’s Guide>
2. Other Packages (--exclude Exadata Core Packages) (update optional)
 Update existing Other Packages from ULN (ol5_x86_64_latest channel)
 Install new Other Packages
 e.g. backup agents, monitoring agents
# yum -–exclude=kernel-* --exclude=… update
12
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Updating Grid
Infrastructure and
Database Software
13
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Grid Infrastructure and Database
Quarterly Patch Update
 OPatch installed and rolled back
– Patches are cumulative
– One invocation per dbnode
 Methods to reduce downtime
and risk
14
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
README
Grid Infrastructure and Database
Out-of-Place (OOP) Patching
Patch inactive home
to reduce risk
Benefit for Quarterly Patch Update
1.
Clone home (GI or DB)
2.
OPatch cloned home
3.
Switch current home to cloned home (rolling or non-rolling)
Performed during normal business
hours, ahead of “patching window”
 Enterprise Manager 12c and OPlan supported
15
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Grid Infrastructure and Database
OPlan
$ oplan generateApplySteps /tmp/13556724
 Generates step-by-step
patching instructions
– Specific to your
environment
– 3 options
– Like OPatch, always
get the latest
16
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
ApplyInstructions.html
Grid Infrastructure and Database
Patching Exadata with EM
Database Grid
•Grid Infrastructure
•Database
 Apply Exadata Quarterly
Database patches
– GI and DB homes
– Orchestrates across all database
servers
 Multiple modes supported
– Out-of-Place or In-Place
– Rolling or Non-Rolling
17
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
EM Out-of-Place Patching - Recommended
Analyze (Before patching window)
1. Stage patch and perform pre-apply check
Prepare (Before patching window)
2. Clone Oracle homes (GI and Database)
3. Apply patch to cloned Oracle homes
Deploy (Switch, Rolling or Non-Rolling)
4. Switch GI to cloned Oracle home
5. Switch databases to cloned Oracle home
6. Apply SQL portion of the patch
Grid Infrastructure and Database
Patch Sets
• Follow Exadata validated steps
• 11.2.0.3 - MOS 1373255.1
• 11.2.0.2 - MOS 1315926.1
• Grid Infrastructure upgrade
• Rolling
• Database upgrade
• Requires downtime
• Storage Server compatibility
18
Database
Required Storage
11.2.0.3
11.2.2.4 or later
11.2.0.2
11.2.2 or later
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
• Reduce downtime and risk
• (Transient) Logical standby
• Golden Gate
Security Updates
19
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Security Updates
Exadata Storage Server
Exadata Database Server - Oracle Linux
 All software (“Closed” system)
 Exadata Core Packages - Review MOS
1405320.1
 Update to later Exadata release
 Update to later Exadata release
 Open Service Request for CVEs not covered
 Other packages
 Update typically from ol5_x86_64_latest
 -- exclude Exadata Core Packages
Database Server Grid Infrastructure and Database
 Quarterly Database Patch for Exadata
 Contains concurrent PSU/CPU content
20
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Best Practices
21
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Methods
Rolling versus Non-Rolling
Rolling
Lowest Database Downtime
Lowest Installation Time
 Patch Non-Rolling if maintenance
window allows
 Patch Rolling during low workload
 Mix-and-match
22
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Non-Rolling


Exadata Storage Server Rolling Patching
 Reduce disk failure risk w/ high redundancy (or DG)
Database Server (Exadata & GI/DB) Rolling Patching
 Use Automatic Workload Management
Reduce Risk and Downtime with Data Guard
Data Guard
 Reduce Planned Maintenance Risk and
Downtime with Data Guard
– Upgrade standby system and switchover
Independent Components (Upgrade Standby with no impact to Primary)
 Exadata software (Storage and Database Servers)
 InfiniBand Switch software
 Database Server Grid Infrastructure home
Dependent Components
 Database Server Database home (MOS 1265700.1)
 Data Guard Standby-First Installable qualified patches only
23
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Consolidation and Large Environments
Consolidation
Large Environments
GI version >= DB version
Mixed Exadata version supported
 Required to 4th digit, recommended to 5th digit
Advance diskgroup compatible.rdbms with care
 Once advanced cannot be reset
Share database homes whenever possible
 Storage servers and database servers may
run different versions
(recommended only during rolling upgrade)
Split updates to multiple components across
more than one maintenance window
 Extend /u01 size
 In general – update up the stack
 Patch Out-of-Place
1. InfiniBand switches
 Move databases individually to new home
2. Exadata Storage Servers
3. Database Servers OS / Firmware
4. Grid Infrastructure / Database
24
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Avoid Common Pitfalls
Exadata Storage Server
 Do NOT make unsupported configuration changes
Exadata Database Server OS and Firmware
 Test previous configuration changes to avoid latent patching failures
 Backup OS using validated method (e.g. dbserver_backup.sh)
Database Server Grid Infrastructure and Database
 Follow Exadata step-by-step MOS note for patch set upgrade
 CV_DESTLOC avoids cluvfy conflicts during concurrent opatch auto
 Ensure unzipped bundle patch location has proper permission
 Shutdown software as documented
25
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
General
 Stay up-to-date
 Familiarize with doc and
known issues
 Test and rehearse
 Run latest Exachk regularly
 Fix Exachk warnings and
alerts before patching
Oracle Support for Exadata
ORACLE PLATINUM SERVICES
Complete. Integrated. Proactive.
High Availability. No Additional Cost.
 Higher support levels for the complete Oracle stack
 24/7 support
–
 Specialized Engineered Systems Support Team
Includes higher support levels for Database software
 Proactive remote monitoring for faults
 2-hour onsite response to hardware issues1
 Industry leading service level response times:
 New Updates and Upgrades for Database, Server,
Storage, and OS software
 My Oracle Support proactive support portal
 "Phone home" automated service requests (ASR)
–
5 Minute Fault Notification
–
15 Minute Restoration or Escalation to Development
–
30 Minute Joint Debugging with Development
 Oracle Engineers perform quarterly patching and
updates
Available for certified configurations on Exadata
26
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
1 Covered
system must be within an Oracle two-hour service area to receive two-hour response as a standard service.
27
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.