Transcript Northern Trust - Lanyon Events

NORTHERN TRUST
Northern Trust’s Strategy for Adopting Multitenant as a
Foundation for DBaaS
Oracle OpenWorld 2015
Kevin Hacker, Manager Database Engineering,
The Northern Trust Company
©12015 Northern Trust Corporation
northerntrust.com
About Northern Trust

Chicago-based Global leader in delivering innovative investment
management, asset and fund administration, fiduciary and banking
solutions to corporations, institutions and affluent individuals.

Established in 1889

Over 20 international locations:



2
USA, Europe, Asia, Canada, Middle East, Africa, Australia & New
Zealand
As of June 30, 2015, Northern Trust Corporation had:

$120 billion in banking assets

$6.2 trillion in assets under custody

$946 billion in assets under management
16,000+ employees
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Industry Awards
Recent Distinctions
•
Best Banks in America
Forbes, 2015
•
World’s Most Admired Companies
Fortune Magazine, 2015 – Ninth Consecutive Year
•
Best Private Bank in U.S.
Financial Times Group, October 2014 – Sixth Consecutive Year
•
A World's Most Ethical Company
Ethisphere Institute, 2015
•
Best Places to Work
Pension & Investments, 2014- Second Consecutive Year
Technology
3
•
Best Infrastructure Initiative-American Financial Technology Award
Waters Magazine, 2014
•
Mobile Marketing Excellence in Wealth Management
Digital Marketing of Financial Services, 2014
•
Top 500 Innovators in IT
Information Week, 1997 - 2013
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
About Myself
Kevin Hacker
Manager, Database Engineering
Email: [email protected]
4

25 Years experience as DBA (DB2, SQL Server, Sybase, Oracle)

13 Years experience working in Oracle technologies

Worked at Northern Trust since 2001

Northern Trust Manager Database Engineering since 2014

Master’s degree in Mathematics from Purdue University

Bachelor of Arts in Mathematics and Computer Science from
University of Indianapolis
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
About Northern Trust Database Organization
Engineering
Operations
Database
Organization
Engineering
and Projects
DB2
(and IMS)
7 DBAs
5
Engineering
Projects
5 DBAs
1 Consultant
3 DBAs
8 Consultants
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Production
Health
3 DBAs
Operations
and Support
3 DBAs
24 Offshore
Consultants
Northern Trust Current Database Environment
Oracle
Release
Operating
System
Total
9i
Solaris 8
7
Solaris 10
58
Solaris 8
8
Solaris 10
238
Solaris 10
30
Linux REL 5
726
11g RAC
Exadata OEL 6
24
12c
Containers
Exadata OEL 6
39
10g
11g
DBA Employees (Oracle): 10
DBA Consultants:(Oracle): 23
6
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Northern Trust Timeline with Oracle
P
E
R
F
O
R
M
A
N
C
E
7
M
A
N
A
G
E
A
B
I
L
I
T
Y
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Northern Trust Challenges
Lack of Standardization on Database Platforms
•
•
•
•
Solaris 8 and 10
Red Hat Enterprise Linux on VMware
Custom built RAC on Red Hat Enterprise Linux with HP blades
Exadata on Oracle Linux
Lack of Standardization on Database Software
•
•
Oracle 9i, 10g, and 11g
Different licensing options for RAC, Partitioning, Advanced Compression, Diagnostics and Tuning
Lack of Standardization on Patching
•
OS Team to patch Solaris and Linux OS on physical, VM guests, and home-grown RAC nodes
•
DBA Teams to patch databases (multiple flavors)
•
VMware Team to patch ESX Clusters
•
Oracle ACS to patch Exadata Clusters (Storage Cells, OS, and Databases)
Lack of Standardization on DR Solution
•
•
8
Hitachi Storage Replication for most VM guests and physical servers
Oracle Data Guard for Oracle RAC, Exadata, and some critical VM guests
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Northern Trust Challenges
Lack of Standardization Consequences
•
High Effort of Management
•
•
•
•
•
Virtualization led to copy of OS and database binaries for every database
One database per application (no schema sharing)
One VM guest per database (no OS sharing)
Thousands of database environments to patch
Multiple DR procedures to follow
• Increasing Pressure to Deliver Databases Faster
• Long Turnaround Time to Provision Database: 2 to 3 weeks for VM - Longer for RAC
9
•
Multiple teams required to provision VM database
• Storage Team to provision storage
• VMware Team to provision VM Guest
• Platform Team to provision OS
• DBA Standard Build Team to provision database software and database
•
Oracle RAC databases (Highly critical apps) very complicated to build
• No automated build processes
• 200+ page document to manually build RAC database and setup Data Guard
• Manual process is error prone
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Northern Trust Challenges
The Solution: A Standardized Foundation with Consolidation
•
Standardize the Platform
•
•
Standardize the Software
•
•
•
Oracle Data Guard Replication
Standardize the Offerings
•
•
10
Engage Oracle Platinum Support
Standardize the DR Solution
•
•
Oracle 12c with the Multitenant option
Standard software option: RAC, Partitioning, Advanced Compression, Active Data Guard,
Diagnostics and Tuning
Standardize the Patching
•
•
Exadata on Oracle Linux
Develop automated build scripts to lay groundwork for DBaaS
Develop service catalog with silver, gold and platinum offerings
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Northern Trust 12c Infrastructure Project
Thus was born …
The Oracle 12c Multitenant Project
Date of Conception:
Spring, 2014
Delivery Date (First Live Application): September 17, 2015
Date of Maturity:
TBD
11
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Oracle 12C Infrastructure Project
Build Infrastructure for Oracle 12C
Goals
12

Create and Design Oracle 12C Standards, Infrastructure, and Project Procedures

Consolidate Databases using Oracle 12C Multitenant Strategy (CDBs and PDBs)

Standardize on one Platform (Exadata – ULA)

Reduce Deployment Time for Databases

Provide Different Tiering Options (Silver, Gold, and Platinum)

Develop Patching Strategy

Position Oracle 12C for “Database as a Service” Offering
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Oracle 12C Infrastructure Project
• Storage Savings – Shared Binaries and OS
• Less Environments to Patch – Platinum Agreement
• More Efficient Use of Resources – Fewer Background
Processes
• High Availability – All UAT and Prod will be RAC
Databases
Benefits
• Granular Failover for DR (Data Guard) – Container Level
• Fewer Databases to Maintain – Backups at Container
Level
• Quick Deployment – One Team Controls Everything
• Consistent Builds – Automated Deployment Procedures
• High Performing Hardware – Exadata
• Multiple Tiers Allow for Different Service Levels
13
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Oracle 12C Infrastructure Project
• New Release of Oracle (12.1.0.2)
• Starting with Less Critical Apps (Silver)
• 12.2 Not Scheduled for Release until 2016
Mitigation
• Few Companies using Multitenant
• Reference Calls (Pulte Homes and Sherwin Williams)
• Working Closely with Oracle (SCP Agreement)
Risks
Mitigation
• Potential Performance Impact (Sharing Containers and Hosts)
• Instance Caging
• Resource Manager (CDB and PDB Levels)
• IO Resource Manager
• Quick Ability to Unplug/Plug/Clone to another Container
Mitigation
• More Impact for Planned/Unplanned Outage
• Data Guard for Switchover
• Rolling Patches for Gold/Platinum Environment
14
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Mitigation
Oracle 12C Tiering Strategy
Tier Definitions and Characteristics
Silver
Gold
Platinum
Silver
Gold
Platinum
15

Non-Critical Databases

Few Densely Packed Containers (1 CDB to many PDBs)

Non-Rolling Patching of Exadata Cluster

Oversubscription

Business Critical Applications (Key Applications)

Some Multitenant Databases (Shared Containers for same
Business Unit)

Rolling Patching of Exadata Cluster

Little to No Oversubscription

Mission Critical, Revenue Generating, or Customer-Facing
Applications

No Multitenant Databases (CDB’s can have only 1 PDB)

Rolling Patching of Exadata

No Oversubscription
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Oracle 12C Tiering Strategy
Sample Application Dimensions and Exadata Tier Assignments
Exadata Tiers
Application Dimensions
Maintenance Window
Sat 5:00pm to Sun 4:00am (Current Tier 2)
Sun 12:00am to Sun 4:00am (Current Tier 1)
Capacity
Size of db
Memory Requirements
CPU Requirements
Criticality of Application (SLA)
Mission Critical (Money Movement) - Top 10
Key Apps - Top 50
Non-Critical
Predictable Performance
Critical
Not Critical
Shareability
Can share with no issues
Can share with related apps (sub-TLAs,
common ownership, etc.)
Can not share
Use Active Data Guard for Read-Ony Reporting
Silver
Gold
Platinum
x
x
x
x
>= 2 Terabytes
> y GB
> 16 (?)
>= 2 Terabytes
> y GB
> 16 (?)
x
x
x
x
x
< 2 Terabytes
< x GB
< 16 (?)
x
x
x
x
Disclaimer: This is a sample guideline and not a “hard and fast” set of rules
16
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
x
x
x
x
Oracle 12C Naming Standards
Needed to Develop Naming Standards for 12c Databases (Containers and Pluggables)
•
•
•
•
•
Distinguish between existing pre-12c databases
Distinguish between CDB and PDB
Embed Application Identifier in PDB Name
Embed Environment Indicator in Name
Force same set of PDBs to reside in same Container Series throughout Migration Lifecycle
Development (D)
Production (P)
CDB001A
CDB001P
pdbabc01d
pdbabc01d
pdbjkl01a
pdbabc01a
pdbjkl01p
pdbabc01p
pdbdef01d
pdbdef01d
pdbxyz01a
pdbdef01a
pdbxyz01p
pdbdef01p
pdbghi01d
pdbghi01a
pdbghi01p
CDB001D
17
UAT (A)
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Discovery Testing of 12C
Discovery Testing Phase - Kicking the Tires
•
No Knowledge or Experience with Multitenancy
•
Acquired Exadata Loaner Machine from Oracle
•
Developed Formal Test Plans and Parallel Testing Streams
•
4+ Months of Testing and Recording Results
•
Met Weekly to Discuss Findings
•
Created SRs for Bugs
18
Testing Streams
Cloud
Mgmt Pack
Testing
Multitenant
Testing
RAC
Testing
Data Guard
Testing
Backup and
Recovery
Testing
Resource
Manager
Testing
Security
Testing
Monitoring
Testing
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Discovery Testing of 12C
Bugs (and Features) Discovered
•
•
•
Cloud Mgmt Pack did not meet our Requirements
•
No support for Container Placement of PDBs and no full control over PDB Name
•
Inconsistent error logging and cleanup
•
Oracle incorporated our suggestions into next release
•
Decided to develop custom automated scripts for provisioning CDBs, PDBs, and Data Guard
“Max Parallel Server Limit” not honored in CDB Resource Plan
•
Oracle will fix in release 12.2
•
Our Workaround:
Set “max degree of parallelism” = 4 and the “Bypass Queue” = “Y” for each Consumer Group in PDB Resource Plan
Our Workaround: Ignore “Save State” and Use Startup Trigger to Open all PDBs on Primary and Standby Containers
Oracle fixed in one-off and quarterly patch
PDB Cloning/Plugging Complicated in Data Guard Environment
•
19
•
RMAN Resynch Issue when Backing up on Standby after adding PDB or data file to Primary
•
•
Change “parallel_degree_policy” from "manual" to "auto“ in spfile
Cannot Enable Auto Open of PDB's on Standby using “Save State” (Bug 19728794)
•
•
•
Requires turning off “log apply” during operation
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Resource Manager
Instance Caging: Controls CPU Usage of Each Container Database (CPU_COUNT)
Resource Manager: Two Levels of Plans
1.
CDB Resource Plan: Controls Resources for each PDB
Considerations
2.
•
Cap each PDB at 10% as starting point
•
Load test to ensure max is sufficient
•
•
Pluggable
Database
Shares
PDB1
2
Increase max value if necessary
PDB2
Ensure consistent performance as
PDBs are added
Max
Utilization
limit
10%
2
10%
$x BN
10%
PDB3
2
10%
10%
DEFAULT
2
10%
10%
10%
PDB Resource Plan: Controls Resources within each PDB
•
Map Services to Consumer Groups as
needed
Consumer Group
Shares
Utilization limit
TNT_Priority_1_Group
12
10%
•
Give “Admin” Top Priority
TNT_Priority_2_Group
4
10%
TNT_Priority_3_Group
2
10%
IO Resource Manager: Exadata Only Feature
•
20
Max Parallel
server limit
We use “IORMPLAN Objective” value of “Auto” (Best Practice “Starting Value”)
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Design and Development
Configuration Management Database Changes
•
•
•
21
Northern Trust has Internally Developed CMDB (Oracle database backend)
•
Inventory of Hosts, Clusters, Databases, Applications (Oracle, Sybase, SQL Server, MySQL)
•
Application Information (Maintenance Windows, Services, Contacts, Configuration Parameters, etc.)
•
Collection Scripts update nightly (Release, Parameters, Space information, etc.)
•
Metric Reporting for Management
•
Used to Drive Administrative Processing against all Databases
Data Model Changes Required to store Oracle 12C Metadata
•
Extensive Schema Changes to Support CDB and PDBs
•
Reduce Impact to Existing Processes and User Interface
•
Conversion Effort to fit Existing Data into New Structures
Script and User Interface Changes
•
Hundreds of Script Changes Needed for Updated Data Model
•
New Collection Scripts Required for 12C databases
•
Many User Interface Changes for 12C Objects
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Oracle 12C Automated Build Scripts
Developed Automated Scripts for Deploying CDBs and PDBs

Replaces 200+ Pages of Documentation for Manual RAC Database Build and Data Guard Setup

Consistency of Builds (Eliminates Human Error from Manual Process)

Integration with internal CMDB database – Validates and Inserts Rows when CDB and PDB are Created

Faster Deployment -- Reduction in Time from 3 to 4 Weeks to a Few Hours
Automated Scripts

“Northernize” Exadata Cluster -- Distribute Maintenance Scripts, etc.

Create Container Database
Add DBA Users

Configure RMAN for Backup

Setup Auditing

Setup CDB Resource Plan and Instance Caging

Register CDB with OEM and Assign Department and Life Cycle Property

Create PDB

22


Create TLA Objects

Setup Auditing

Register with OEM and Set Life Cycle Property
Build Physical Standby CDB for Production
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
-- One Time per Exadata Host
Oracle 12C Exadata Environment
Exadata Purchases
•
Purchased and Installed 7 ½ Rack Exadata Clusters (Late 2014 – Early 2015)
•
Separate Physical Clusters by Tier
•
•
•
1 Exadata Cluster for Shared Development for Silver, Gold, and Platinum
3 Exadata Clusters for Silver Tier (UAT, Prod, and DR)
3 Exadata Clusters for Gold/Platinum Tier (UAT, Prod, and DR)
Data Center A
Data Center B
UAT
Development
Silver Cluster
Shared Tier Cluster
Silver/Gold/Platinum
Gold/Platinum Cluster
DR
Prod
Silver Cluster
Gold/Platinum Cluster
Silver Cluster
Gold/Platinum Cluster
Data Guard Replication
Data Guard Replication
23
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Oracle 12c Exadata Environment
Q: Biggest Pain Point Prior to 12c Project …. ?
I Needed it
Yesterday !
A: Pressure to Deliver Environments … FAST
Where is
my Database ?
Two Most Noticeable Benefits of Oracle 12c Multitenancy on Exadata:
1. Speed of Deployment -- Deploy a PDB in less than 30 minutes
•
•
•
•
•
Container RAC Databases already pre-built
No waiting on Storage or OS
Backup jobs already scheduled and deployed (CDB level)
Data Guard already established
DBA has full control (storage and OS
2. Extreme Performance
•
24
Exadata is Fast!
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Oracle 12C Exadata Next Steps
Next Steps …..


25
Refine and Formalize Container Placement Process for New PDBs

Determine Container

Determine Instance

Determine Capacity
Transition to OEM Cloud Management Pack for PDB Deployment

Upgrade OMS and Plugins to Required Release Level

Investigate Self Service Features

Integrate custom-built automated deployment scripts into OEM

Use OEM GUI to Build PDBs (End of Year 2015)

Investigate Incorporating CDB and Data Guard Scripts via OEM (2016)
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
BUILD PDB
Oracle 12c Consolidation Cost Benefits
Feature
Benefit
Simplified, dense consolidation
Reduced capital expenditures
Rapid Provisioning
Reduced operational expenses
Reduced time to market (30 minutes to deploy PDB)
One Team in Control (DBA)
Resource Manager
Tenants get fair share of computing resources
RAC Compatibility
Greater Scalability , Availability & Resource Utilization
Quick Deployment with Automated Scripts
Data Guard Replication
(instead of SAN Replication)
Reduced DR switchover time from hours to minutes
One Team in Control (DBA)
Exadata Hardware
Reduced Capital Expenditures(ULA)
Extreme Performance
26
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS
Q&A
Questions?
27
Northern Trust’s Strategy for Adopting Multitenant as a Foundation for DBaaS