Transcript DBOD_rubenGx

Ruben.Gaspar.Aparicio_@_cern.ch
On behalf DBoD team, IT Department
HEPIX 2014
Nebraska Union, University of Nebraska – Lincoln, USA
Agenda
•
•
•
•
•
•
•
•
•
•
•
Manifesto
Current status
Architecture
Demo
Management
Infrastructure
High Availability
Monitoring
Data protection: backups and recoveries in detail
Future development
Summary
3
Agenda
•
•
•
•
•
•
•
•
•
•
•
Manifesto
Current status
Architecture
Demo
Management
Infrastructure
High Availability
Monitoring
Data protection: backups and recoveries in detail
Future development
Summary
4
Manifesto
•
https://cern.ch/twiki/bin/view/DB/DBOnDemandManifesto
•
Making users database owners
•
•
Covers a demand from CERN community not addressed by the
Oracle service
•
•
•
•
•
•
Full DBA privileges
Different RDBMS: MySQL, PostgreSQL and Oracle (private use)
No access to underlying hardware
Foreseen as single instance service
No DBA support or application support
No vendor support (except for Oracle)
It provides tools to manage DBA actions: configuration,
start/stop, upgrades, backups & recoveries, instance monitoring
5
Agenda
•
•
•
•
•
•
•
•
•
•
•
Manifesto
Current status
Architecture
Demo
Management
Infrastructure
High Availability
Monitoring
Data protection: backups and recoveries in detail
Future development
Summary
6
Current status
7
Current status
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Openstack
Puppetdb (MySQL)
Lhcb-dirac
Atlassian databases
LCG VOMS
Geant4
Hammercloud dbs
Webcast
QC LHC Splice
FTS3
DRUPAL
CernVM
VCS
IAXO
UNOSAT
…
Database usually supports an open source/commercial software
8
Agenda
•
•
•
•
•
•
•
•
•
•
•
Manifesto
Current status
Architecture
Demo
Management
Infrastructure
High Availability
Monitoring
Data protection: backups and recoveries in detail
Future development
Summary
9
Architecture
Oracle VM &
physical servers
Syscontrol
FIM
FIM DB
DAEMON
DBOD DB
Storage network
DBOD WS
https://cern.ch/resources
WEB APP
CERN AI
MONITORING
https://cern.ch/dbondemand
RACMON DB
RACMON
Data
https://oem.cern.ch
Logs
ORACLE
EM
Diag (Oracle)
DB client
10
Agenda
•
•
•
•
•
•
•
•
•
•
•
Manifesto
Current status
Architecture
Demo
Management
Infrastructure
High Availability
Monitoring
Data protection: backups and recoveries in detail
Future development
Summary
11
https://cern.ch/dbondemand
(Please check http://indico.cern.ch/event/313869/ from 12’13’’ )
12
Agenda
•
•
•
•
•
•
•
•
•
•
•
Manifesto
Current status
Architecture
Demo
Management
Infrastructure
High Availability
Monitoring
Data protection: backups and recoveries in detail
Future development
Summary
13
DBOD Daemon
•
Small program which:
• Fetches to-be executed jobs from the database
• Manage jobs execution (via IT-DB framework)
• Carries job post-execution tasks, if necessary
• Updates the application DB with job results and
instance status
• Executes around ~350 jobs per day for DBoD
and ~ 900 jobs for MiddleWareOnDemand
•
Modular design with focus on expansion
•
•
Easy to add support for new systems
(MiddleWareOnDemand)
Reusable code
14
DBOD State Checker
•
•
•
Part of the daemon package.
Cron managed script which periodically checks
each instance availability and accordingly updates
its status in the DB
Necessary to correctly display externally caused
changes to the status of the service instances (e.g.
host downtime, network issues, etc) in the user
interface
15
Migration to the CERN Agile
Infrastructure
•
•
IT-DB Virtualization infrastructure is being
migrated from RHEL + OVM to the standard
CERN AI OpenStack setup (KVM + SLC)
Storage access performance is vital to DB
applications
•
IT-DB runs its own OpenStack installation on
servers physically connected to its storage
servers for performance reasons
16
Migration to the CERN Agile
Infrastructure
•
•
DBOD customized RPM packages for MySQL
and PostgreSQL servers already built using Koji
A Puppet module configures each host
according to the instance-resource relations
stored on the Syscontrol LDAP directory
•
NAS Volumes, service startup scripts, users, etc.
17
Agenda
•
•
•
•
•
•
•
•
•
•
•
Manifesto
Current status
Architecture
Demo
Management
Infrastructure
High Availability
Monitoring
Data protection: backups and recoveries in detail
Future development
Summary
18
High Availability
•
•
•
Driven by demand/need, not initially in the plans
Not relying on virtualization features so far (it may
change in the future, as OpenStack evolves)
4 node clusters
•
•
Nowadays two clusters running under Oracle cluster
ware 12.1.0.1.
Clusterware controls:
•
•
Virtual IP
RDBMS instance
•
PostgreSQL and MySQL instances can co-exist, different
versions supported.
19
High Availability
Testing the cluster
(MySQL & Postgresql
instances)
•
Failover test\Downtime
Avg. (s)
Min. (s)
Max(s)
Kill process
16.9
4
39
Kill process (different node)
21.7
10
34
Network down
39.9
37
47
Server down
37
33
43
Relocate
6.2
5
7
For instances running on an Oracle cluster ware, care
must be taken in case of server crash for MySQL
instances.
•
"InnoDB: Unable to lock ./ibdata1, error: 11" Error Sometimes
Seen With MySQL on NFS (Doc ID 1522745.1)
20
Agenda
•
•
•
•
•
•
•
•
•
•
•
Manifesto
Current status
Architecture
Demo
Management
Infrastructure
High Availability
Monitoring
Data protection: backups and recoveries in detail
Future development
Summary
21
Infrastructure: Hardware servers
•
Dell blades PowerEdge M610
•
•
2x Quad-Core Intel Xeon @ 2.53GHz
48 GB RAM
Public
Network
NetApp
cluster
Private
Network
10GbE
Next release
•
Transtec Database server
•
•
2x eight-core Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
128 GB RAM
22
Agenda
•
•
•
•
•
•
•
•
•
•
Manifesto
Current status
Architecture
Demo
Management
Infrastructure
Monitoring
Data protection: backups and recoveries in
detail
Future development
Summary
23
Monitoring
•
•
Critical component for both DBoD managers
and DBAs (our clients)
Different monitoring tools, two main categories
•
•
•
Servers, OS, Storage →
Netapp tools
DBoD instance:
+ Scripts +
+ RACmon
Trying to outsource DBoD instance monitoring
•
Very demanding task: configure alerts, adapt to
new RDBMS releases, add functionality like
reporting, execution plans,etc.
24
Monitoring
•
Three main axes
•
•
•
IT GNI service, IT dashboards
Oracle being monitored by
→ fine
grained access per pluggable database in
Oracle12c
MySQL, PostgreSQL monitored by
• Very intuitive interface
• Database – storage volumes – protocol correlation
• DB activity view and SQL analysis
25
Monitoring (nowadays view)
26
Monitoring: Enterprise manager 12c
Pluggable database
Container database
27
Monitoring: AppDynamics
28
Monitoring: AppDynamics
29
Monitoring: AppDynamics
30
Agenda
•
•
•
•
•
•
•
•
•
•
Manifesto
Current status
Architecture
Demo
Management
High Availability
Monitoring
Data protection: backups and recoveries in
detail
Future development
Conclusions
31
Storage evolution
FAS3240
FAS8060
NVRAM
1.0 GB
8.0 GB
System memory
8GB
64GB
CPU
1 x 64-bit 4-core 2.33 Ghz
2 x 64-bit 8-core 2.10 Ghz
SSD layer
(maximum)
512GB
8TB
Aggregate size
180TB
400TB
OS controller
Data ONTAP®
7-mode
Data ONTAP®
C-mode*
scaling up
scaling out
* Cluster made of 8 controllers (FAS8060 & FAS6220). Shared with other
services.
32
Data protection
•
•
•
•
•
•
2 file systems: data + redo logs on different
Netapp appliances
Storage is monitored: Netapp tools + home made
tools
Multipath access to disks (redundancy +
performance) → disks are seen by two controllers
(HA pair) → Transparent interventions
RAID6
Automatic scrubbing (based on checksum)
Rapid RAID Recovery + Disk Maintenance
Center
33
Backup management
•
•
Same backup procedure for all RDBMS. Only data
volume is snapshot.
Backup workflow:
… some
time later
snapshot
resume
mysql> FLUSH TABLES
WITH READ LOCK;
mysql> FLUSH LOGS;
or
Oracle>alter database
begin backup;
Or
Postgresql> SELECT
pg_start_backup('$SNAP');
new
snapshot
mysql> UNLOCK TABLES;
Or
Oracle>alter database end backup;
or
Postgresql> SELECT
pg_stop_backup(),
pg_create_restore_point('$SNAP');
34
Snapshots
•
Taken programmatically via our API using ZAPI
(NetappManagementSDK)
•
•
Logs can be controlled via DB On Demand site
It is a very fast operation. Example:
Hc_atlas (MySQL) about 3 secs
Indico_p (PostgreSQL) about 3 secs
35
Tape backups
•
Driven by demand/need, not initially in the
plans
• Likely to be removed
• Possible only on PostgreSQL and MySQL
•
•
•
•
•
Oracle12c solution comes already with a tape
backup!
Consistent snapshot + redo logs sent to tape
Database activity is not impacted
Tape backups are not validated
Manual process to set them up, need to
contact us (DBOD + TSM service)
36
Instance restore
Automatic snapshots
Data files
TIME
Now
Point-in-time
recovery
Binary logs
Manual snapshot
37
37
Agenda
•
•
•
•
•
•
•
•
•
•
Manifesto
Current status
Architecture
Demo
Management
High Availability
Monitoring
Data protection: backups and recoveries in
detail
Future development
Summary
38
High density consolidation: LXC
•
•
Scaling up servers (128GB RAM, 32 CPUs),
LXC should help to consolidate even more.
Red Hat 7 Atomic Host
Fine control on memory and
CPU using control groups
MySQL 5.5.30 - sysbench 0.5 query test - data set fits into innodb buffer
39
Data protection: SnapVault
Based on snapshots
*Image taken from Netapp documentation.
It should cover tape backup functionality → Disaster Recovery location
40
Agenda
•
•
•
•
•
•
•
•
•
Manifesto
Current status
Architecture
Demo
Management
Monitoring
Data protection: backups and recoveries in
detail
Future development
Summary
41
Summary
•
•
Many lessons learned during the design and
implementation of the DBoD service
Building Database as a Service helped CERN DB group to
•
•
•
•
•
•
Face new use cases from CERN community
•
•
Gain experience with MySQL, PostgreSQL and multi-tenancy Oracle
12c
Provide a solution for Oracle database with special needs e.g.
Unicode character sets
Improve tools and operations
Standardize on tools and frameworks
Consolidate
e.g. Increase data protection
On-going integration with IT central services
42
Acknowledge
•
IT-DB colleagues
•
•
Our former colleagues Daniel Gomez Blanco and
Dawid Wojcik
Ignacio Coterillo and David Collados as
members of DBoD team
43
Questions
44