Transcript Slide 1

Deploying E-Business
Suit 11i on Oracle RAC
11g on AIX
Oracle OpenWorld
September 2010
Introduction
Agenda
•
•
•
•
•
•
•
•
•
•
•
•
•
Bunnings at a glance
Our Business Benefits of RAC
Infrastructure Capacity Management
Initial Build and Deployment Experience
Our Build and Design
Preparation and pre-requisites
Patching and Installation
RAC conversion
– RDBMS
– Required Mid-tier changes
OEM Configuration
Issues and Resolution
Useful References
Next steps
Q&A
WIDEST RANGE
LOWEST PRICES
BEST SERVICE
Sales = IT Challenges
Sales ($m)
7000
6000
5000
4000
3000
2000
1000
0
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Sales
Our Business Benefits of RAC
• Ensuring business core applications remain and have
the needed Flexibility and Agility
• High Availability 24x7 Australia and New Zealand
Operations
• Deliver on Performance
• Provide a level of Workload Segregation
“Our Mission critical systems must be agile to support our
business by improving operational effectiveness as well
as supporting decision making which is based on timely,
accurate and flexible information”
User Uniqueness
• PDT – ordering
• Reporting – Discoverer
• Store Support
– Forms
– Self-service
– Custom Forms and Self-service
• Stores
– MC-75 PDA/PDT running self-service
– Touch screen and pen input
– WIFI and 3G Wireless
• Live reporting via XML Publisher and Discoverer 10g
Our Mobile Procurement Solution
Infrastructure Capacity Management
No. Transactions Per Month
Transactions Processed
** Chart is actual data from 2002 to 2009, covering total Transactions for Year on Year growth
Infrastructure Capacity Management
• Information Lifecycle Management is part of the solution
• EBS partitioning needs to be done early otherwise
special issues come up if regular purging cannot be
undertaken due to business requirements
– Inventory, GL, AP, AR, PO can grow quickly
depending on use
– Moving data in live systems is tricky
Metalink article 554539.1 (Using Database Partitioning with Oracle E-Business Suite )
Has all the information needed to implement partitioning in EBS
Infrastructure Capacity Management
• RAC - scales very well… but some caveats!
– OLTP/Warehouse type queries, Global Cache
Service (GCS) contention
– Concurrent gatherstats across nodes can cause lock
ups
– Any contention in a single node environment
magnified in multi-node RAC
Initial Build and Deployment Experience
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
Clustered with 0C4J
EBS 11i Web Cache 10.1.2.3.0
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
11.5.10
Application Server
Production Second Phase :
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
Production Initial Phase :
Our Early and Rapid
Deployment
Clustered with Jserv& Servlet mode
EBS 11i Forms, Reports & Concurrent Managers
11i EBS Database
11i EBS Database - RAC Node 1
11i EBS Database - RAC Node 2
11i EBS Database - RAC Node 3
Full System Configuration:
Processors – 58x 1.65GHz P5 (595)
Memory – 512Gb RAM
Full System Configuration:
Processors – 58x 1.65GHz P5 (595)
Memory – 512Gb RAM
Initial Build and Deployment Experience
Initial Build and Deployment Experience
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
BLADE
(2x 3.6Ghz EM64T
48Gb RAM)
Current Production – Physical Representation
Clustered with 0C4J
EBS 11i Web Cache 10.1.2.3.0
Clustered with Jserv& Servlet mode
EBS 11i Forms, Reports & Concurrent Managers
11i EBS Database - RAC Node 1
11i EBS Database - RAC Node 3
11i EBS Database - RAC Node 2
Full System Configuration:
Full System Configuration:
Processors – 23x 4.14GHz P7 (780) Processors – 21x 4.44GHz P7 (780)
Memory – 512Gb RAM
Memory – 512Gb RAM
Our Build and Design
LBR with
cookie
persistence
Radware Web Accelerator
Oracle Web Cache
Application
Layer Linux
RH5
Oracle Web Cache
Clustered with OC4J
Clustered with Jserv/servlet mode
Forms S1
Report S1
Concurrent Mgr 1
Form S2
PCP
Report S2
Concurrent Mgr 2
Application
Layer Linux
RH5
Primary/Secondary
RAC1
(OLTP)
RAC2
(OLTP)
RAC3
(Read-Mostly)
Database
Layer
AIX 5.3-08
Our Build and Design
• Our design and build phase experiences
– Don’t try to do it all at once
• Implement in multiple steps and give time to bed in changes
and gauge their full impact before moving on
– Follow stream deployment and testing strategy
• Start in your Sandpit/Test environments to work out the
specific requirements of your environment
• Most systems are integrated, CIT and SIT testing is next..
Beware of TNSNAME changes during RAC conversion for
DB Links
Our Build and Design
• Our design and build phase experiences
– Test thoroughly
• Don’t rush testing and thoroughly regression and load test
the new architecture, some issues only come out under load
with custom code
Testing Approach
EBS Backend Load Tests
Cluster Controller
• Swingbench Architecture
Co-ordinator
• Custom class extensions to enable
more control
Node 1
• Junit integration with Swingbench
to allow OA Framework testing
Node 2
Node 3
Workload simulated can include multiple concurrent
transactions like TPC, Concurrent jobs and stored
proceedures
EBS Environment
Testing Approach
BOT
(xN)
BOT
(xN)
BOT
(xN)
BOT
(xN)
Store
WiFi
NextG
Existing
3G
EBS Open Interface
• Infrastructure testing using SwingBench bots deployed across our
varied network topology to calculate latency and load.
• Packeteer was installed at end points to enable granular monitoring of
network utilisation.
Our Build and Design
• Our design and build phase experiences
– First transformation was to move to ASM
11g/CRS11g 10g database
• Requires a patch to solve locking issue when changes
happen in ASM
– Second transformation was to upgrade 10g RDBMS
to 11g RDBMS
• Use best practices, prepare for changes in execution plans,
baseline 10g system and compare, make sure you are fully
patched up on 11g
Metalink document E-Business Suite 11i on RAC (Configuring Database Load balancing & Failover )
ID 294652.1] and Metalink note 244366.1.
Our Build and Design
• Our design and build phase experiences
– Third transformation was to go to RAC with 3 nodes
•
•
•
•
More reliability and less impact if loss of one node
Better ability to distribute workload
Mixed workload OLTP/Reporting
Discoverer and other reporting tools can be segregated to
specific nodes
Metalink document E-Business Suite 11i on RAC (Configuring Database Load balancing & Failover )
ID 294652.1] and Metalink note 244366.1.
Our Build and Design
• Specialise workloads with RAC managed services and EBusiness suite application profile settings (load direction)
– We plan Active/asymmetric service descriptions, 2
nodes per service for HA
– TNSNAMES and JDBC are used for external access
such as reporting tools (Discoverer, Cognos, Portal) to
isolate workloads
Metalink document E-Business Suite 11i on RAC (Configuring Database Load balancing & Failover )
ID 294652.1] and Metalink note 244366.1.
Our Build and Design
• Specialise workloads with RAC managed services and EBusiness suite application profile settings (load direction)
– Resource Manager Consumer Groups and Mapping
are used to manage resource utilisation which are then
mapped to services
– Can go to application level via application profiles, dbc
and session_cookie_name settings which can be
changed to match RAC services
Metalink document E-Business Suite 11i on RAC (Configuring Database Load balancing & Failover )
ID 294652.1] and Metalink note 244366.1.
Preparation and pre-requisites
• The minimum software levels for various EBS modules
can be found on Metalink.
Key Patches
– Minipack 11i.AD.I.5 (233044.1)
– 11GR1 APPS Interoperability Patch (6328363)
– 11i.ATG_PF.H RUP6 – (5903765)
– 11.5.10:SFM Unable to Process Orders in RAC
Config (4022732) ( this can be applied before setting
up Oracle RAC)
Metalink document : Using Oracle Real Application Clusters and Automatic Management with
Oracle e-Business Suite Release 11i and Oracle 11g. Metalink note 455398.1
Minipack 11i AD.1.5, Metalink note 233044.1
Patching and Installation
• Follow best practices for RAC and 11g
– Separate homes for CRS, ASM, DB
– Patch up each component to latest patchset available
on 11g starting with Clusterware
– Check key Metalink documents for known
performance patches/issues with EBS 11i
– Shared file systems for directory objects (output
directories), OCFS2, NFS, IBM GPFS
– RAC memory overhead over single instance needs to
be considered (about 10% more for buffer cache and
15% for shared pool)
Review the two key Metalink notes, 950896.1 and 244040.1 for known performance patches/issues
with EBS 11i
RAC Conversion RDMBS
•
•
•
•
CRS installation on each node
ASM conversion if required
RAC conversion (follow Metalink)
Edit ConvertToRAC.xml
– Use RConfig utility and plan a few dry runs
– NetCA listener reconfiguration for cluster (RDMBS
level)
– UNDO and TEMP tablespaces
•
FLASH area may need adjusting
– Enable AutoConfig on all database nodes
EBS to RAC Conversion, Metalink note 455398.1 and 455398.1
RAC Conversion RDMBS
• Standby
– Archive logs generated from each instance
– Archive logs name change
– RMAN scripts may need to be modified
• Cloning implications
– RAC to RAC cloning, multiple to single node RAC or
multiple to multiple adds complexity and will be
dependant on site
– Rapid Clone, Oracle Application Management Pack,
manual
– IBM SVC Metro Mirror and Flash-Copy, block level
copying (high speed cloning)
Cloning Considerations and implications , refer to details on Metalink note 760637.1
RAC Conversion Mid-Tier changes
• Auto configuration, get ready to run it a few times on all
nodes!
• Implement Load Balancing using context editor by pointing
to correct balance listeners
– Tools OH TWO_TASK (s_tools_twotask)
– iAS OH TWO_TASK (s_weboh_twotask)
– Apps JDBC Connect Alias (s_apps_jdbc_connect_alias)
• Parallel Concurrent Processing configuration
– In multiple application tier only
• Transaction managers
Known Issue, Service Manager Processes Are Not Starting After A Restart - usdsop cannot redirect
standard output, Metalink note 733901.1
OEM configuration
• Oracle Enterprise Manager is an essential part of our
administration, control and support structure
• Moving the instance from non-RAC to RAC had some
impacts
– Need to re-register the instance as RAC
– User Defined Metrics need to be moved from the
instance level to Cluster level, we have a lot of these
and process was manual
– Challenges in doing live performance management with
OEM in RAC context
• need to hone in with SQL scripts looking at session information
combined with various GV$views
• drill down in OEM goes from cluster to instance to session view
which makes it difficult to pinpoint cross instance locking =>
identifying library cache pin locks is a good example
Issues and Resolution
• Initial issues identified in testing addressed with patches
• Unique environment challenges
• Testing stream alignment (UAT/SIT/CIT/SP)
– extra infrastructure required to reproduce
– 1 node RAC can be used for development and
preliminary test environments
– Right sizing test data and environments
– UAT must be sized to an adequate ratio to production
to outline issues in contention and scaling
Next Steps
• Extending EBS Environment Externally
– OLM
– Procure to Pay
– Credit Card (own - closed loop)
• Further Growing Reporting Requirements
– Improve Performance
– Partitioning
• Continue with archiving
Useful Resources
Oracle E-Business Suite Release 11i High Availability Documentation Roadmap – Metalink note 1072807.1
Oracle Applications Release 11i with Oracle 11g Release 1 (11.1.0) –Metalink note 452783.1 - Interoperability
Notes
Using Oracle Real Application Clusters and Automatic Storage Management with Oracle E-Business Suite Release
11i and Oracle Database 11g – Metalink note 455398.1
John Olszewski
[email protected]
THANK YOU