Transcript Archive or Die!

Archive or Die!
“Information Lifecycle Management”
Mike Ryan
VP Systems Management Products
1
The Critical systems have
1 to
to be available
My Soldiers
turn up for a fight
run the business
22
The are
coreinapplications
They
a condition are
to put up
in
a
usable
condition
a fight when they get to the
battlefield
There are “two” requirements a business
There are “two” demands I press upon my
demands from the IS department
Armies
Do you have a strategy….
“Deploying a successful database archiving strategy
is critical to ensure database availability.”
Its all about performance…
“Organizations should incorporate data archival policies within current
design or implementation plans to maintain future performance and
reduce related costs.”
Does it feel like you run your own disk plant ?
“Most large enterprises are rightly concerned with the implications of
ever-growing database size on disk storage costs, application
performance and administrative overhead.
Take control of growth…
“Through 2007, data growth estimates will continue to tax the practical
limits of database performance, maintenance capabilities, and time
requirements.”
“ Although processing power doubles and disk prices are halved every 18
months, no such scalability model applies to database software.”
Conclusions
•
1
Critical systems have to be available to run the business!
The core applications must be in a usable condition!
Archive or Die!
...Time for Information Lifecycle Management
ILM – A term well used
Information Lifecycle Management
"Information lifecycle management (ILM) is a process for managing
information through its lifecycle, from conception until disposal, in a
manner that optimizes storage and access at the lowest cost."
Source: IBM – Compliance and Data Retention
Information Lifecycle Management
Initial
input
Day to day
Operational
Nearline &
Off-line
End of life
Information
Replication & HA
System
Management
Data
Management
The Birth of JIC Information
Fast/High
JIC – ‘Just In Case’ Information
Accessibility – Speed/Frequency
Typically information becomes JIC
somewhere between 2 and 18
Months.
You need to consider when your
information becomes JIC!
Slow/Low
Source: Cohasset Associates Inc.
10 YEARS
3 YEARS
2 YEARS
2 MONTHS
1 DAY
1 HOUR
How much? How fast?
“With growth rates exceeding 125%,
organizations face two basic options:
continue to grow the infrastructure or
develop processes to separate dormant
data from active data.”
Source: META Group 2003
Current Data
Historical Data
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011
Industry Watchers
LET US BE CLEAR!
"Having a lot of storage is like having children. It can
cost you almost nothing to have a child, but the upkeep
costs a lot. In fact, Aberdeen's research and other
people's research show that there can be an order of
magnitude difference between the cost to acquire the
storage and the cost to maintain it."
Source: iSeries Network – Newswire Daily
Out of context quote by Dan Tanner – Analyst, Aberdeen Group
Database Overload:
• …continue to add
– People
– Transaction
– Processes
– Infrastructure
• …continue to limit
– Performance
– Availability
“How much before it affects your performance”
Alternatives
•
•
•
•
Add more capacity
– Bottom line impact
– Uncontrolled cost spiral
Institute rigorous database tuning
– Does not directly address data growth
– Reaches point of diminishing returns
Delete/ purge data
– Legal and data retention issues
Archive
– Complex undertaking?
– In house
– ERP Vendor
– 3rd Party
The traditional assumption!
75%
Op.
Sys.
Other
Main ERP Application
…there IS another way
Op.
Sys.
Application
Other
Operational
History
75%
Additional
Workload
Additional
DASD
•Linux partition
•AIX partition
•Application Copy
Very Good
Value
…there IS another way
Op.
Sys.
Application
Other
Operational
History
75%
Application
Copy
…existing
or new
iSeries?
Retaining value by adding intelligence!
PURGING - WHY
Business Requirements
Why purge a file or files?
Data removal
Benefits obvious!
1.
Stored data takes up disk space.

The more historical data we keep the more DASD is required.
(Disk upgrade annual event?)
2.
Backup time.

Larger files, longer backup times. (Archives do not need daily
back-up)
3.
Disaster Recovery

More records, longer delays, larger access path builds.
4.
Overnight processing.

More records, more processing time. More overnight window
requirement.
5.
PERFORMANCE.

Unacceptable response times enquiring over large files

The effect on Query or SQL processing is incredible!.
Consider Query Performance
•1/2 Million records in each file
1 Minute
•1 Million records in each file
3 Minutes
•2 Million records in each file
8 Minutes
…..so, why aren’t we all archiving?
Terror!!
Is SOX coming to Europe?
…..June 15th, 2006
• Damien Wild from Accountancy Matters comments on the possible SOX fall-out of a
take-over of the London Stock Exchange by NASDAC. He fears US regulations will
apply to companies listed at the LSE as well, notwithstanding the reassurance of the
chairman of the Financial Services Authority to the contrary.
• With a successful merger of NYSE and EuroNext this development would be
spreading across Europe.
• I think that the picture he paints is very real. US Government has already stepped
halfway across Europe's doorstep with SOX and may not be inclined to stay there.
Even if US regulations do not reach Europe for all listed companies through mergers
of stock exchanges, the pressure is on European governments to create SOX-like
legislation anyway. In the Netherlands, legislation is being passed that increases
governance requirements and power of the Financial Market's watchdog, the AFM.
Why? Amongst others, as a pre-emptive strike to keep US regulations outside the
door. If Europe does not follow its lead, the US government will cross that doorstep
without a doubt.
• So, whether or not US legislation will get a foothold in Europe through LSE and
EuroNext, Europe is already on the path towards US like regulations as we speak.
Hey! Legislation was mentioned…
SOX… Just a sample…
“(ii) The electronic storage media must:
(A) Preserve the records exclusively in a non-rewriteable,
non-erasable format; .”2
2 SEC 17a-4 (f) (2)
It puts a persuasive argument…
“SEC. 802. CRIMINAL PENALTIES FOR ALTERING DOCUMENTS.
(a) IN GENERAL.—Chapter 73 of title 18, United States Code, is amended by
adding at the end the following:
‘‘§ 1519. Destruction, alteration, or falsification of records in Federal
investigations and bankruptcy
‘‘Whoever knowingly alters, destroys, mutilates, conceals, covers up, falsifies, or
makes a false entry in any record, document, or tangible object with the intent to
impede, obstruct, or influence the investigation or proper administration of any
matter within the jurisdiction of any department or agency of the United States or
any case filed under title 11, or in relation to or contemplation of any such matter
or case, shall be fined under this title, imprisoned not more than 20 years, or
both.”1
1 Sarbanes-Oxley Act of 2002
ARCHIVING - WHY
Business Requirements
Why archive a file or files?
1.
Government regulations

There are various requirements that forces us to keep
records for many years.
2.
Legal requirements

We may need to retrieve historical data for legal
purposes.
3.
Issue resolution

We may need to retrieve historical data to resolve an
old bookkeeping or customer issue.
Definition of an Archive
“The removal of historical data to safe and separate storage that can be made available to
the users”
1 Million objects where do we begin?
Finding redundancy can be like a needle in a haystack
Where can we start?
What
next?
Data!
iSeries AS/400
Where can we start?
What
next?
Data!
iSeries AS/400
Where can we start?
G/L Transactions S/L Transactions
Test Databases
back to 1995
back to 1996
Inventory Movements
back to 1997
Order:Headers
Lines
Details
Items
What have
we got ?
Vendors
Customers
Just what do we need?
ERP Database
Typical database prior to
any archiving
Current Year
Last Year
Year 3
Year 4
Year 5
Year 6
Year 7+
Operational Data:
This is dynamic data that is
processed daily. The data needs to be
available 24/7 and it needs daily
backups.
Reference Data:
This is static data that is seldom used,
(Mostly for statistical/reference
purposes) It has no 24/7 requirements
and it does not need daily backups.
Historical Data:
This is static data that is practically
never used unless for audit purposes.
Operational DB
Current Year
Typical after
Advanced Archiving
Last Year
On Line Data
Reference DB
Year 3
Year 4
Operational and Referece Data is
stored on-line but in two separate
databases. Thereby the management
of the Operational data is optimized
and the Reference data is still
accessible
Historical DB
Year 5
Year 6
Year 7+
Off Line Data
Data that is looked upon very
seldom is best stored off-line
on tape media, DR550,NAS
etc.
The Solution - Automation
Production Database
• Holds data for 2 years
Movex Database
Current Year
Last Year
Archive Database
Periodically
Year 3
Periodically
Year 4
Online Archive Data
• Holds a further 4 years
Year 5
Year 6
Fully Automated
Or/And
Offline
Archive Data
DR 550
Archive Models
Decision Files
Order
Header
Orphan Control
Customer
Master
Not
dispatched
Order
Line
Tag Files
Archive Criteria
Order
Line
Dispatch
Notes
Delivery
Instructions
Eat an elephant all in one go?
G/L Transactions S/L Transactions
Test Databases
back to 1995
back to 1996
Inventory Movements
back to 1997
Order:Headers
Lines
Details
Items
Customers
Vendors
Summary…
Make the decision to Archive – Register a project
Look at what is available – Base function in ERP S/W
Be tough - Some data is not important - Purge it!
Test databases are a good start
On-line but not day-to-day operational is still good
Not every file needs to be archived
Keep good records of data movement (Audit)
Objects can be archived, not just files
There are generic tried, tested and operational solutions.
An investment with a lasting legacy…
–
Improve Performance of Query, SQL and many
enquiry programs
–
Reclaim Disk space
–
Still be able to enquire on Reference data
• Archive data to On-line DASD
• …and/or Off-line media
• Use standard enquiries (on-line)
• Track/audit all Data Movement (SOX)
–
Clean up after yourself!
(deleted record space recovery)
–
Once set up, can be automated and
scheduled
Business Benefits
Save valuable operator time
Improve interactive performance
Improve batch performance
Improve HA performance
Reduce tape backup window
Reduce disk utilization
Archive or Die!
“Information Lifecycle Management”
Mike Ryan
VP Systems Management Products