Transcript Using Periodic Audits to Prevent Catastrophic Project Failure

Using Periodic Audits to Prevent
Catastrophic Project Failure
Dr. Paul Dorsey
Dulcian, Inc.
May 22, 2008
1 of 30
The Vasa
17th century (1628)
 “The greatest ship of all time”
 Early
3
years to build
 2 gun decks with 64 cannons
 King
Gustavus Adolphus of Sweden set the
measurements.
 1000
trees were required
 Triple laminated oak walls (18in/46cm thick)
 Mast = 190 ft/57m
 Length = 201 ft/61m
 Cost
= 5% of Sweden’s GNP
2 of 30
Maiden Voyage
 Set
sail on a beautiful summer day
 August
10, 1628
 Passed the royal palace of Stockholm
 Fired proud salutes from cannons
 Sailed 1400 yards
 Gust of wind came
 Ship sank in less than 1 minute
3 of 30
Why is the Vasa relevant?
 What
sank the Vasa sinks FMIS projects.
 We are still building Vasas.
 We can’t stop bad decisions.
 We
 If
CAN stop ignoring them.
you spend $1M and fail, it is bad.
 If
you spend $100M and fail, it impacts the whole
organization

 If
If you spend $1B and fail, it impacts the country.
you are going to fail - fail cheap.
4 of 30
The Essence of Project
Management
 Leading
kids on a hike
 From time to time, climb a tree
 Check for obstacles
 Adjust direction
 Call everyone together
 "LET'S GO!"
5 of 30
Periodic Audits
 Critical
success factor for failure prevention
 Must be external
 Developers
 Big,
substantial effort
 Weak

cannot self-assess.
audit is worse than useless
Provides illusion of safety
6 of 30
Audit Costs
 Delay
project
 Expensive
 5%-10%
of project cost
 Intrusive
 Annoying
 Politically
costly
7 of 30
Team response to audit
 Project
Manager
 "Why
don't you trust me?"
 "Waste of time"
 Developers
 "We
 Team
would rather be coding."
may feel threatened…
 If
the team feels threatened, they probably have
reason to be.
 If
the team welcomes the audit….
 Sign
of professional maturity
8 of 30
Audit Benefits
 Early
detection of failure
 If
a $300 million project fails after $20 million, that
is a huge savings.
 Validation
of system architecture
 Second set of eyes
 Give team time to take stock
 Mid-project course correction
9 of 30
Auditor Independence

Auditor must be told that there
will be no chance for followon work.



During the audit:


Otherwise the audit is suspect

To enforce independence:



No economic attachment to the
results
No incentive to skew the results
No relationship to the
development team
Limit contact with development
team

"Stockholm Syndrome"
After the audit, auditors can
help with the project plan.
Auditor is the agent of the
person who hired him/her (no
one else)



Only reports to the contract
owner are objective.
No professional standard or
certification for IT auditors
exists.
Contract creates objectivity.
10 of 30
Audits are never 100% objective
 Auditors
bring their own biases.
 There are IT "religions."
 .Net
vs. Java
 "Thick database" vs. middle tier logic
 Service Oriented Architecture (SOA)
 Repository-based development
 Business rules
 Agile
 A professional
with a different perspective can
still detect a "Vasa."
11 of 30
Justifying Audit Cost
 An
extensive audit requires approximately 10% of the
total project cost.
 Industry project failure rate is ~ 60%.
 Example: Audit at the halfway point of a $1 million
project


Cost: (50% x 1,000,000) x 10% = $50,000
Benefit: (50%) x 1,000,00) x 60% = $300,000
 Given
the high failure rate, audits are very cheap
insurance.
 With other benefits, it is obvious that audits are a good
deal.
12 of 30
Finding the right auditor
 Not
internal
 Not from the same company as the developers
 Not dedicated auditors
 Must
be real developers, DBAs, architects
 Must have built systems of similar scope and subject
area
13 of 30
The Audit Team
 Project
Manager
 Experience
in the subject area with projects of similar
scope
 Database Administrator
 Experience
(DBA)
with same platform and similar database
size
 Application Architect
 Skill
in the same area (Java, .Net, Oracle, etc.)
 Security
Specialist
 Military,
financial system or health care experience
14 of 30
Structure of the Audit
 Knowledge



transfer
Deep understanding
As if auditor were taking over the project
Understand the system before evaluating
 Infrastructure


Evaluate the existing infrastructure to support system
Every area needs to be assessed.
 Ability

audit
to meet current and future user requirements
Auditor must understand the requirements
 Financial
review
15 of 30
Detailed Structure of Audit

A. Knowledge Transfer




Allows auditors to understand the entire
system architecture as if they were
taking over system development.
The following areas should be reviewed
for the knowledge transfer portion of
the audit:

















System Overview
Data Model Walkthrough
Review/Identify Transaction Use Cases
Review User Interface Code
Architecture
Review Reporting
Requirements/Architecture
Review System Architecture
Review System Installation/Upgrade
Mechanism.
Internal Control review
User Access review
Handling system bugs/enhancements
Multi-Lingual capabilities
Basic System Requirements
Process Flows
Custom Framework
Performance
Standards
Training
B. Infrastructure Audit

Examine from technical soundness perspective.
Compare to current industry best practices;
document any discrepancies.










C. Ability to Meet Current/Future
Requirements

Examine the current system requirements, identify
use cases, and review for suitability, specifically:





System and User Documentation
Data Model Audit
Database Review
User Interface (UI) Architecture Review
Distributed System/ETL Audit
Security Audit
Hardware/Software/Networking Review
Backup/Recovery Procedures
Appropriateness of system upgrade mechanism
Compare documented requirements to the existing
use cases and how they are handled.
Assess user satisfaction with the existing system.
Are existing backup/recovery procedures sufficient to
meet maximum downtime requirements?
Assess system flexibility to meet new requirements.
D. Financial review

Review how resources have been spent over the
lifetime of the project including budgeted vs. actual
expenditures
16 of 30
Knowledge Transfer
 "Seek
first to understand." Stephen Covey
 Learn enough to take over the process:








Architecture
Data Model
Use Cases
Report Audit
Configuration Management
Internal Controls
Documentation
Training
 System


may be bad enough that this is not possible.
Do it anyway.
Preventing the Vasa from sailing is hard work.
17 of 30
Infrastructure Audit


Compare what was learned in
the knowledge transfer with
best practices
Each area needs to be
assessed:








Documentation
Data model
Database design
User interface architecture
Security
Backup & Recovery
Configuration Management
Internal Controls
 Identify
weaknesses in
each area:



Corrective actions
Exposure
Controls needed
 One
mistake can sink the
Vasa.



System won’t scale
Security hole
Inflexible design
18 of 30
Ability to Meet Requirements
 Requires
careful use case documentation
 Assess user satisfaction



Sit with users
Survey
Request queue is not a good measure. If system is poor, users
give up.
 Assess



Functional requirements
Performance
Downtime
 Future


each use case
requirements
Flexibility
Deployment
19 of 30
Financial Review
 Stewardship
 Financial
History
Audit




If requirements
change, price
can balloon.
Does this make
sense?
Sunk costs are
"somewhat"
relevant
Measure
decision quality
Date Budget
$ Spent
MileNotes
stones/
Achieved
20 of 30
COTS Project Audits
 Not
very different from custom projects
 COTS projects fail just as often.
 Review COTS architecture
 Be careful about how well COTS satisfies
requirements:
 COTS
customizations can be VERY expensive.
 Customized COTS cannot be upgraded.
21 of 30
Audit Report
 2-5
page Executive Summary Report
 Are
 10-15
 Are
 100
we OK?
page CIO Report
we OK? Why?
page detailed report
 What
we did
 What we found
 What needs to be fixed
 Next steps
22 of 30
Acting on the Audit Report
 If
report concludes "Vasa…."
 Get
a second opinion
 Let the development team respond
 Sunk
costs are sunk costs.
 The
amount of money budgeted for the project is
irrelevant.
 "Upgrade"
is a way to change direction without
admitting failure.
23 of 30
Case Study: Ethiopia FMIS
 Harvard
University Project
 Small part of major financial reform
 $38
M USD over 12 years
 $3 M USD over 3 years for IT (very frugal)
 Harvard
was ending the project and wanted to
assess quality of system.
 Custom development project
 Dulcian was brought in to do the audit.
24 of 30
Audit Scope
 Standard
 As
 Total
audit
described in this presentation
knowledge transfer and team back-up
 Support
for any “what if?” scenarios
 Total system back-up
25 of 30
Result of Audit
 Quite



a good design
Excellent documentation
Very good developer coding
Supported current requirements
 Some

Database design issues





architecture portions needed modification.
No Foreign Keys
Odd design (inherited from previous team)
Poor performance for parts of system
Scalability questions
Would not work on Ethiopian internet due to slow/unreliable
connections
26 of 30
Audit Recommendations
 Maintain
current system
 Upgrade system internals
 Business
rules approach
 Oracle DBMS
 Ultra-light Web architecture
27 of 30
Audit Results
 Government
and Harvard accepted
recommendations.
 Maintain/evolve
current system $1.5M USD
 Redesign architecture $1.5M USD
 Dual
nature of the audit (audit + handoff) made
existing team very uncomfortable.
 Top
three IT people resigned.
28 of 30
Conclusions
 Audits
don't prevent failure; they just catch failures
earlier in the process.
 Audits don't replace good design.

The audit may only help fail more small projects rather than
one big project.
 Audits
are resource intensive, expensive, annoying,
politically charged.

But they are cheaper than not doing them at all in the long
run.
 Auditors

must be kept independent.
No follow-on work
 Both
COTS and custom projects need audits.
29 of 30
Contact Information
Paul Dorsey – [email protected]
 Dulcian website - www.dulcian.com
 Dr.
Design Using UML
Object Modeling
Developer Advanced
Forms & Reports
Designer
Handbook
Latest book:
Oracle PL/SQL for Dummies
30 of 30