Transcript PA`s audit value should be compared to the

A statistician’s view on the EU sampling guidelines
Paul van Batenburg
CB Conference, June 1st, 2010, Budapest
© 2010 Deloitte Touche Tohmatsu
A statistician’s view on the EU sampling guidelines
• The situation:
1. PA executes on-the-spot checks on samples from claims. These risk based and
random samples are performed for different support schemes
2. CB investigates whether the PA sample checks can be relied upon by:
a) Performing a MUS sample on the population of items that were randomly selected
by the PA
b) Choosing a sample size using sheet A1 to compare the PA’s audit values with the
CB audit values
c) Evaluating by using sheet A2 confronting the most likely absolute error rate with a
2% materiality threshold
• Questions I would like discuss (in reverse order):
1. Is this the right sample from the right population?
2. Is this the right sample size?
3. Is this the right evaluation?
• If one or more answers turn out to be negative, a new question comes up:
‒ Is there an alternative method to recommend?
• I believe so, and will briefly introduce:
‒ AOQL
‒ Regression estimates
2
© 2010 Deloitte Touche Tohmatsu
Presenter
•Paul van Batenburg
•Statistician, not an auditor
•Deloitte Netherlands
•“Effective and efficient audit by using statistical methods and techniques”
•Relevant assignments: NL Ministry of Economic Affairs, NL Ministry of Agriculture (PA’s and CB),
EU DG Budget, Regione Puglia
•www.steekproeven.eu (goto: CB Conference for Excel sheet and this presentation)
3
© 2010 Deloitte Touche Tohmatsu
Question 3: evaluation
•Do the percentages that make up (j) have the same denominator?
•MUS samples are not equipped to evaluate understatements; taking their absolute values does not
solve that
•The sample was designed for a 95% maximum error to meet 2% materiality, so why confront the most
likely error with this threshold?
4
© 2010 Deloitte Touche Tohmatsu
Question 2: sample size
•The audit risk model was designed to economize on substantive testing in situations where controls
were in place to prevent errors. A low required confidence from substantive testing combined with a high
expected error rate is inconsistent with the rationale behind the ARM
•The form of the ARM used is additive:
total confidence level = confidence level from control testing plus confidence level from substantive
testing
•The AICPA audit risk model (as used by audit firms globally) is multiplicative:
•audit risk = control risk * detection risk, or
total confidence level
= 1-(1-confidence level from control testing)*(1- confidence level from substantive testing)
•I personally believe both models are flawed, but the AICPA model is better since it cannot yield a
required confidence level of 0%, conform ISA 330 par.20 and A 42
•Error rates mentioned are percentages of materiality and not percentages of the population
•Sample sizes should have been rounded up, see www.steekproeven.eu for an Excel sheet
5
© 2010 Deloitte Touche Tohmatsu
Question 1: population and sample (in bold: what I read from
the guidance)
•If the CB’s sample is meant to confirm the accuracy of the PA’s statistics
The sample should be selected from all risk based and random samples by the PA
The sample should be a line item sample
The PA’s audit value should be compared to the CB’s audit value
A materiality level should be chosen instead of the 2% that is meant for financial audits
•If the CB’s sample is meant to confirm with 95% assurance that the error rate in the payments is
below 2%
The sample may be selected from the PA random sample items (not MUS, not the risk
based items)
The CB sample should be MUS
The PA’s recorded value should be compared to the CB’s audit values
The (95% reliable maximum) error rate should be compared to 2% materiality
In my view, 2 approaches are mixed: conclusion and comparison do not match population,
sample selection and materiality definition
6
© 2010 Deloitte Touche Tohmatsu
On understatements
•A MUS sample of 1 from this population of 300 in 3 transactions
based on recorded values gives probabilities of selection of 1/3 for
each item
•If no errors had been made, the 2nd transaction would have had a
smaller probability: MUS samples are an excellent tool to audit on
overstatements as the error increases the chance of being caught
•And the 3rd would have had a larger probability: MUS samples are
not equipped to audit on understatements
•Either use MUS from the difference between a maximum value and
the recorded value (completeness of VAT invoiced) or use a Line Item
Sample
•Taking absolute error amounts yields an incorrect population audit
value: understated amounts are no part of the population so it does
not contain understatement errors
7
© 2010 Deloitte Touche Tohmatsu
Alternative: relying on CB sample
•The horizontal axis shows the PA’s audit values and
the vertical axis shows the CB’s audit values in the
CB’s sample from all PA sample items
•The statistical technique of regression yields a line that
best expresses the relation to predict CB audit values
from PA audit values for all other PA sample items
(Cochran, 1977)
•If only small differences (+ and - !) are found, the
correlation will be high and the CB can rely on the PA’s
sample checks
• When large differences occur, the PA may not have
followed the audit protocol and the CB should consider
rejecting the PA sample
•The decision to accept or reject the PA sample needs
not to be taken for the sample as a whole but can be
applied in batches of, say, 10% of the total PA sample
based on AOQL (Dodge and Romig, 1933, 1957)
•References:
• Vera Raats, Monotone missing data and repeated
controls of fallible auditors, Ph.D. thesis, Tilburg
University, 2004
8
© 2010 Deloitte Touche Tohmatsu
Questions?
9
© 2010 Deloitte Touche Tohmatsu