Transcript ClaudeJulien

Quality Review of Key
Indicators at Statistics Canada
ICES-III, June 2007
Claude Julien and Don Royce
Outline
Quality Management at Statistics Canada
Quality review process
Recommendations
Observations on the review process
Conclusion
Quality Management at STC
1997 Quality Audit
Quality Assurance Framework (QAF)
Audit on 4 key surveys
Office of the Auditor-General
recommended to examine a wider
application of enhanced self-assessment
Quality Assurance Framework
(QAF)
Quality definition
Fitness-for-use: relevance, accuracy, timeliness,
coherence, accessibility, interpretability
Management of each dimension
Policies, standards, guidelines and practices
Five main systems: users liaison, planning, methods
and standards, dissemination and reporting
(Brackstone 1999, Survey Methodology)
Integrated Program Reports (IPR)
Quadrennial report and biennial follow-up
Quality, costs, management of HR and
strategic direction
Self-assessment and user survey or
consultation
Focus on quality of outputs
Review and feedback from Policy
Committee
Quality Review Process
Initiating the review
Some serious errors in the preceding 18 months
International trade
Productivity estimates
Consumer Price Index
Questions from Finance, other key users and
media
Policy Committee created Steering Committee of
DGs to review quality assurance practices
Headed by DG-Methodology and assisted by head of
Quality Secretariat
Objectives of the review
Identify “best practices” that should be
promoted to other programs
Identify specific weaknesses where action
is needed
Programs Reviewed
Monthly programs
Labour Force Survey
Retail Trade Survey
Manufacturing
International Trade
Consumer Price Index
GDP by industry
Quarterly programs
Income and Expenditure Accounts
Labour Productivity
Balance of Payments
Challenges – Types of programs
Labour Force Survey
Traditional
Retail Trade Survey
Manufacturing
Administrative data
International Trade
Consumer Price Index
Mixture of data
GDP by industry
Income and Expenditure Accounts Derived data
Labour Productivity
Balance of Payments
Mixture of types
Challenges – Program process
Process = complex series of steps
Set objectives, define population, create frame,
select sample, collect data, etc.
Identify a reasonable number of
recognizable steps
Focus on one or two
Challenges – Aspects of a step
Design
Implementation
Execution
Evaluation
Challenges – Dimensions of quality
Relevance
Accuracy
Timeliness
Coherence
Interpretability
Accessibility
Mandate of the review
Review the practices during production
(execution) to prevent the release of
erroneous data (accuracy)
Particular focus on the certification and
data dissemination of data
Review of Dissemination and Communications
Program
Identify the underlying risk factors and their
importance.
Review Teams
Identify and “volunteer” 10 reviewers
10 review teams
3 reviewers + member of Steering
Committee per team
Lead reviewer
Organization
Policy Committee
Mandate
Directors General
Recruitment
Directors
Program
Assistant Directors
Introductory meeting
Program director, lead reviewer and head
of QS
Introduce the review process
Introduce the program
Identify main steps of program process
Main steps of program process
Preparation for certification
Data collection
Editing and transformation
Imputation and estimation
Certification
Data release
Post-release
Gather information
Semi-structured questionnaire
Cover the topics of the mandate
Not to be completed
Interviews with review team and program
director and managers
Available documentation
Meta-data
Program reports
Assessment
Individual reports
Report by lead reviewer
Reviewed by Steering Committee
Vetted by program director
One-day debriefing session with all
reviewers
Summary report
Report by Steering Committee
Presentation to Policy Committee
Taking action
Disposition of recommendations by Policy
Committee
Funding for corrective actions
Follow-up in Integrated Program Reports
Recommendations
Main recommendations
Program managers are well aware of the
risks factors
The programs are well-positioned to
identify the improvements
Some programs would benefit more than
others from additional investment
Human resources concerns dominate all
other risk factors.
Main recommendations
The existence of a research and analysis capacity
separate from the production operation is a key factor in
assuring quality.
There are numerous “best practices” in all programs
that can usefully be shared within the Agency.
Statistics Canada should develop a formal Quality
Incident Response Plan
Proposals to further increase the timeliness of the
programs should be regarded with extreme caution
Statistics Canada should establish an ongoing program
of Quality Assurance Reviews
Observations on the review
process
Aspects to bring forward
Clear mandate from Policy Committee to focus the review
Participation at the DG level to assure legitimacy
Strong cooperation from program directors and managers
Concurrent reviews to assure attention and team work
Knowledge and experience of reviewers
Composition of review teams to assure some
standardization
Excellent learning, development and networking
opportunity for reviewers
Overall positive experience for the reviewers
Collection approach and tools
One-day debriefing session
Aspects to improve
Better assessment and reporting of level of
various risk factors
Timeliness
10 reviews in five months (Oct-Mar)
Reviewers spent around 15 days
More interviews than expected
Time to write, review and translate reports
More timely process and dissemination
More time to schedule meetings around production
More standard program reports
Conclusion
The review was developed and carried out very
quickly
The review was a success thanks to all
participants
The review was not thorough; it complemented
other key quality management mechanisms
Mandate to propose an on-going review program
Build on recent review
Criteria for selecting surveys
Tool box / Flexible approach
Part of learning and development program
Merci
For more information, or
to obtain a French copy
of this presentation,
please contact:
Pour de plus amples
informations ou pour obtenir
une copie en français du
document, veuillez
contacter :
Claude Julien
Email / Courriel: [email protected]
Phone number / Téléphone: 613-951-6937