Transcript An Integrated Approach to Performance Measurement, Evaluation

An Integrated Approach to Performance
Measurement, Evaluation and Risk
Monitoring of S&T Programs
Presented to:
Presented by:
Tel:
Fax:
Email:
AEA Evaluation Conference
November 2, 2006
George Teather
Performance Management Network Inc.
(613) 236-2320 ext. 231
(613) 236-8644
[email protected]
Presentation Outline
• Discuss Canadian integrated approach to
performance measurement, evaluation
and risk monitoring for funded programs
• Present a specific example of the process,
as it applies to the Stem Cell Network
(SCN)
2
Overview
•
Many S&T programs are funded on a 3-5 year cycle, and have to go back
to central agencies (Treasury Board) for continued funding. Renewal
involves examination of performance over the funding period, often includes
an evaluation
•
Most new programs are required to develop a “Results-based Management
and Accountability Framework” (RMAF) as part of the initial funding
process.
•
An RMAF describes the program’s objective, the design and delivery
strategy, identifies how the performance will be monitored and reported on
annually, and identifies the evaluation questions that will be addressed and
the approach that will be used to conduct the evaluation.
•
A Risk Based Audit Framework (RBAF) is also developed to identify major
risks that may impact on project success, describe how to monitor those
risks and describe strategies for managing and mitigating those risks.
3
RMAF Table of Contents
• Profile – summary description of objectives, design and
delivery, intended outcomes
• Governance – description of organization,
responsibilities for accountability and reporting
• Logic Model
• Performance Measurement and Reporting Strategy –
performance measures, indicators, sources of
information and reporting schedule
• Evaluation Strategy – identification of issues (relevance,
objectives achievement and design and delivery
alternatives / improvements), evaluation methodology,
and schedule
4
Program Management, Measurement
and Evaluation Lifecycle
Design an
Intervention/
Develop
Strategic Plan
Initial
Situation
Identify
a need
Program
delivery
Program or
policy start-up
Direct Outcomes
Baseline
RMAF
RBAF
Risk Assessment
Reconsider,
redesign, expand,
reduce or end
Intermediate Outcomes
Mid-term or
Formative
Evaluation
Final Outcomes
Outcome or
Summative
Evaluation
Ongoing Performance Measurement,
Program Management
Ongoing Risk Monitoring
5
The Stem Cell Network
•
SCN is one of a number of programs under the National Centres of
Excellence initiative funded by Industry Canada, to develop scientific
networks in specific targeted areas to support scientific development and
innovation leading to economic growth and benefits to society
•
Funding of $5.3 million / yr for an initial seven year period
•
SCN managed by a Board of Directors, provide annual reports on
performance to NCE program office
•
RMAF developed to support the information needs of Board of Directors,
NCE and federal government:
– Identify information to support ongoing operational management by the Board of
Directors, and provide the basis for annual performance review and reporting
(accountability)
– Support periodic in-depth examination of the extent to which the program is
achieving its intended outcomes and objectives
6
SCN Program Description - Activities
• Seek proposals, select and fund research linked to clinical,
commercial or policy objectives
• Fund initiatives to attract, develop and retain stem cell researchers
within the Canadian system
• Disseminate and exploit knowledge and technology developed
under SCN
• Develop and support networks and partnerships within Canadian
and international stem cell community and with those applying SCN
knowledge and technology
• Provide strategic and operational program oversight and
management
7
Generic Program Logic Model
Program Objective: high level strategic purpose
Resources
Reach
Results
HOW?
WHO / WHERE?
WHAT do we want? WHY?
activities
outputs
• Program / Service
Delivery
• Program
deliverables
• Client
Management
• Policy guidelines,
regulations
• Policy & Issue
Management
• Communications
- plans
- internal
communications
- promotion
- info transfer
- consultations
- meetings/events
• Financial
Management
• Human Resources
Management
• Asset
Management
users / clients /
co-deliverers /
beneficiaries
• Primary Targets
(clients, ultimate
beneficiaries)
• Co-delivery Agents
• Other Stakeholders
direct
outcomes
intermediate
outcomes
• New knowledge
• Client Service
- addresses needs
- meets / exceeds • Improved
capability
expectations
- service quality
• Improved
decision making
• Behavioral Influence
- awareness
• Target group
- understanding
changes in
- attitude /
behaviour / other
perception
outcomes
- support
ultimate
impacts
• Sector / Industry /
Regional Impact
• Economic/
Environmental/
Societal Impact
• Contribution to
organizational
objective
• Funding
• Service Outputs
8
Spheres of Influence*
State
Your environment of indirect influence
e.g., Industrial sectors, government decision makers, other
communities of interest where you do not
make direct contact
Behavioural Change
Your environment of direct influence
e.g. people and groups in direct contact with
your programs, staff (i.e. clients, target
audience, co-delivery partners
Time
Operational
Your operational
environment
You have direct control
over the behaviours
within this sphere
*reference S. Montague, www.pmn.net
9
Stem Cell Network Results Logic
SCN
Operations
Activities
and Outputs
(control)
Networking and Partnership
Development
HQP / Training
Research
Knowledge and Technology
Exchange and Exploitation
Awareness, cooperation, participation, engagement of Network partners and stakeholders
Immediate
Outcomes
Technology Exploitation
SCN
Key Partners
Intermediate
Outcomes
• Improved PI entrepreneurial
capability
• Protection and transfer of IP
Research / HQP
• Well managed program
• High quality research results
relevant to partners
• Increased research capacity
• Training of HQP
Public Policy / Regulatory /
Framework
• SC research findings contribute
to discussions and decisions
• Public awareness of SC
research findings
(direct influence)
10
Stem Cell Network Results Logic
SCN
Operations
Activities
and Outputs
(control)
Networking and Partnership
Development
Knowledge and Technology
Exchange and Exploitation
HQP / Training
Research
Awareness, cooperation, participation, engagement of Network partners and stakeholders
Immediate
Outcomes
Technology Exploitation
SCN
• Improved PI entrepreneurial
capability
• Protection and transfer of IP
Key Partners
Intermediate
Outcomes
• Well managed program
• High quality research results
relevant to partners
• Increased research capacity
• Training of HQP
Public Policy / Regulatory /
Framework
• SC research findings contribute
to discussions and decisions
• Public awareness of SC
research findings
(direct influence)
SCN Stakeholders
(contributing influence)
Final
Outcomes
Research / HQP
Marketplace / Society
(contributing influence)
•
•
•
•
Development of key enabling technologies
Translation of research into clinical applications and commercial products
Attract, train and retain stem cell researchers and entrepreneurs in Canada
Development of ethical and legal stem cell research framework that has public support
Realizing the Full Impact of Stem Cell Research for Canadians
Societal Impacts
• Improved treatment of illness and disease using stem cell therapies
• Improved health and quality of life for Canadians
Economic Impacts
• Growth in Canadian firms commercializing stem cell research
• New jobs created in Canada outside the network
11
Performance Measures
• Resources
– Annual expenditures, % from other sources (leveraging)
– Allocation among program activity components
• Outputs – knowledge creation
– Number and type of publications and reports generated from SCN
funded projects
• Immediate Outcomes
– Awareness, cooperation, participation and engagement (networking and
collaborations, interactions with other groups)
– Research (achievement of milestones, quality and relevance, training of
new researchers)
– Technology Exploitation (improved entrepreneurial capability, protection
and transfer of IP)
– Public Policy / Regulatory Support (input to public policy, public
awareness of reports)
12
Performance Measures (continued)
•
Intermediate Outcomes
– Development of key technologies (patent applications, patents)
– Transformation of research into clinical applications and commercial
products (licenses, nondisclosure agreements)
– Attract, train and retain stem cell researchers (SCN funded students
graduating and staying in Canada)
– Public supported ethical and legal stem cell research framework
(evidence of contribution)
•
Final Outcomes- measured in evaluation only, not annually
– Economic growth and productivity (profile of firms commercializing SCN
research, effect on existing industry)
– Societal impacts / health and well being (examples of contributions)
•
SCN Management, interactions with other groups)
– Board and stakeholder support for program delivery and performance monitoring
strategy
– Management and delivery of SCN program (use of RMAF, level of participation,
% of total costs for admin, overall success)
– Internal communication (SCN publication frequency, level of participation at
AGM)
13
Evaluation Issues
• Relevance
– contribution to meeting needs of SCN
– relevance to achievement of NCE and federal objectives for innovation
and economic and social benefits
• Objectives Achievement
– Engagement of target communities (researchers, partners and
stakeholders)
– Project level outcomes (technologies, products, processes, networking)
– Program level outcomes (technologies, clinical and commercial
applications, retention of HQP)
– Contribution to final outcomes (improved treatment of disease, health
and quality of life and economic growth of stem cell related firms)
– Examination of unintended outcomes
• Alternatives / Design and Delivery
– Factors facilitating or impeding objectives achievement
– Changes to improve efficiency, effectiveness, level of success
14
Evaluation Methodology
(multiple lines of evidence)
•
Review and analysis of internal and external documents and reports
•
Surveys of researchers, partners and stakeholders (following prepared
questionnaire)
•
In-depth interviews of selected researchers, partners and stakeholders
(in-person or telephone, following open ended interview guide)
•
Case studies of selected projects (to examine relationships, outcomes,
barriers, sector and societal impacts)
•
Cost / benefit analysis
•
Benchmark studies comparing program to other similar programs
•
Integration and analysis of evidence collected on each issue
15
Summary
• RMAF is an integrated program management tool to support
operational and strategic program management and meet the
funding agencies requirements for accountability and outcomes /
impact information
• RMAF identifies evaluation issues at the beginning of the program
well ahead of implementation and communicates evaluation
purpose and approach to both performers and funders
• Critical to have stakeholders participate in development rather than
have it developed by consultants separately and imposed
• In the case of SCN, RMAF is being used by Board of Directors as an
important source to identify program performance information.
Reporting templates have been developed based on the RMAF
16
Bibliography
• Treasury Board Website – RMAF / RBAF
– www.tbs-sct.gc.ca/eval/eval_e.asp
• Stem Cell Network Website
– www.scn.ca
• Performance Management Network
– www.pmn.net
17