Transcript Change Processes in Garda Diversion

Family Support Agency
Draft Outline of
Framework for Evaluation
of Family Resource Centres
Kieran McKeown
Version V6 – 7th September 2011, CDI Conference in Dublin
Evaluation Questions
1.
2.
The first set of evaluation questions are:

To what extent is the FRC programme achieving its objectives, that is, one or
more of its seven outcomes?

Would these outcomes, for individuals and/or communities, have been
achieved without the programme?

In order to answer these questions, it is necessary to make a before-and-after
comparison of two identical groups: those who participate in the programme
(the ‘treatment group’) and those who do not (the ‘comparison group’).

A more tentative answer could be provided by comparing how FRC areas have
changed relative to national norms for each of outcome
The second set of evaluation questions are:

For any given FRC outcome, what are the factors or ‘pathways’ influencing its
effectiveness, for whom, in which group of FRCs, using what type of
intervention(s), etc?

In order to answer these questions, it is necessary to have a theory about how
the FRC intervention works and to test this against the evidence. This requires
a ‘logic model’ for each programme outcome that is being evaluated.
Draft Logic Model of FRC Programme
Inputs
Interventions with
parents, children
& community:
- Type
- Duration
- Intensity
Uptake and impact of
interventions may
be influenced by
characteristics of:
- Target group
- Target area
- Outcome area
Pathways
Changes in Behaviour &
Attitudes of Parents & Children:
- Individual well-being
- Relationships
- Support networks
- Neighbourhood (trust, safety)
Changes in FRC &
Community Capacity:
- Staff skills, attitudes, practices
- Engagement & ‘empowerment’
of community
Changes in Local & National
Service Capacity:
- Response of other services
- Inter-agency cooperation
- National policy
Outcomes
Improved Child &
Parent Outcomes:
1. Healthy /
healthier
2. Actively learning
3. Safe from harm
4. Economic
security
5. Safe
environment
6. Positive
networks
7. Participating in
society
Comments on Draft Logic Model of FRC Programme

This model is the first draft of a theory on how FRCs work. It is only the
first step and requires further refinement within the ‘learning community’
of the FRC programme.
 It is essential that the evaluation is developed in tandem with the roll-out of
the framework in those FRCs who participate in the evaluation, focusing on
a small number of substantial and sustained interventions.
 This logic model also provides a simple framework for writing up an
illustrative case study of an intervention.

This approach involves the FRC programme preparing itself for evaluation
by specifying exactly what outcomes it is trying to achieve, ensuring that
they are achievable and measurable, deciding on the pathways by which
they are to be achieved, with whom, and over what time period, etc.
 This is already part of the task of preparing a local family support strategy,
but could now be done by FRCs in the knowledge that their local strategy
will be evaluated.
 In this sense, developing the evaluation framework could be seen as an
integral part of the FRC strategy and building FRC capacity.
Further Comments on Draft Logic Model of FRC Programme
 From an evaluation perspective, each ‘box’ in the model represents data to be
collected; from an implementation perspective, each box represents a series of
actions to be considered and delivered.
 ‘Inputs’ are ‘FRC interventions’, both ‘programmed’ and ‘other developmental’
activities. ‘Inputs’ also include indicators of need and the factors that may
influence participation in FRC activities.
 ‘Pathways’, based on other evaluations, could include:

Changes in behaviour and attitudes of individuals, families and
neighbourhoods (based on a range of indicators)

Capacity of FRC and community (based on a range of indicators)

Local and national service capacity (including inter-agency working)
 ‘Outcomes’ are the seven nationally are agreed objectives for children and
families as summarised in the next slide
 The big challenge in the evaluation is to refine this model and find robust ways
of measuring each concept. There are also challenges for implementation.
Outcomes to be Achieved by FRC Programme
 The FRC programme is aligned with national policy on children; in the
absence of an explicit family policy, this is national family policy de facto.
 There are seven national outcomes for children in the Agenda for Children’s
Services (2007), building on the National Children’s Strategy (2000). Every
child should be:
1. Healthy, both physically and mentally
2. Supported in active learning
3.
4.
5.
6.
Safe from accidental and intentional harm
Economically secure
Secure in the immediate and wider physical environment
Part of positive networks of family, friends, neighbours and community
7. Included and participating in society
 The FRC framework has adopted these as outcomes for both children and
their parents.
Measuring Outcomes
 A major challenge for both the FRC strategy and its evaluation is that
there is no agreed method for measuring national outcomes for purposes of
monitoring and evaluation.
 Some outcomes (eg 3, 4, 5, 7) do not seem to lend themselves to
assessing the effectiveness of child and family services such as FRCs.
Outcome
Possible Sources of Comparative Data
1. Healthy
GUI 2007-9; SLAN 2007; HBSC 2006
2. Actively learning
GUI 2007-9; DES; NEWB 2004/5;
3. Safe from harm
HSE or DOH; GUI 2007-9;
4. Economic security
SILC 2009; QNHS
5. Safe environment
GUI 2007-9
6. Positive networks
GUI 2007-9; SLAN 2007; HBSC 2006
7. Participating in society
GUI 2007-9
Next Steps


Further work will involve:

Working with an advisory group of 4 FRCs to identify a set of
‘substantial and sustained interventions’ (2-4) that could be the
subject of evaluation.

Developing the logic model for these interventions.

Finding instruments to measure the concepts, including availability
of comparative data.

Developing a sampling framework and method of data analysis

Case studies could have a useful role within this framework as
illustrations of pathways to outcomes that have already been
established as effective by the evaluation.
These steps will lead to the preparation of a paper for the FSA detailing
the framework for the evaluation.
ENDS HERE
GUI Instruments: Parent (9-yr old cohort)
1.
Socio-economic and demographic data on parents/caregivers,
including social class, education, income, employment status, marital
status, housing type, family type, support networks, friendship
networks, leisure activities, childcare arrangements
2.
DAS Relationship with partner by primary and secondary caregiver
3.
Physical and mental health, including CES-Depression by primary
and secondary caregiver
4.
Neighbourhood problems by primary caregiver
5.
Neighbourhood services by primary caregiver
Note that GUI is weak on the measurement of individual parent well-being
despite the fact that it has substantial influence on child well-being
GUI Instruments: Child (9-yr old cohort)
1.
Child’s EAS [Emotionality, Activity, and Shyness] Temperament by
caregiver
2.
Pianta child-parent relationship by primary and secondary caregiver
3.
Child’s SDQ by teacher and caregiver
4.
Child’s academic performance by teacher and primary caregiver
5.
Drumcondra Reading & Maths competence by child
6.
Attachment to school and teachers by child
7.
Bullying and being bullied by child
8.
Piers-Harris Self-Concept by child
9.
Parenting Style Inventory II by child
10.
Neighbourhood problems by child