Transcript 11. How to evaluate social inclusion measures - Laurie

•CIVITAS PLUS
Evaluation of mobility and social inclusion
Dr Laurie Pickup
Regional Director for Europe
WYG Environment, Planning and transport
•Venue : Coimbra, Portugal
Date: 22nd/23° March 2012
The scope of this session
1.
2.
3.
4.
5.
The nature of mobility and social
disadvantage – defining the scope of our
evaluation?
Setting out your evaluation - the baseline
expectations, objectives, target groups,
success criteria
Understanding the mobility behaviour of
different social groups and attitudes – what
are the key variables to evaluate?
Measuring and analysing mobility and
inclusion situations and response to change
- Qualitative assessment - Quantitative
surveys
Conclusions
Coimbra, 23/24 March, 2012
Part 1
The nature of mobility and
social disadvantage –
defining the scope of our
evaluation?
Coimbra, 23/24 March, 2012
Mobility is Freedom
One of the strongest freedoms we possess
Coimbra, 23/24 March, 2012
Mobility is not just a transport phenomenon
Spatial/temporal mobility - horizontal
Social mobility - vertical
(Virtual mobility)
Coimbra, 23/24 March, 2012
Barriers to mobility 1/2

Ability - Physical access to transport
facilities and services in terms of vehicle
and system design, waiting and interchange
facilities and information services;

Opportunity - Spatial and temporal
distribution of transport services and their
connectivity with key activities;

Affordability - cost of transport services
and its relation to incomes;
Coimbra, 23/24 March, 2012
Barriers to mobility 2/2

Cognitive and psychological factors such as ability of users to understand
and navigate the system, perceptions of
personal safety and fear of crime;

Cultural and social factors - regarding
acceptability of the services which are
provided, including safety, comfort but
also attitudes and social norms.
Coimbra, 23/24 March, 2012
Social Exclusion: What are we actually measuring?
The concept of ‘social
exclusion’ is contested, and
has multiple meanings.
These meanings are being
continually redefined over
time and have different
policy implications.
(World Health Organisation review 2008)
Coimbra, 23/24 March, 2012
Social exclusion is a myth?
 ‘’Exclusion” is not a concept rooted in the
social sciences, but an empty box given by the
French state to the social sciences in the late
1980s as a subject to study… The empty box
has since been filled with a huge number of
pages, treatises and pictures, varying degrees
academic, popular, original and valuable’.
(Murard, 2002)
 ‘The excluded made up one tenth of the French
population: the mentally and the physically
handicapped, suicidal people, aged invalids,
abused children, drug addicts, delinquents,
single parents, multi-problem households,
marginal, asocial persons, and other social
misfits’ (Lenoir)
Coimbra, 23/24 March, 2012
Definition of social exclusion
‘A shorthand for what can happen
when people or areas suffer from a
combination of linked problems
such as unemployment, poor skills,
low incomes, poor housing, high
crime environments, bad health and
family breakdown’(UK, Social
Exclusion Unit, 1997)
So what are we evaluating?
Coimbra, 23/24 March, 2012
Parameters of mobility disadvantage



Frailty




Low income
Dependency (young and very old)
Disability (physical, sensory, learning
capacity)
Gender role
Ethnicity
In most cases of exclusion, these aspects
arise in combination
Coimbra, 23/24 March, 2012
Table of mobility disadvantage
Children
Personal
Need to be
escorted or
driven
Geographic
Greater
Urban/
exposure to
Peripheral/ Rural traffic in urban
areas
Coimbra, 23/24 March, 2012
Young people
Semiindependent
but with some
restrictions
More excluded
from activities
in rural areas
Older people
Disabled people
Very mixed
picture in terms
of abilities.
Can be frail and
confused in
later life
Problem of
retirees moving
to remote
locations.
Longer term
resident of
social housing
estates.
Can be frail and
may experience
Mobility and
cognitive
difficulties
More likely to
be located in
urban centres.
Part 2
Setting out your
evaluation - the baseline
expectations, objectives,
target groups, success
criteria
Coimbra, 23/24 March, 2012
What is the focus of your evaluation?

A specific initiative dedicated to
addressing mobility and
exclusion/inclusion issues (e.g. Mobility
and poverty in Liverpool)

An initiative primarily with a sustainable
mobility focus – but which may have
social impacts (e.g Mobility policy in
Perugia)

An initiative where the social dimension of
the issue is to be evaluated, but not
specifically for its inclusion components
(Most CIVITAS CITIES)
Coimbra, 23/24 March, 2012
LIVERPOOL – October 1986
Study of impact of bus
privatisation on the poor
Bus fare increase of 300% on
privatisation
The start of ‘Community links’
Followed by ‘Travel safe’
Coimbra, 23/24 March, 2012
Birmingham - 2012
Coimbra, 23/24 March, 2012
Evaluation life-cycle
What stage of the process are you
addressing?
Identification of needs and
requirements of marginalised groups
Identification of behavioural
responses to a change in provision
for marginalised groups
Both
Coimbra, 23/24 March, 2012
An example evaluation life cycle









Systematically assess the accessibility needs;
Generate accessibility strategies that are fully inclusive
to all social groups, based on both state of the art
experience and local needs;
Generate individual measurement options that integrate
to meet the strategy objectives;
Conduct pre-implementation assessment of options on
the benefits to different social groups, including
economic assessment;
Generate a chosen strategy;
Implementation guidelines;
Scheme monitoring and quality evaluation (social and
economic evaluation, but also mobility efficiency and
sustainability criteria);
Feedback and strategy revision, input to standards and
protocols; and
Reporting and intelligence sharing.
Coimbra, 23/24 March, 2012
Defining your target group(s)
Mobility exclusion among:
- specific age groups or gender e.g.
Elderly/frail
- income groups e.g poverty definitions
- specific personal mobility disadvantaged
groups e.g sensory impaired people
- poor access to specific activities (e.g
employment)
- overall mobility in specific types of
regions (city centres, peripheral suburbs,
rural areas)
Coimbra, 23/24 March, 2012
Taking a cross sector approach
 Social exclusion is a cross-sector issue
about restricted life-style
 The evaluation has to include this
perspective
 In 1998, the Government’s Social Exclusion
Unit found that mobility services in the UK
covered the decisions of 11 ministries.
 Assess the link between mobility policy and
social policy (including employment etc.)
 Work with other sectors in the evaluation
 The best solution may not be a transport
one
Coimbra, 23/24 March, 2012
What indicators will you measure?
Hard indicators to measure needs
and/or impacts (quantitative)
Soft indicators to measure needs
and/or impacts (qualitative)
Only mobility indicators?
Indicators of social inclusion?
What level of aggregation are you
addressing?
What indicators will define success?
Coimbra, 23/24 March, 2012
Indicators (1)
Mobility indicators
Use the wide interpretation of accessibility
Accessibility models of varying complexity
Reviews of accessibility indicators
- increased number of activity
opportunities (e.g. Jobs)
- increased number of public transport
services (routes, frequency)
- increased patronage of public transport
services by the target group
- increased satisfaction scores with
access to different facilities
Coimbra, 23/24 March, 2012
Inclusion indicators
National Action Plans
11 primary indicators, 3 secondary
indicators and 11 context indicators;
these are known as the streamlined
indicators
Examples
 Primary - Early school leavers not in
education or training, total
 Secondary - Persons with low educational
attainment Aged 25-64
 Context - Self-reported limitations in daily
activities by income quintiles (Eurostat)
Coimbra, 23/24 March, 2012
A simple approach to planning your evaluation
4 dimensions:
 Target group(s) – e.g. Young unemployed
 Type of measure – e.g.
- Policy: Achieving equal access to
employment
- Strategy: Improve access to job
interviews
- Method: Ticket subsidy
 Type of disadvantage - e.g. Low income,
restricted residential mobility
 Type of mobility – e.g. public transport
modes
Coimbra, 23/24 March, 2012
Part 3
Understanding mobility
behaviour and
attitudes as part of
lifestyle – what are the
key variables?
Coimbra, 23/24 March, 2012
“To change travel
behaviour, you first
have to fully understand
what motivates it”
Coimbra, 23/24 March, 2012
Transport Planning
Follow-the-leader model / optimal velocity model : Newell
dv (t ) ve ' (d (t ))  v (t )

dt

 ; relaxation time,
d (t )  x (t )  x 1 (t ) ; headway
ve ' (d )  (v0 / 2)tanh( d  dc )  tanh dc  ; Optimal velocity function
Intelligent driver model : Treiber et al. (PRE 2000)
  v   s* (v , v ) 
dv (t )
 a 1   0       
dt
s
  v  


Coimbra, 23/24 March, 2012
2



relative velocity
Time and Cost
Cost to driver made up of time, distance and
other costs (such as tolls)
Cost = (A x t) + (60 x B x d) + (C x p) where
A = time coefficients
t = travel time in minutes
B = distance coefficient in minutes per mile
d = link length in miles
C = toll coefficient in minutes per
monetary cost
p = price of the toll in monetary cost units
Coimbra, 23/24 March, 2012
Disaggregated Models

Could you put a value on a person’s time?

Are travel decisions simply a matter of trading
time and cost?
disaggregating
models even further
the travel decisions of
into more varied types
people were in fact
of household, and
much more complex
personal/household
characteristics
Coimbra, 23/24 March, 2012
Disadvantaged Groups
 growing mobility problems of the poor,
of women in traditional households, of
the disabled, of children and of the old
and frail
 Policies began to greatly emphasize the
need to encourage public transport use
 car pooling schemes emerged
Coimbra, 23/24 March, 2012
Stated Preferences- Imposed choices?
Coimbra, 23/24 March, 2012
Time-geographic/activity approaches
• the chances to travel are dictated
by the periods of time and the
spatial opportunities available
• Emphasis on ‘accessibility’ rather
than ‘mobility’
• policies should be targeted at
travel reduction and access
improvement
Coimbra, 23/24 March, 2012
Life-cycle and habit
•
travel patterns showed similarities within
household types at different life-cycle stages
• ‘Life shocks’ experience
•
Short term travel behaviour is dominated by
habit
•
the actual numbers of trips have not
changed greatly over the years
•
the distances people travel changed
Coimbra, 23/24 March, 2012
Realistic Choices
•
one third of households’ trips could
not be changed for various reasons
•
‘realistic choice theory’
•
‘individualised marketing’
•
‘Personalised Travel Planning’
Coimbra, 23/24 March, 2012
Evaluating travel behaviour
•
people make travel decisions based
on time and cost
•
travel behaviour is influenced by
habitual behaviour
•
providing information and support to
produce the changes in behaviour
•
expenditure on travel remains stable
except in lower income households
•
More focus on mobility mind-sets
Coimbra, 23/24 March, 2012
Part 4
Measuring impacts and
understanding why
they occurred: impact
and process
evaluation
Coimbra, 23/24 March, 2012
Measurement methods
 Qualitative research
 Quantitative research
- Non-statistical surveys
- Statistical surveys of samples to predict
population parameters
- Sample selection (random/non-random)
- Survey type
- Statistical error and sample disaggregation
 Analysis
- Single variable, bi-variate, multi-variate
Coimbra, 23/24 March, 2012
Qualitative evaluation methods (1)
 To understand the types of issues arising
and the nature of processes and change
 Good for social exclusion/inclusion
evaluation
 Small samples, semi-structured using
topic guides, recording, skilled moderator
 Focus groups
 One to one interviews
 Use of mentoring intermediaries for ‘hard
to reach groups’
Coimbra, 23/24 March, 2012
Qualitative research (2)
 Qualitative analysis
 Record and transcribe the interviews word
for word (ensure good microphone
positioning)
 Unless intermediaries are necessary, try to
interview yourself
 Analyse ranges and avoid numbers.
 Quote ranges of opinions and arguments
 Use the results for design of quantitative
survey
Coimbra, 23/24 March, 2012
Survey objectives (1)
•
•
•
•
Identify features of (and changes to) behaviour
and attitudes - in the ‘population’ - in a statistical
sense – through measuring the ‘variables’ we
define.
Qualitative research and quantitative surveys
Identify through variable measurement, the
factors that shape attitudes – how this relates to
behaviour patterns
Importance – satisfaction – expectations –
prejudice
- measurement of the ‘scale’ of the variable
- measurement of the relative imoportance or
‘ranking’ between variables
Southampton (UK), 20/03/2009
Coimbra, 23/24 March, 2012
Survey approaches
•
•
Randomly sample ‘a defined population’
•
•
Use of ‘experimental and control samples’
•
•
Survey the sample in the ‘change’ area and in
potential ‘transfer’ sites to infer results
Single random sample surveys at a point in time
(retrospective – e.g short term after survey)
Repeat surveys of random samples from the same
population (Before and after, periodic monitoring)
Repeat interviews with the same sample subjects
(Panels)
Coimbra, 23/24 March, 2012
Statistical error – designing your sample size
Percentile error of sample
percentages
x1 y1 x2  y 2
 1.96

1899 2000
Coimbra, 23/24 March, 2012
Ex-post car Driver
Survey
Male
Sample 2000
18-29
30-39
40-49
50-60
over 60
0,4735
M
0,5265
F
Female
18-29
30-39
40-49
50-60
over 60
Survey methods
 Household surveys
 Doorstep surveys
 In-street/at stop/roadside/on vehicle surveys
 Hall tests
 Postal surveys
 Telephone surveys
 Internet surveys
Coimbra, 23/24 March, 2012
Household surveys (1)
•
Pre-arranged interviews (initial screening)
•
All household members? (Use of incentives)
•
Interviewer administered – 30 to 40 minutes
•
Fully structured questionnaire? (show cards)
•
Possible use of travel/activity diaries
•
Piloting essential
•
Interviewer training and monitoring important
•
Random sampling – assess non-response bias
•
Expensive
Coimbra, 23/24 March, 2012
Household surveys (2)
•
Achieve a full picture of household mobility
•
A picture of trip generation and distribution by
mode/ time of day/ day of week etc.
•
The attitudes of household members to their
mobility issues
•
The primary variables of the mode choices of
household members
Coimbra, 23/24 March, 2012
Doorstep surveys
•
Shorter interviews – maximum 10 minutes
•
Screening for target group
•
Only one person (spokesperson) from the
household
•
Diaries are sometimes left for collection/posting
•
Careful piloting and interviewer training
•
Structured questionnaires (show cards)
•
Random address/walk /random person (repeat
visits – sampling with/without replacement)
•
Assess pattern of non-response
Coimbra, 23/24 March, 2012
In-street/at stop/roadside/in vehicle surveys
•
Random sampling at chosen/random sampling
points/ passing vehicles/ on PT vehicles (specify the
population to be sampled)
•
Possible screening questions for target groups
•
Interview individuals/maximum 4/5 minutes
•
10 key questions – design well to avoid
uncompleted forms
•
Careful piloting and interviewer training
•
Structured questionnaires (show cards)
•
Common use of automatic data recorders
•
Assess pattern of non-response
•
Less cost
Coimbra, 23/24 March, 2012
Hall tests
•
•
•
•
•
•
•
•
Random sampling at chosen sampling points
(specify the population to be sampled)
Usual in-street screening questions for certain
target groups
Respondents invited into a ‘hall’ to complete a
more detailed questionnaire.
Halls provide the opportunity to show plans,
demonstrations, stated preference exercises etc.
Careful piloting and interviewer training
Structured questionnaires (show cards)
Possible use of automatic data recorders or
computers
Assess pattern of refusals to attend the hall test
Coimbra, 23/24 March, 2012
Telephone surveys
•
•
•
•
•
•
•
•
•
•
Specialist Computer Assisted Telephone
Interviewing sysems (CATI)
Structured questionnaire (piloted)
Screening questions for certain target groups?
Telephone panel surveys
Low respondent tolerance with telephone
interviewing
Only useful for short interviews/well defined issue
Possible bias in sampling
Used as an initial screening stage for household
interviews
Used for ‘follow-up’ interviews
Less cost
Coimbra, 23/24 March, 2012
Postal/self-completion surveys
•
Lower cost
•
Structured, short, simple questionnaire
•
Large differences in response rates across
Europe
•
Bias in sampling (check non-response by postal
area?)
•
Generally not reliable for mobility information
Coimbra, 23/24 March, 2012
Internet surveys
•
Lower cost
•
Structured questionnaire
•
Possibility for obtaining qualitative
information – chats
•
More tolerance than for telephone
surveys
•
Self-selecting sample or pre-selection
Coimbra, 23/24 March, 2012
Target Groups (1) – defining variables
•
Mobility categories
- car ownership/licence holding/access to use
(driver and passenger)
- public transport (service level available/ level of
use)
- cycle ownership and use
- walk trip frequency/ length etc.
- mobility impairment (physical, sensory, learning
capacity)
- realistic mode choice groups etc.
- perception of balance of mode use
- ‘I consider myself mainly as.....’
Coimbra, 23/24 March, 2012
Target Groups (2)
•
Socio-economic categories
- Age and gender
- Occupational group
- Individual/household income
- Educational attainment
- Ethnic origin
- Household structure
Coimbra, 23/24 March, 2012
Target Groups (3)
•
Traveller categories
- Commuters
- Business travellers
- Tourists
- Students
•
Personality
- Use of psychological variables
- Control theory
- Personal construct theory
Coimbra, 23/24 March, 2012
What questions (1)?
Questionnaires
• Printed (administered
self-completion)
• Data recorders
• Topic guides
Types of questions
• Closed questn.
• Open questns.
• Single responses
• Choices – SP
Quality/data protection standards
Coimbra, 23/24 March, 2012
What questions (2)?
•
Ranking
- orders
- groups
- allocation of points
•
Scales
- size of the scale
- middle scale?
•
Defining frequency categories
•
Multiple response questions
•
Do not know/DNA
Coimbra, 23/24 March, 2012
The European Dimension
•
Common definition of behavioural
indicators
•
Common survey designs
•
Common types of surveys
•
Common definitions of target groups
•
Minimum standard errors for analysis and
reporting (95% confidence intervals)
Southampton (UK), 20/03/2009
Coimbra, 23/24 March, 2012
The scope of this session
1. The nature of mobility and social
disadvantage – defining the scope of our
evaluation?
2. Setting out your evaluation - the baseline
expectations, objectives, target groups,
success criteria
3. Understanding the mobility behaviour of
different social groups and attitudes – what
are the key variables to evaluate?
4. Measuring and analysing mobility and
inclusion situations and response to change
- Qualitative assessment - Quantitative
surveys
Coimbra, 23/24 March, 2012
Thank you
[email protected]
Coimbra, 23/24 March, 2012