Transcript 1. Interviewer - ORB - University of Essex

Questionnaire & Interview
Item-writing Part 1
Dr Desmond Thomas, MA TESOL
University of Essex
Useful References
• Denscombe, M., 1998, The Good Research
Guide for Small-scale Social Research
Projects, Open U. Press
• Foddy, W. 1993, Constructing Questions for
Interviews and Questionnaires - Theory and
Practice in Social Research, Cambridge:
Cambridge University Press
• Oppenheim, A.N., 1992, Questionnaire
Design, Interviewing and Attitude
Measurement, London: Pinter
Questionnaire or interview?
What’s the difference?
• Not a great deal, when questionnaire items
are read out and responses filled in by the
researcher
• But a world of difference between an
anonymous structured survey questionnaire
and an unstructured 1-to-1 interview
• QUESTIONNAIRE OR INTERVIEW: WHICH
SUITS YOUR RESEARCH?
• And what kind of interview?
A research questionnaire should …
• Be designed to collect information which can
be used as data for analysis: this can be
quantitative, qualitative or a mix
• Gather information by asking people about
issues directly related to a research project
(demonstrating validity)
• Consist of a written list of questions with
each respondent reading an identical set and
following identical procedures (reliability)
Questionnaires are useful when …
• Used with large numbers of
respondents in multiple locations
• Information required is straightforward
• Standardized data from identical
questions is required
Questionnaire issues: Sampling
• Is a sample representative of a total
population really necessary?
• Or can the sample in a qualitative survey
represent itself?
• If a representative sample is needed are
there ready-made sampling frames?
• Will sampling be random or in clusters?
• How important is sampling size?
Quantitative Survey Issues: Sampling 1
• If the target population is : secondary
school teachers of English in Bulgaria
• How large a sample?
30 or more people? 5-10%?
• Problem of self-selection in survey
responses. How to solve this?
Quantitative Survey Issues: Sampling 2
• Sample A: teachers belonging to five
different schools (cluster sampling)?
• Sample B: randomly selected?
• Sample C: randomly selected within
large clusters (eg the five schools)?
• Sample D: Every 10th teacher on a
national register (or other sampling
frame)?
Questionnaire issues: Piloting
• What should be piloted? Everything!
• Questionnaire layout, length, question
types, question wording, order of
questions, rubrics all need to be tested
• It is impossible to get things right at a
first attempt; second attempts, in turn,
will need to be piloted
Advantages of questionnaires
• Supply a large quantity of data for a
relatively low cost: not labour-intensive
• Standardized pre-coded answers can
enable speedy data collection,
management and even analysis
• Eliminate ‘interviewer bias’ – at least to
a certain extent
• Face validity
Questionnaire problem areas
• Low response rates
• Frustration for the respondent: boxticking can deter respondents when no
answer seems appropriate
• Frustration for the researcher: no scope
for clarification of answers
• One chance only to ‘get it right’ –
especially in terms of item wording
Creating web-based questionnaires
• Go to: http://www.surveymonkey.com/
• Design and edit your survey
• Send the link by email to potential
respondents
• Collect and analyse the data
• Note: limitations of the free vs the paid
version of this software
Issues for all types of questionnaire
• What makes a good questionnaire item?
• Why is it so difficult to formulate the precise
questions that we need to ask in order to
obtain the required answers?
• Why is it sometimes difficult to interpret the
answers that we obtain?
Fundamental Problem 1
• “It is almost impossible to ask a
question without suggesting answers.
The very fact that a question is asked
implies that the researcher thinks the
topic is of interest. Moreover, the way a
question is asked inevitably reflects the
researcher’s preconceptions. Unstated
presuppositions always underlie a
question. “ (Foddy 1993: 53-4)
Fundamental Problem 2
Q: Which soft drink do you usually buy?
(What is a ‘soft drink’ and what isn’t?
How often is ‘usually’?
What is understood by ‘buy’ and by ‘you’?
What if you buy more than one?)
A: Probably the first brand that comes to mind
The question may not be understood as it is intended
and results may therefore be invalid.
(Starting point: what information does it seek to elicit?)
Question questions such as …
1. How many journeys have you made on
London Underground over the past month?
2. Do you enjoy going to coffee bars and
restaurants in the evening?
3. Do you regularly do your shopping in large
supermarkets?
4. The library facilities at Essex University are
adequate for my needs.
Strongly agree ( )
Disagree ( )
Agree ( )
Strongly disagree (
)
And these as well …
1. Complete the following sentence.
I enjoy studying at Essex because …
2. Do you think that the library should do more
to improve its facilities for Masters
students?
3. How important is exercise as part of your
daily routine?
Very important (
Not sure (
)
)
Important (
)
Unimportant ( )
Framework for questionnaire design
1. Interviewer
Encodes question, taking into
account own purposes,
presumptions/knowledge about the
respondent, and perceptions of the
respondent’s presumptions about
self (ie the interviewer.)

2. Respondent
Decodes question, taking into
account own purposes,
presumptions/knowledge about the
interviewer, and perceptions of the
interviewer’s presumptions about
self (ie the respondent)
4. Interviewer
Decodes answer, taking into
account own
presumptions/knowledge about the
respondent and perceptions of the
respondent’s
presumptions/knowledge about self
(i.e. the interviewer.)
3. Respondent
Encodes answer, taking into
account own
presumptions/knowledge about the
interviewer and perceptions of the
interviewer’s
presumptions/knowledge about self
(i.e. the respondent.)


Design principles
• Choose the right instrument!
• Target information required first →
afterwards the right questions to ask.
Required information depends on aims.
• Rigorously monitor all items
• Pilot and re-pilot the questionnaire
• Provide clear instructions and standard
procedures (for reliability)
Open vs closed questions
OPEN: Answers have more depth but
data processing is more complex.
CLOSED: Easy to process but many
useful insights are lost and respondents
can become irritated by being put ‘in
boxes’. (Oppenheim 1992, p.115)
Some pitfalls to avoid
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Respondents often answer questions when they don’t really know
the answer.
Respondents can deliberately not answer or answer incorrectly (ie
they lie)
Respondents misinterpret questions
Small changes in wording can produce major changes in distribution
of responses.
Attitudes and opinions can be unstable; circumstances can change
Memory can be unreliable.
Relationship between what respondents say they do and what they
actually do is not strong.
Cultural context has an impact on responses (eg attitude scale
grades)
The format itself can affect responses
The order of questions and answers to earlier questions can affect
responses.
NEXT WEEK !!!
Questionnaire Item-Writing Workshop:
Coursebook Feedback Survey
Types of 1-to-1 interviews
• Structured: with tight control over
question format and possible answers.
Like a face-to-face questionnaire
• Semi-structured: with a clearly-defined
question schedule but some flexibility
and more open-ended answers
• Unstructured: a narrative prompted by
one general question, perhaps
1-to-1 Interviews are useful when …
• Detailed information is needed from
respondents
• A smaller number of respondents is
acceptable
• Attitudes or feelings are investigated
• Sensitive issues are explored (?)
• ‘Key players’ are targeted
• Ideas for a questionnaire need to be explored
or ‘fine-tuned’
Advantages of 1-to-1 interviews
•
•
•
•
Depth of information
Insights eg gained from key informants
Flexibility of formats
Validity: direct contact means that data can
be checked for accuracy and for relevance
• High response rate
• Opportunity for targeted individuals to make
their voice heard – element of advocacy
Interview problem areas
• The ‘interviewer effect’ and power
relationships: face-to-face contact can directly
influence answers
• The ‘instant position’ effect: respondents feel
obliged to supply answers of some sort
• Investment of time and resources
• Complexity of data analysis
• Reliability issues for multiple interviews
Reliability issues
• How can we know if respondents
consistently understand the question?
Or if they have misinterpreted it?
• How can we know if respondents are
consistently telling the truth? Or if they
are misleading the interviewer
deliberately or at a subconscious level?
Features of interview design
•
•
•
•
•
•
•
How many questions?
In what order?
How long/short?
Format? (open qqs, statements etc.)
Follow-up probes and/or prompts?
Wording of questions?
Mode of recording/analysing answers?
Conducting an interview
• Establishing a relationship/trust
• Explaining aims & procedures
• Low key presentation of self and
involvement
• Active listening
• Clarifications, probes and prompts
• Recording, transcribing and analysing
Focus groups
• Consist of a small group of people who
are brought together by a trained
‘moderator’ (usually the researcher) to
explore attitudes, feelings and ideas
about a particular topic or set of issues
Main characteristics of FGs
• A question schedule is prepared by the
moderator to focus the discussion
• Place value on interaction within the
group as a means of eliciting
information – a social experience
• No requirement to reach consensus
• Less pressure to hold a fixed point of
view – in fact opinions can develop
FGs are useful when …
• Topics or issues need to be explored in
depth, or attitudes revealed
• Typical groups of key informants are to
be targeted eg language teachers
Advantages of FGs
• A less directive interview format
• ‘Interviewer bias’ can be greatly
reduced
• Provide a more natural social forum for
the exchange of ideas. Interviewees are
able to take the initiative – not just
respond
• The right to silence
FG problem areas
• Less directive means less predictable
• Huge amounts of data can be collected,
much of which may be unusable
• Perception that FGs are a market
research tool (Tony Blair & New Labour)