Transcript Presentation 3

Applied Opinion Research
Training Workshop
Day 3
Preparing to
Conduct Research
Sharon Felzer
Conducting Research
Going into the Field
 Prepare Materials
 Translate
and back-translate questionnaires and
guidelines
 Provide sufficient copies, return mail envelopes, etc.
 Determine Schedule
 Do
not field right before or during holidays
 Allow enough time to recruit sample, conduct research,
analyze data, and prepare reports
Conducting Research
Going into the Field
Gather Your Resources
 Translators
 Local contacts to recruit samples
 Local interviewers/moderators
 Transcribers/Data entry
 Data analysts
 Contractors (ESOMAR Website)
Conducting Research
Instrument Quality Control
 Vet among core research team

Be sure research objectives will be met
 Pilot test



Be sure participants will understand instructions, questions, and skip
patterns
Be sure length is appropriate
Be sure guideline encourages discussion
 Back-translate

Be sure questions ask what they were intended
to ask
Conducting Research
Avoid Common Pitfalls
 Poorly designed instruments

Research is only as useful as the instruments used
 Poorly recruited sample







Findings are highly dependent on the sample used
Unskilled interviewers/moderators
Generalizing beyond research/sample
No buy-in
Poor timing of research
Poorly designed Terms of Reference
Cost overrun
Conducting Research
When to Bring in Contractors/Experts






Language barrier
Cultural barrier
Large or multiple samples
Local contacts needed to encourage participation
Preventing bias
Complicated research objectives
Analyzing and
Reporting Results
Mary McIntosh
Analyzing & Reporting Results
Essential Components of Successful Research:
 Appropriate sampling
 Valid instruments
 Accurate translations
 Skilled interviewers/moderators
 Effective timing
 Accurate data entry/transcriptions
 Appropriate analyses and reporting
 Reasonable response rate
Analyzing & Reporting Results
Analyzing Data
Use statistics appropriate to responses




Rating scale: means and standard deviations or
ranges
Dichotomous and multiple choice:frequencies of
respondents’ responses
Ranking: counts of #1 ratings, etc.
Open-ended/Qualitative: no statistics appropriate
unless transcripts are coded
Analyzing & Reporting Results
Analyzing Quantitative Data
Pay attention to:
 Sample base
 Number of respondents
– The smaller the n, the less stable the parameter
estimates
 Mean or frequency of response
 Standard deviation
– The larger the standard deviation, the less reliable is
your estimate of the mean
 Test of significance
– If the test is not significant, you cannot say it is a
significant difference
Analyzing & Reporting Results
Analyzing Quantitative Data
Tests of significance
Tells you whether differences in numbers are meaningful, or
significant
 Most commonly conducted on frequency and
mean data
 A significance test of p < .05 tells you that there is less
than a 5% chance that this difference in mean responses
is due to chance
 A test of significance must be conducted before you can
say that a difference exists
 However, if a test is significant, it is not necessarily
meaningful
Analyzing & Reporting Results
Analyzing Quantitative Data
Tests of significance
Group 1
– Mean = 3.00
– Std. Dev. = 1.00
– 95% of respondents’
ratings between 1 and 5
Group 3
– Mean = 3.00
– Std. Dev. = 2.00
– 95% of respondents’
ratings between 1 and 7
Significantly
different
Not
different
Group 2
–
–
–
Mean = 7.00
Std. Dev. = 1.00
95% of respondents’
ratings between 5 and 9
Group 4
–
–
–
Mean = 7.00
Std. Dev. = 2.00
95% of respondents’
ratings between 3 and 10
Analyzing & Reporting Results
Analyzing Quantitative Data
Tests of significance
Group 1
– 45% agreed
– N = 200
Group 3
– 45% agreed
– N = 200
Significantly
different
Not
different
Group 2
– 55% agreed
– N = 300
Group 4
– 55% agreed
– N = 100
Analyzing & Reporting Results
Analyzing Qualitative Data
Pay attention to:




Sample base
Trends in responses
Potential group differences in responses
Any potential bias in responding
Analyzing & Reporting Results
Reporting Results
Keep in Mind:

Your data is from a sample of the population only
– Be very careful about generalizing your results to the entire
population

Only a subset of your intended sample actually took part
in your research
– Low response rates suggest potential response bias

You may want to consider weighting your data to correct
for bias
Analyzing & Reporting Results
Reporting Results
Keep in Mind:
 Your questionnaire or guideline asked a finite
number of questions
– It is quite possible that you did not ask about an
important factor

Despite your best efforts, there may still have
been biases
– Researcher biases (more likely in qualitative)
– Sample biases
– Cultural biases
Analyzing & Reporting Results
Reporting Results
Keep in Mind:


Although the data from quantitative data
seems quite scientific, it is still subject to
interpretation
Reporting attitudes and opinions is not the
same as reporting facts
Analyzing & Reporting Results
Reporting Results
Reporting the Sample:







The population
Sampling frame
Sample design
Rationale for sample design
Sample size
Response rate
You may want to report the number of respondents who
answered each question if there are high numbers of
“Don’t Know” or “Refused” responses
Analyzing & Reporting Results
Reporting Results
Reporting the Sample:
 What if your response rate is lower than expected
or what is traditional for that type of research in that
country?
 Report that this may be a potential limitation. For instance,
some respondents may have been discouraged to participate by
superiors. Therefore, the sample of respondents that did take
part in the research may not be representative of the population
 If possible, check the characteristics of those who participated
versus those who did not
Analyzing & Reporting Results
Reporting Results
Reporting the Sample
Because your data is only from a sample of the
population, you must be very careful about
generalizing your results to the entire population
WRONG: NGOs believe the Bank is not effective
RIGHT: NGOs who were interviewed believe that
the Bank was not effective

Analyzing & Reporting Results
Reporting Results
Quantitative Data:

Although the data from quantitative data seems
quite scientific, it is still subject to interpretation
– A mean of 7 on a 10-point scale may be a
somewhat positive response in one context (e.g.,
culture) or a very positive response in another
context (e.g., culture)
– Don’t assume scales are uniform and consistent
across topics and cultures
Analyzing & Reporting Results
Reporting Results
Quantitative Data:

Part of your interpretation involves deciding how to
report the data
– Mean scores
» Respondents from NGOs gave Bank effectiveness a mean
rating of 7.5
– Aggregate frequencies (e.g., High/Medium/Low)
» 35% of respondents from NGOs gave a high rating for the
Bank’s effectiveness
– Response frequencies
» 25% of respondents from NGOs gave a rating of 6 for the
Bank’s effectiveness
Analyzing & Reporting Results
Reporting Results
Quantitative Data:
 Report the data in the way that your readers
will be most likely to understand
 Include appropriate charts, graphs, or tables to
represent the data pictorially to assist your
readers
Analyzing & Reporting Results
Reporting Results
Qualitative Data:
 Because this data is highly subjective and completely dependent on
the particular sample you have drawn, you must be very careful in
reporting qualitative findings
WRONG: The Bank program failed because it was poorly
designed
RIGHT: Respondents in a beneficiaries focus group reported
that one potential reason the Bank program was not that effective
was that the design was not optimal given the situation
 Always include the caveat that your findings are not necessarily
representative of the population (or the +/-)

Analyzing & Reporting Results
Reporting Results:
Right & Wrong
– Finding: When asked to choose what the Bank’s greatest
value was, 25% of respondents (the largest %) chose
financial resources
WRONG: The Bank’s value is only in its financial
resources
RIGHT: A quarter of respondents chose financial
resources as the Bank’s greatest value. This was
followed by…

Analyzing & Reporting Results
Reporting Results
Right & Wrong
– Finding: When asked about the Bank’s overall effectiveness,
NGO’s mean was 6.5 and private sector’s mean was 7.8, a
significant difference
WRONG: NGOs do not think that the Bank is an
effective organization. In contrast, people in the private
sector think that the Bank is highly effective
RIGHT: Respondents from private sector organizations
rated the Bank’s overall effectiveness significantly higher
(7.8) than respondents from NGOs (6.5).

Analyzing & Reporting Results
Reporting Results
Right & Wrong
– Finding: When asked about the Bank’s overall effectiveness,
the 3 media respondents gave a mean of 9.0
WRONG: The media think that the Bank is very effective
RIGHT: Although respondents from the media rated the
Bank’s overall effectiveness quite high (9.0), there were
only three media respondents, thus, these results are
suggestive at best
» Another alternative is to not report results from
samples sizes that you judge too small to be reliable

Analyzing & Reporting Results
Reporting Results
Right & Wrong
– Finding: In a focus group of medical professionals,
several said that the Bank needs to initiate a vaccine
program
WRONG: The Bank needs to initiate a vaccine program
RIGHT: In a focus group of medical professionals, it was
recommended that the Bank initiate a vaccine program

Hands On Work:
Analyzing and Reporting Results