Transcript WebCT Update

Designing Web Surveys
Dr Lisa Wise
Dr Lisa Wise 20/09/2002
Overview
•
•
•
•
•
web surveys
overview of survey design
when to use surveys
how to design survey questions
data analysis and reporting
– data be collection, storage and analysis
– ethical issues to do with handling sensitive or
personal data
Dr Lisa Wise 20/09/2002
Web-based Surveys
• can do from anywhere with web access
– but samples are biased against those with no
internet access (> 50% of Aust households are
connected - AC Nielson Mar 2002)
• data can be “validated” before submission
• note that validation is in terms of correct data
type rather than meaningfulness
• issue of multiple submissions vs anonymity
• quick data collection phase for researcher
Dr Lisa Wise 20/09/2002
Web Survey
survey form
processing scripts
database
web server
filesystem
Survey data might go to a
database or might be stored
in a file in the file system
or could be emailed to the
researcher
email
Dr Lisa Wise 20/09/2002
Web survey design
• use standard web design principles
• especially ensure that
– survey is not too long (or can be saved)
– information travels correctly between screens
– error messages, if they occur, are meaningful
– all fields are validated and can handle
appropriate input and reject inappropriate input
– check field length, data type, illegal characters
Dr Lisa Wise 20/09/2002
Making a web survey
• put the questions into an HTML form
• get the data formatted for analysis using
the scripting tools of your choice
• set up access controls if required
• test across a range of browsers with a
range of good and bad input
– ensure security / integrity of your data, but also
of the system collecting your data
Dr Lisa Wise 20/09/2002
Data collection and storage
• use cgi-mailer or FormMail to email
formatted form variables to researcher
– http://www.its.monash.edu.au/web/resources/cgi-mailer.html
– http://www.its.monash.edu.au/web/resources/formmail.html
• use a script (eg Perl, PHP) to write to a file
on the filesystem
• use a script to put data into a database
– don’t put into a database if you are then going
to take it out into SPSS / excel to analyse it !!
Dr Lisa Wise 20/09/2002
Privacy
• “Privacy laws impose quite specific requirements
regarding data storage and security, access and
correction rights, ensuring data is accurate before
use, used only for the purpose for which it was
collected or for which consent has been given,
and disclosure only in limited circumstances.”
• “Legal requirements may be more detailed than
ethical requirements …”
• http://www.monash.edu.au/resgrant/human-ethics/privacy/index.html
Dr Lisa Wise 20/09/2002
Handling data
• shared responsibility of technical staff and
researchers to ensure that privacy and
confidentiality requirements are met
– “information is only de-identified only if all
identifying information has been irreversibly
removed from the record”
– “retention of codes which allow recovery of
identifiable information means record is not deidentified”
Dr Lisa Wise 20/09/2002
Cost effectiveness
• even a simple web survey takes a couple
of days to implement and test
• cost-effectiveness of solution needs to
include real costs
– eg email vs database solution might involve 3
days of coding for programmer versus 1 day of
data entry (cutting and pasting from email to
analysis program) for researcher
– cost saving or cost shifting?
Dr Lisa Wise 20/09/2002
Making a survey ...
• technical “design” requirements covered in
previous slides but what about designing
the actual survey questions ?
• there is a whole discipline area focused on
designing surveys and questionnaires and
analysing data collected using these tools
• it is not a trivial exercise and committees /
managers are not ideal survey designers
Dr Lisa Wise 20/09/2002
Overview of survey design
• survey design requires that you have
clear research questions
• survey questions need to be focused on
answering your research questions
• survey design includes generating the
questions and planning the data analysis
• planning the data analysis happens before
the survey is released !!
Dr Lisa Wise 20/09/2002
When to use surveys
• surveys allow you collect lots of data
relatively quickly and cheaply
• lots of poor quality data is never better
than smaller amounts of high quality data
• lack of time and money are not excuses
for collecting poor quality data - if you
can’t afford to collect reasonable quality
data, don’t do the research.
Dr Lisa Wise 20/09/2002
Personal data
• most surveys request demographic
information about respondents
• usually ask for opinions about something
• collecting either type of information has
ethical and privacy implications
• survey designers should be familiar with:
– http://www.monash.edu.au/resgrant/human-ethics/
– above page has link to Use of Personal Information
Dr Lisa Wise 20/09/2002
… from Privacy statement on web
“On-line Surveys
All research surveys conducted on-line by
University staff and /or students which involve
the collection of personal information, will have
received approval from the University's
Committee for Human Ethics in Research.
A survey might ask visitors for unique identifiers
(such as login information).”
Dr Lisa Wise 20/09/2002
Sensitive information
• “Do the records or information you are
collecting or using include any sensitive
information (such as political opinion or
memberships, religious beliefs or
affiliation, philosophical beliefs … )”
– note that peoples’ opinions are considered to
be personal information
– be aware of perceived power relationships and
potential access to confidential information
Dr Lisa Wise 20/09/2002
Classroom and workplace surveys
• work / class surveys have ethical issues
related to perceived power relationships
between respondents and researchers
• even if survey does not require specific
ethics committee approval, the ethical and
privacy principles should be considered
• go through all the ethics forms whether or
not they need to be submitted
Dr Lisa Wise 20/09/2002
SCERH ethics forms
• based on National Statement on Ethical Conduct
in Research Involving Humans (NH&MRC, 1999)
• apply to “anyone who is gathering information
about human beings and organisations through
interviewing, surveying, administering
questionnaires, observing human behaviour,
taking human tissue / fluids …”
• “ … there are no exceptions, exclusions or
blanket permissions”
Dr Lisa Wise 20/09/2002
Sampling
• Sampling design and survey design must
be tightly coupled
• Who or what are you planning to draw
conclusions about ?
• Are they a homogenous group or are
there sub-groupings in the population ?
– do you need a stratified sample?
– do you want to compare between groups?
Dr Lisa Wise 20/09/2002
Sampling
• Surveys usually use convenience samples
• Demographic information is collected to
allow comparisons between target groups
• If targeting specific groups is critical to
your research, consider interviewing
participants, or delivering and collecting
surveys from selected participants
• convenience sample =/= random sample
Dr Lisa Wise 20/09/2002
Sampling
• for statistical techniques, calculate the
sample size required for valid conclusions
• consider whether you want lots of general
data or whether you are actually interested
in very specific focussed data
• a large survey doesn’t give more objective
data than eg a focus group unless it follow
rigorous methodology
Dr Lisa Wise 20/09/2002
Types of questions
• Open ended
– more difficult to answer and to code or to
analyse objectively
• Closed questions
– forced choice (one of two mutually exclusive)
– multiple choice (one of several)
– checklist (one or more of several)
– partially closed (alternatives including “Other”)
Dr Lisa Wise 20/09/2002
Questions
•
•
•
•
•
avoid double-barrelled questions
avoid leading questions
avoid motherhood statements
avoid undefined terms
ensure that your questions lead to
responses that interest you and
conversely that responses that interest
you are elicited by your questions
Dr Lisa Wise 20/09/2002
Order effects
• funnel questions from general to specific
• can use general filter questions to
determine whether respondent should be
asked detailed questions
• can have some practice questions
• counterbalance order of presentation
• prevent response sets
• can use alternate forms of questions
Dr Lisa Wise 20/09/2002
Content analysis
• talk to expert about content analysis
• consider whether you are happier with the
assumptions underlying content analysis
or your research team’s ability to interpret
and code responses
• if you code responses, take measures of
inter- and intra-rater reliability
– do raters make similar / consistent judgements
Dr Lisa Wise 20/09/2002
Response scales
• response scales should allow people to
communicate what they want
– eg there are differences between neutral,
undecided, don’t know, don’t care, n.a.
• anchor points should be bipolar
– need to test this on pilot sample
boring
boring
interesting
fun
Dr Lisa Wise 20/09/2002
Measurement scales
• Ratio scales (true zero - eg age, height)
• Equal Interval scales
– Thurstone scales, Likert scales, Guttman
et al, Social
scales, semantic differentials Albrecht
Psychology, pp 190-198
• Ordered scale (eg <18, 18-30, 31-50, >50)
• Categories (ITS, Med, Sci, Arts …)
• Different types of data are amenable to
different analyses: consult a statistician!
Dr Lisa Wise 20/09/2002
Rating scales
Likert scales should have 5 marked values not 7
strongly
disagree
disagree
undecided
agree
strongly
agree
The following multiple choice format is still an interval scale
1. Strongly disagree
2. Disagree
3. Undecided
4. Agree
5. Strongly agree
Dr Lisa Wise 20/09/2002
Rating scales
• constructing rating scales to have specific
psychometric properties is labourintensive and requires expertise
– multiple forms or minimal number of questions?
– group similar questions or intermix?
– alter format to avoid response bias or keep
consistent to avoid response errors?
– normalise data or trust participants responses?
• consult a statistician or social scientist
Dr Lisa Wise 20/09/2002
Measurement
• rating scales should be equal-interval
scales to use parametric statistics
• the fact that you have numerical data does
not mean it is accurate or reliable
– male (1) - female (0) … obvious that 0 and 1
are codes not numbers
– rarely (1) - sometimes (2) - often (3) … things
that can be ranked are not necessarily equal
interval scales - can you take the mean?
Dr Lisa Wise 20/09/2002
Statistics
• Descriptive
– describe a sample distribution in terms of
shape, central tendency and variance
• Inferential
– draw inferences about population parameters
based on sample statistics
– hypothesis testing - test whether a statement
is true or false based on sample statistics
Dr Lisa Wise 20/09/2002
Reliability and validity
• Reliability
– test - retest
– surveys are only a snapshot at particular time
• Validity
– face validity
– criterion-based validity
– construct validity
– internal / external validity
Dr Lisa Wise 20/09/2002
Hypothesis testing
• null hypothesis H0 (no effect of
experimental manipulation)
• alternative hypothesis H1 (experimental
manipulation has an effect on this test
statistic)
• type 1 error (accept H1 when H0 true)
• type 2 error (accept H0 when H1 true)
Dr Lisa Wise 20/09/2002
Inferential statistics
• use statistics to make inferences about
populations from your samples
• need to be aware of assumptions
underlying statistical tests
– eg t-tests and anovas assume continuous
underlying variable, normal distribution and
homogeneity of variance
– what happens when assumptions are violated?
Dr Lisa Wise 20/09/2002
Statistics
• most survey data is not ratio and may not
be interval data (depending on your
perspective on this)
• non-parametric statistical tests don’t make
assumptions about underlying distribution
• don’t use statistics to show things you
can’t see by eye - statistics help decide if
what you can see is “significant”
Dr Lisa Wise 20/09/2002
Correlational studies
• surveys usually correlate opinions with
demographic information
• correlations show relationships between
variables but don’t address causality
• correlation coefficients indicate strength of
relationship (between 0 and 1)
– r of .8 explains 64% of variance, or the degree
to which knowing X can predict the value of Y
Dr Lisa Wise 20/09/2002
But I’m not doing real research ...
• I just want to run a quick survey
• This stuff doesn’t apply to me cos I’m not
doing real research
• . . . so what are you doing?
– “anyone who is gathering information about human
beings and organisations through interviewing,
surveying, administering questionnaires, observing
human behaviour, … no exceptions ...”
Dr Lisa Wise 20/09/2002
What constitutes research?
– If you are going to summarise responses on
your survey and you are going to act on them
in any way, you are doing research
– for that research to be meaningful, you need be
aware of proper research methodology and the
limitations of what you are doing
– if you can’t do it properly, don’t do it at all !!
– informed professional opinion can far more
valuable than poor quality survey data
Dr Lisa Wise 20/09/2002
When to use web surveys
• small amount of non-confidential data from
wide range of people
– not good for sensitive or confidential data
– not good where loss of data would be a major
problem
– not good for long surveys (usability issues)
• can use web to download a survey which
is printed out and submitted
Dr Lisa Wise 20/09/2002