Transcript Using traditional control systems to reveal changes when

1
Using traditional control systems
to reveal changes when
introducing new data collection
instruments
Sindre Børke ([email protected])
Senior Adviser
Division for Data Collection Methods, Statistics Norway
1
Paper contra web Questionnaires
Information model
(survey definitions)
Web
questionnaire
Paper
questionnaire
Websolution
Survey
database
Data from paper
questionnaires
Editing Process
Data from web
questionnaires
2
Outsourced editing gives better data
• Household census: average 6,2 and 3,4 errors
• Hotel statistics: average 1.7 and 0.7 errors
• Hotels, prize per room: 41% contra 36% with error
3
Without controls, more errors in web?
• Not in household census 2001
• Hotels, number of rooms let out: 12% contra 18% error
• (PPI: 1.27% contra 1.40% outside the defined range)
• Road goods transport: Different patterns to be explained
4
Behind – or on - the screen
• Following the respondent’s filling-in process, and the
warnings presented during this process
5
Errors in incoming web and paper version of the
hotelActivated
statistics
questionnaire
error messages
on Web,
Activated
errorNovember
messages on paper, November
∑ guest from different nations ≠ ∑
foreign guest
11.9
∑ Norwegian arrivals > Norwegian
logdings
6.1
∑ rooms let > ∑ rooms available
1.9
∑ different kinds of lodgings ≠ Total
-30 -25 -20 -15 -10 -5
0
5
10 15 20 25 30
12.4
0
5
10
15
20
25
30
6
Errors detected in the web and paper version of
theActivated
hotelerrorstatistics
questionnaire
messages on Web,
Novembererror messages on paper, November
Activated
∑ guest from different nations ≠ ∑
foreign guest
3.0
∑ Norwegian arrivals > Norwegian
logdings
16.7
∑ rooms let > ∑ rooms available
18.1
-25
6.1
1.9
∑ different kinds of lodgings ≠ Total
25.2
-30
11.9
-20
-15
-10
-5
0
5
10
15
20
25
30 0
12.4
5
10
15
20
25
30
One error message
More error messages
7
Transfer of electronic data
Information model
(survey definitions)
Individual Supporting
adaptions software
Web
questionnaire
Basic
software
Enterprise
database
Editing Process
Data from
Enterprise system
Paper
questionnaire
Websolution
Survey
database
Data from paper
questionnaires
Data from web
questionnaires
8
Transfer of electronic data
• Allready in wide use for accounts and salary data in Norway.
• Presents challenges in development and upkeeping
– Supporting software from the software vendors
– Individual adaptions at the respondent’s
• Looking into one case with traditional controls, finding better
results than from paper questionnaires
• Requires different control/editing, due to different set of
possible errors and different consequences when errors
occurs
9
In short:
• Few surveys seems to have data to observe possible
changes in quality
• Those data we find seem to tell that there are changes
• Possible changes in data that are not part of a control can
not be revealed with the instrument we use in this paper
• This field must be further investigated, and give feed-back/forward to the editing procedure and to questionnaire
developing
10