Transcript The Role of Unmet Expectations in the Psychological Contract Judy

The Effect of the Conscientious Responders Scale on Random Responding Rates in
Psychological Questionnaires
Noor Shubear, Zdravko Marjanovic, Lisa Bajkov, & Tsz Yin Fung
Thompson Rivers University
Introduction
Design and Results
The validity of questionnaire data is based on the expectation that responders are
Results and Discussion
Hypotheses
conscientious responders (CR)—they answer items as honestly and accurately as
1.The CRS-embedded questionnaire will produce smaller ISD scores compared to the
Table 1
they can.
non-embedded questionnaire .
Random Responding Differences between CRS-Embedded and Non-Embedded
Questionnaires
The problem is, for a variety of reasons such as disinterest and fatigue, some
Random
NEQ
CEQ
Responding
Mean
Mean
t-test
p-value
NEU_ISD
0.93
0.93
-0.02
.985
Participants and Procedure
EXT_ISD
0.92
0.89
0.54
.591
The CRS is a 5-item validity scale (see below) that differentiates between CR and RR,
A 109 undergraduate students were randomly divided into two groups. The first group-
OPE_ISD
0.98
0.95
0.56
.579
which has been shown to be an effective tool in detecting RR with 93% accuracy rate.
composed of 55 students- completed a 208-item paper-and-pencil questionnaire that did
AGR_ISD
0.92
0.94
-0.64
.526
(CRS; Marjanovic, Holden, et al., 2015. Marjanovic, Struthers, et al., 2014).
not contain the CRS. The other group- composed of 54 student- completed a 218-item
CON_ISD
0.92
0.82
2.14
.035
However, a question remains to be answered is whether the presence of the CRS scale
paper-and-pencil questionnaire which contained a double size CRS. For the purpose of
__________________________________________________________________
in psychological questionnaires affects the rates of random responding. Does the
this study, we only analyzed data from the 60-item five-factor inventory (NEO-FFI) to
Note. NEU=neuroticism, EXT=extraversion, OPE=openness to experience, AGR=
presence of the CRS items in a questionnaire deter responders from answering
determine the effect of CRS in psychological questionnaires. To test the effect of the
agreeableness, CON=conscientiousness.
randomly by alerting them that the validity of their data can be assessed?
CRS on random responding rates, we used a statistical approach that does not alter the
2. Means and standard deviation will be about equal across both questionnaires.
responders are random responders (RR)—they answer items carelessly or
indiscriminately, without regard for what items are asking them to consider.
Method
look and/or the content of the questionnaire as CRS does. The inter-item standard
The purpose of this study was to test whether the presence of CRS scale in a
deviation (ISD; Marjanovic, Holden, et al., 2015) is statistical technique that reflects how
psychological questionnaire would (1) affect the rates of random responding as
the responses are closely clustered around the mean score. CR produces a score that is
compared to a questionnaire without the CRS scale, and (2) result in any significant
closely clustered around the mean ( i.e., small ISD score), while RR produces the
Table 2
differences in the means and standard deviations.
opposite effect.
Mean-Scores Differences between CRS-Embedded and Non-Embedded Questionnaires
NEQ
CEQ
Measures
Mean
Mean
t-test
Measures
NEU
3.32
3.12
1.69
.095
All items were answered on a 5-point Likert scale
EXT
3.38
3.42
-0.42
.676
Conscientious Responders Scale (CRS; Marjanovic, Struthers, et al., 2014)
OPE
3.43
3.50
-0.59
.559
Inter-Item Standard Deviation (ISD; Marjanovic, Holden, et al., 2015)
AGR
3.57
3.55
0.251
.802
CON
3.44
3.63
-1.84
.069
__________________________________________________________________
Note. NEU=neuroticism, EXT=extraversion, OPE=openness to experience, AGR=
.
The CRS utilizes instructional item content. Each CRS item, randomly embedded in a
questionnaire, instructs responders exactly how to respond to that particular item.
CRS
1. To answer this question, please choose number three, neither agree nor
NEO Five Factor Inventory (NEO-FFI; McCrae, Robert R.; Costa, Paul T.,
Jr., 1978; 2010), Its 60 items measure five factors of personality; neuroticism,
extraversion, openness to experience, agreeableness, and conscientiousness.
disagree.
2. Choose the first option—strongly disagree—in answering this question.
3. To respond to this question, please choose number five, strongly agree.
p-value
agreeableness, CON=conscientiousness.
Results
All studied measures produced high estimates of reliability (Cronbach alpha > .70).
4. Please answer this question by choosing number two, disagree.
5. In response to this question, please choose number four, agree.
Hypothesis 1 statistics are presented in table 1. The results do not support the
hypothesis. The presence of the CRS in a psychological questionnaires seems to have
Compliant responses are scored as 1s and incompliant responses are scored as 0s.
Items are summed to make a score that ranges from 0 (all incompliant responses) – 5
no influence on reducing or deterring rates of random responding as compared to
questionnaires that do not contain CRS.
(all compliant responses). Because, on a 5-point response scale, it is so statistically
unlikely that RR can answer several CRS items compliantly by chance alone, high CRS
sum scores (3 – 5) must reflect conscientious responding whereas low CRS sum
scores (0– 2) probably reflect random responding.
Hypothesis 2 statistics are presented in table 2. The results do support the hypothesis.
The presence of the CRS in psychological questionnaires seems to have no effect on
means and standard deviations of psychological measures. Questionnaires with and
without the CRS in it produced similar descriptive statistics.
Conclusion
1. The presence of CRS items in a questionnaire had no effect of reducing or
deterring rates of participant random responding.
2. A positive finding: the CRS had no effect on means and standard deviations
across the CRS-embedded and non-embedded questionnaires.