Transcript Seven possibly controversial but useful rules

Seven possibly controversial
but hopefully useful rules
Paul De Boeck
K.U.Leuven
Examples
• Political openness and economic
openness
• Life satisfaction and cost of living
• Research performance and innovation
intensity
• Socioeconomic status and intolerance
Multi-level governance structure
Link concept-operationalisation
Participation
Competition
Political
openness
Transparance
various aspects
Accountability
Rule of law
Interfaces
combine
subindicies
Implicit theory
the thing to
be measured
X1
numerical obs
X2
numerical obs
X3 numerical obs
test score
global index
X4 numerical obs
“Measurement”
the thing to
be measured
X1
numerical obs
X2
numerical obs
X3 numerical obs
assignment rule
e.g., sum, first
component score
X4 numerical obs
• Heavy vs light on meaning
Alternative
Can I explain?
Which model
can explain?
X1
numerical obs
X2
numerical obs
X3 numerical obs
X4 numerical obs
The data are not meant as a measurement of something,
but as to be explained.
For example, responses to inventory items,
how can they be explained?
Do the correlations between items stem from overlap in
the information used to respond?
Which information is it?
Why not extract the information directly?
What is the origin of the information?
When explained with a quantitative theory, then
measurement is a by-product
1. Not everything is worth being measured
or can be measured, often the data are
more interesting than the concept
Assignment of numbers
• number finding: counts, percentages
• number asking: ratings
• number construction: apply a rule on
original numbers in order to obtain a
derived number
“Measurement”
the thing to
be measured
X1
numerical obs
X2
numerical obs
X3 numerical obs
assignment rule
e.g., sum, first
component score
X4 numerical obs
Measurement
• A quantity
• Increasing or decreasing doesn’t change
the nature
• Addition from two sources is possible
• Splitting is possible, e.g. in halves
Questions
• Why are you interested in the link between the
two concepts?
Why do you want to measure?
Because I want to test a theory
data for the theory
Why aren’t you interested in the data?
and try to explain the data?
theory for the data
• Aren’t your numerical variables of sufficient
interest to keep them as they are?
Examples
• Woodworth Personal Schedule 1917 to
measure psychological adaptation
Before, lists of questions were used and one
would listen to the responses
• Hirsch index: the maximum obtained by
selecting a number of publications with
each at least the same number of
citations, e.g., 15 articles with 15 or more
citations
• A strong dimension does not mean the
conceptual component is important.
It shows there are large individual
differences in the component.
2. Psychometric criteria such as reliability
and validity are not theory-independent
• The underlying theory is the simple implicit
theory
• Alternatives
- canalization: one behavior has developed into
a the dominant one and excludes the other
behaviors
- behavior competition: the strongest takes it all
- negative feedback: showing a behavior makes
it less likely to occur next
- drop-out: only occasionally it is affected by
Dynamic theories
Dynamic theories
Dynamic theories
Reliability
Repetition over
• Situations
• Behaviors
• Time
Questions
• Do you have the simple theory for your
data that they are a direct and linear
reflection of the concept?
• What is your theory of stability?
Stability over?
3. Always reflect on which type of
covariation is meant when speaking about
the link between two concepts
The case of shame and guilt
• Covariation over situations
guilt vs shame is one of two dimensions
• Covariation over persons
guilt & shame define a dimension together
with fear and anger
• Covariation over cultures
guilt and shame define their own common
dimension
Negative emotions
• Fear and anger are positively correlated
over persons
• Fear and anger cannot co-occur because
they rely on opposite action tendencies
(flight and fight)
Guilt
• Experienced norm violation
• Self-reproach
• Tendency to restitute
Unidimensional in the sense of individualdifferences, and they each contribute
separately to the probability of feeling
guilty
Questions
• Are you interested in individual
differences?
Are you ready to find traits?
• Components of?
Meaning – semantic
Individual differences
Situational differences
Time differences
Probability of occurence
4. Measurement, reliability, validity,
hypothesis testing don’t need to be
sequential steps
Hypothesis:
link between concept A and B
• Step 1: construct a measurement for A, B
• Step 2: test reliability measurements
• Step 3: test validity measurements
• Step 4: test hypothesis
measurement
measurement reliability
validity
measurement
measurement
hypothesis testing
validity measurement reliability
hypothesis testing
Questions
• Do you want to construct a test?
• ?
Meaning – semantic
Individual differences
Situational differences
Time differences
Probability of occurence
5. Always do a PCA
• PCA tells you about the sources of
differences between the row elements
• PCA tells you whether there is interaction
and where it is
• PCA is a quite robust way to check
multidimensionality
• PCA shows the main interactions in a
repeated measures data matrix
- unidimensional & equal loadings
- unidimensional & unequal positive
loadings
- unidimensional bipolar
- multidimensional
Questions
• Show me your PCA before we continue,
especially when complex methods are
going to be used, such as SEMs
6. One does not necessarily have to care
about the scale of the data
• Common concern:
“what is the scale level?”
“are parametric statistics permissible?”
• Scale level only matters when
- numbers are taken for an index of
something else, how does the index relate
to the “something else”?
• Transformations are interesting when a
simpler and better structure can be found
Representations of relations
Example
P(Xpi=1)/(1-P(Xpi=1)) = p / i
• p and i are on a ratio scale,
as far as they represent odds ratios
Questions
• Suppose you forget about the scale level
and you find an interesting relationship
• Do you want to generalize over other
number assignment procedures?
• How meaningful are the numerical
variables as they are?
7. Don’t construct indices of concepts,
unless for descriptive summaries
Problems
• The global index depends on the components,
and hence, on the definition.
• Often definitions are arbitrary or they are mainly
semantic
• Perhaps the relationships of the index follow
from the relationships of the components
Questions
• What is the definition?
• What do others say?
• Aren’t you interested in the components?