Transcript Slide 1

Using Substantive Diagnostics to Evaluate
the Validity of Micro-level Latent Class
Indicators of Measurement Error
Clyde Tucker and Brian Meekins
U.S. Bureau of Labor Statistics
and Paul Biemer
Research Triangle Institute
www.bls.gov
Background
Developed by Lazarsfeld (1950)—unobserved or
“latent” variable drawn from relationships
between two or more “manifest” variables
 Lazarsfeld and Henry (1968) and Goodman
(1974) extended mathematics of theory
 Software for latent class analysis (LCA)
developed (MLLSA, lEM, M-PLUS)
 LCA used to study measurement or response
error (VandePol and deLeeuw 1986; Tucker
1992; Van de Pol and Langeheine 1997; Bassi et
al. 2000; Biemer and Bushery 2000; Tucker, et
al. 2002, 2003, 2004, 2005, 2006, and 2008)

Creation of Manifest Variables
Try to create at least three
 Try to avoid direct relationships with
outcome variable (expenditures, in this
case)
 Use LCA to “triangulate” them to
produce a latent variable with more
information than any one of them alone

Statistical Logic

In mathematical terms, when manifest variables A
and B are not independent, the following relationship
will not hold:
 ijAB   iA   B
j

where i indexes the classes of A, j indexes the
classes of B, πijAB is the probability an individual is in
cell ij, πiA is the probability an individual is in class i,
and πjB is the probability an individual is in class j.
Statistical Logic

For the above expression to be true, A and B must be
independent. The purpose of the latent variable X is to
achieve that independence. Thus, the following latent
class model is desired:
ABX
 ijt
  tX   itAX   BX
jt

where t indexes the classes of X, πijtABX is the probability
of being in cell ijt of the unobserved ABX table, πtX is the
probability that an individual is in one of the mutually
exclusive and exhaustive classes of X, πitAX and πjtBX are
the conditional probabilities that an individual is in a
particular class of A and B, respectively, given that a
person is in a certain class of X. Equation (2) indicates
that, within a class of X, A and B are independent.
Purpose of Paper
Concept of LCA relatively straightforward—
create a variable to account for common
variance among observed variables
 Issues:

What is the new variable?
What do its classes mean?
Does it really tell us anything useful?
Statistical diagnostics don’t help us here. We
need substantive ones.
 Paper explores some of this type of
diagnostics

Data Sources

CED
2 week diaries
All expenditures
Small items and grocery expenditures
Used for CPI cost weights

CEQ
5 quarters (first for bounding) PV
All consumer expenditures
2 hours
Larger consumer items
Used for CPI cost weights
Three Examples
1985 CED Operational Test (micro level)
 3 treatments—specific, nonspecific, control
 800 households in each
 Latent response error measure of underreporting of grocery
expenditures using manifest performance indicators
 CEQ (1996-2001) (micro level)
 Only analyzed the 2nd wave
 43,000 completed 2nd wave interviews
 Latent response error measure of underreporting for 7
expenditure categories for purchasers using manifest
performance indicators
 CEQ (1996-2001) (micro level)
 Analyzed all four waves
 14,877 remained in sample throughout
 Latent response error measure of underreporting for almost 30
expenditure categories for all households (purchasers and
nonpurchasers) using manifest performance indicators and
indicators of pattern of wave nonresponse

Critical Assumption

Response errors in CE only come from
underreporting of expenditures and not
overreporting
Tedious
Time-consuming
Recall problems
Lack of knowledge
Methodological Issues
Weighted vs. unweighted
 Variances for complex sample design
vs. SRS
 Local vs. global maxima
 Sparse cells (too many manifest
variables)
 Restricted vs. unrestricted models
 Boundary problems (no overreporting)

1985 Diary Test

Manifest variables
Difference in first and second week grocery
expenditures
Difference in usual and average weekly grocery
expenditures
Amount of expenditure information collected by
recall
Respondent’s attitudes and behavior with respect
to diarykeeping

Latent variable
3 classes (low, moderate, high response error)
CEQ Micro-level Manifest Indicators
for First Study

Interview level indicators considered:
1.
2.
3.
4.
5.
6.
7.
Number of contacts
Ratio of respondents/household members
Missing income data
Type and frequency of records used
Length of interview
Ratio of expenditures in last month to
quarter
Combination of type of record and
interview length
Indicator Coding







#contacts (1=0-2; 2=3-5; 3=6+)
Resp/hh size (1= <.5; 2= .5+)
Income missing (1=present; 2=missing)
Records use (1=never; 2=single type or
sometimes; 3=multiple types and always)
Interview length (1= <45; 2=45-90; 3= 90+)
Month3 expn/all (1= <.25; 2= .25-.5; 3=
+.5)
Combined records and length (1= poor; 2=
fair; 3=good)
Latent Variables

Three-class latent variables (poor, fair,
good reporting) for
Kid’s Clothing
Women’s Clothing
Men’s Clothing
Furniture
Electricity
Minor Vehicle Repairs
Kitchen Accessories
Second CEQ Micro-level
Study





Based on results of first CEQ study, analysis
of purchasers and nonpurchasers together
Used Interviews 2-5 data.
Not limited to within-interview indicators
Developed model using all Interview 2
respondents
Latent variable is still intended to represent
quality of reporting
New Manifest Indicators

Overall Panel level indicators considered
1.
2.
3.
4.
5.
Number of completed interviews (1-4)
Attrition combined with # of complete interviews
Average number of commodity categories for
which CU had expenditure
Number of interviews the ratio of third month
expenditure to quarter was between .25 - .5
Panel averages of interview level indicators from
first CEQ study
Model Selection
Ran both ordered (fixed or restricted ordinal
constraints) latent class models and unordered.
Order was determined based on theoretical
relationship between values of indicators and
level of underreporting.
 Ran all combinations of indicators in groups of 3 &
4, using 3 or 4 category LC variable for each
commodity category & overall
 Multiple iterations to avoid local maxima
 Best model candidates were selected based on fit
 From those candidates, models selected based on
relationship of indicators to latent construct

Application of Model

For the final models for each commodity:
Each combination of indicators was assigned to a
latent class based on probability of being in that
class given the value of the indicators
Ran demographic analysis to identify
characteristics of members of each latent class
Expenditure means were found for each latent
class
Examined the pattern of mean expenditure and
the contribution of the latent variable in predicting
these expenditures
Second CEQ Micro-level Study–
Expenditure Categories


















Cable/satellite TV
Men’s apparel
Women’s apparel
Men’s clothing only
Women’s clothing only
Men’s accessories
Women’s accessories
Men’s shoes
Women’s shoes
Kid’s apparel
Kid’s clothing only
Kid’s Accessories
Kid’s shoes
Dental care
Drugs and medical supplies
Electricity
Gas (household)
Eye care

















Sports equipment
Televisions, video, & sound equip.
Vehicle service, major
Vehicle service, minor
Vehicle service, oil changes only
Vehicle expenses, other
Pets and pet supplies
Sports equipment
Trash collection
Televisions, video, & sound equip.
Vehicle service, major
Vehicle service, minor
Vehicle service, oil changes only
Vehicle expenses, other
Pets and pet supplies
Kitchen accessories
Other household items
Conclusions






When doing LCA for measuring response error, one cannot rely
on statistical diagnostics alone. Substantive diagnostics are
needed to judge the meaningfulness of the results.
Sometimes the models work and sometimes they don’t.
Unfortunately, this is likely to depend on the characteristic
you’re analyzing.
We need better manifest variables to explain more variance.
We have been unable to develop meaningful latent variables
with more than three or four categories, and, in some cases, we
could only identify two. LCA software really does work best
with large sample sizes.
Besides only defining a few latent classes, we certainly will not
progress beyond the most rudimentary ordinal rankings any
time soon.
LCA problems are likely to be multiplied many times for
response error measures for non-factual items such as attitudes
or opinions.
Contact Information
Clyde Tucker
Senior Survey Methodologist
OSMR
202-691-7371
[email protected]
www.bls.gov
www.bls.gov