Transcript Slides
Outline • National Assessment of Educational Progress (NAEP) • Multivariate Design Problem • Implications for analysis • Example with similar structure in Biostatistics NAEP • On-going surveys at national and state levels • 4th, 8th, and 12th grade students and their teachers • math, reading, writing • background demographic and educational environment questions Excellent web site • http://www.nces.ed.gov/nationsreportcard/ NAEP Mathematics • Mathematics – 5 domains/sub-scales/traits/latent proficiencies – Algebra – Geometry • Several hundred potential test questions NAEP Objectives and Constraints • Goal is population estimates • Individual students (and schools) are NOT rated based on NAEP • 45 minutes for cognitive questions (items) • 15 minutes for background questions/administration Matrix Sampling (1984) Algebra Student Q1 1 x 2 Q2 Geometry Q3 ... x x Q76 Q77 Q78 x x x ... x Model for Cognitive Data • Longitudinal data model – A : Student algebra proficiency – G : Student geometry proficiency • IRT (Item response theory) P Correct algebra answer A logit 1 item A item Design Issue • Fixed number of items per student – How many algebra items? – How many geometry items? • Obtain (equally) accurate population estimates of both algebra and geometry proficiencies Balanced Designs • Give each student approximately the same number of algebra and geometry items • Up to 5 or 6 sub-domains, so the number of items per sub-domain is very small • Extended collections of related items may make a balanced design infeasible Split Designs (symmetric) • Some students assigned only algebra items • Same number of other students assigned only geometry items • Remaining students are assigned equal number of algebra and geometry items Optimal Design and Estimation • The balanced design is optimal • Maximum likelihood estimation – The joint MLE for the algebra distribution and the geometry distribution is the same as the univariate MLE with the geometry and algebra proficiencies estimated separately • Balanced design – There is no gain from multivariate estimation – Estimates for individual student proficiencies are much improved by multivariate estimation • Split design – Multivariate estimation is much better than univariate – Multivariate estimation for the split design approaches balanced design efficiency as the proficiency correlation approaches 1 Bivariate outcomes (Jessica Mancuso) • Experimental biomarker for stroke patients – Measurement error – It can be applied to the infarct and non-infarct sides of the brain – Anticipated that the non-infarct side of the brain will be predictive of the infarct side – Evaluate an oral compound using the biomarker Study design • Placebo/drug in parallel blinded randomized groups • Measurements – Baseline – On-dosing measurements (longitudinal) – Measurements on the infarct and non-infarct sides of the brain (bivariate) Estimation • The primary goal is to estimate the treatment effect on the infarct side of the brain • What is the role of the measurements on the non-infarct side in the primary estimation? Depends on other information • If there is no effect (or a known effect) on the non-infarct side of the brain, the noninfarct data can improve estimation – Baseline non-infarct measurement may be very helpful – If the treatment does not effect the non-infarct side of the brain, the on-dosing measurement(s) are like covariates and may improve estimation Depends on design • Balanced design – Both sides of brain measured each time – No planned or unplanned missing measurements – On-dosing non-infarct measurements do not contribute to estimation of the drug effect on the infarct side (mostly true) – Lack of contribution despite improvement in estimation for individual patients • Split design – At some on-dosing times, the non-infarct measurement is available but the infarct is not available – The on-dosing non-infarct data may contribute substantially Summary • The use of multiple outcomes to improve inference is very complex • The fact that an outcome can be used to improve the estimation/prediction of another outcome at the level of an individual person is not sufficient