Transcript PRESENTATION NAME

Though this be madness, there is a method in it:
Using methodological exemplars to improve pedagogical research
Jordan D. Troisi
Widener University
Andrew N. Christopher
Albion College
Presented at the 12th Annual Society for the Teaching of Psychology Best Practices Conference
October 11, 2013
“Though this be madness, there is a method in it”
• A brief summary of the beginning of “Hamlet:”
– Hamlet is visited by his father’s ghost (The King)
– The ghost says that his own brother (Claudius) has killed him
– 1-2 months later, Claudius has married Gertrude, Hamlet’s
mother
– The ghost says “kill Claudius”
– Hamlet says “sure,” and that he’s going to act mad for awhile,
presumably to cover his tracks
• Perhaps there is a method in it?
“Though this be madness, there is a method in it”
• Hamlet is acting(?) mad, and a bunch of characters develop
hypotheses as to why
– Gertrude hypothesizes he’s mad because his father has
died and she married his brother in 1-2 months  FAIR
– Claudius hypothesizes he’s acting mad because he wants
to steal the throne he has recently assumed  FAIR
– Polonius hypothesizes he’s mad in love with his daughter
Ophelia and that he just wants to sleep with her  FAIR
• All valid hypotheses, which they each test (poorly)
• But they ignore a crucial confound. . .
– A freaking ghost is telling Hamlet what to do!
Guidance in pedagogical research
from Hamlet
• There’s a lot of madness out there, and it’s often
hard to understand
• Luckily, psychologists are trained to determine the
reasons why behavior occurs
– Also luckily, such behavior is rarely guided by ghosts
• What happens in the classroom may seem like
madness, but there’s a method in it
– We need the methodological tools to make sense of the
madness
Overview
• Scholarly teaching, the scholarship of
teaching and learning (SoTL), and the
learning sciences
• Issues of control in pedagogical research
• Effective methodology exemplars from SoTL
and the learning sciences
• Take-home messages for effective
pedagogical research
SoTL & Learning Sciences
Daniel & Chew (in press)
SoTL
• Extension of higher
education
• Evaluates specific
pedagogical methods
• Application-driven
• Constructs may lack
theoretical precision
Learning Sciences
• Extension of “basic”
research in psychology
• Findings may not translate
across teaching contexts
• Theory-driven
• Construct
operationalizations tend to
be widely-accepted
SoTL & Learning Sciences
Daniel & Chew (in press)
SoTL
• Predicting the weather
Learning Sciences
• Knowing the laws of
thermodynamics
Research on Note-Taking
• Williams and Eggert (2002) examined the notes of
125 students in an undergraduate human
development course
• Students took notes in a 125-page study guide on
both their:
• Class notes
• Textbook readings
Research on Note-Taking
• Three blind raters coded these notes for:
1. Completeness
2. Length
3. Accuracy
Research on Note-Taking
• Completeness, length, and accuracy used to predict:
1. Brief essay quizzes linked to the notes
2. Exam questions from text only
3. Exam questions from lecture only
4. Exam questions from both text and lecture
Research on Note-Taking
• Peverly et al. (2013) randomly assigned 200
undergraduates in an educational psychology
course to either:
• Outline provided
• No outline provided
while taking notes on a 23 min video-taped lecture
Research on Note-Taking
• Predictors
1. Handwriting speed
• Wrote letters of alphabet horizontally in
orders as many times as possible in 45 sec
2. Language comprehension
• 7 reading passages with questions about each
3. Verbal working memory
• Listening span test
• Attention
Research on Note-Taking
• Dependent Variables
1. Quality of notes
2. Written summary of the lecture
Research on Note-Taking
• Used structural equation modeling to assess
predictors’ relationships with quality of notes and
written summary of the lecture
Overview
• Scholarly teaching, the scholarship of
teaching and learning (SoTL), and the
learning sciences
• Issues of control in pedagogical research
• Effective methodology exemplars from SoTL
and the learning sciences
• Take-home messages for effective
pedagogical research
SoTL: The Early Days
• Focused on student attitudes (i.e., Did students like
it?)
– e.g., Christopher and colleagues (2004)
• Wesp and Meile (2008, p. 362)
– “It appears that student opinions about the effectiveness
of teaching techniques are inaccurate.”
– “Researchers should prefer direct measures (of student
learning) because they provide a more accurate
assessment of pedagogical effectiveness….”
“Learning Might not Equal Liking”
• Attitudes about courses and topics may not equate
to learning, and vice versa
• After a course on research and statistics, students
reported increases in knowledge but no changes in
favorable attitudes toward the subject (Sizemore &
Lewandowski, 2009)
SoTL: The Early Days = A Vicious Cycle
• Psychologists are control freaks
“Correlation does not equal causation”
equates to
“Correlational research is not as valuable as
experimental research”
SoTL: Why is it so hard to do?
• Special challenges in doing SoTL:
1.
2.
3.
4.
How often you teach a particular class
Number of students in the class
Time of day a class (or sections of a class) is offered
Lack of control relative to lab-based research
Overcoming the “Control” Issue in SoTL
• As psychologists, we love control
• As teachers, we want to do whatever we can to
maximize student learning and sometimes produce
attitude change
• But, classrooms are very “noisy” places
Finding Appropriate Comparisons
• Smith (2008)
– Use a previous class
• Compare and (ideally) statistically control for ACT/SAT
scores
– Use an in-tact class
• Divide the class into 2 “equal” groups
• Group 1 attends the first half of class; Group 2 attends
the second half of class
• Use your intervention on one of the 2 groups
– Measure outcome variable on the same day
More Methods to Use Within 1 Class
• Bartsch et al. (2008)
– Typical pre/post contains same questions
– Using different tests at pre and post =
instrumentation confound
– Focus on using a 1-group pre/post with alternate
forms
1-Group Pre/Post with Alternate Forms
• Prepare 2 versions of an assessment
instrument (i.e., Version A & Version B)
• Give half the students Version A at pretest
and Version B at posttest
– Switch for the other half of the students
1-Group Pre/Post with Alternate Forms
• 2 (Version A first or Version B first) x 2 (pre
and post) mixed ANOVA
• Allows us to learn if:
– Groups were equivalent to start with
– Version A was equivalent to Version B
– The intervention does or does not facilitate
student learning
1-Group Pre/Post with Alternate Forms
• Use random assignment to:
– Form your 2 groups of students
– Put questions on the 2 versions of the
assessment instrument
Overview
• Scholarly teaching, the scholarship of
teaching and learning (SoTL), and the
learning sciences
• Issues of control in pedagogical research
• Effective methodology exemplars from SoTL
and the learning sciences
• Take-home messages for effective
pedagogical research
Rapport, Student Motivation, and
Course Attitudes
• We’d all like good rapport, motivated students, and
favorable attitudes toward our courses
• Legg and Wilson (2009) examined how sending a
welcome email to students before a course began
could create these outcomes
• Building rapport before the first day of class!
Rapport, Student Motivation, and
Course Attitudes
• Half of an Introduction to Psychology course’s
students (n = 66) received a welcome email one
week prior to the beginning of the semester
– Randomly assigned, instructor blind to participant
condition
• Surveyed rapport, student motivation, and course
attitudes at:
– End of 1st class period
– At mid-term
– After the final exam
Rapport, Student Motivation, and
Course Attitudes
• Most favorable ratings for email group during the
first class period
• Fewer effects for subsequent measurement periods
– Some interactions with gender of student
• Students in the email group were less likely to drop
the course
Kudos to Legg and Wilson (2009)!
• A welcome email is a low-tech, time-efficient way to
contact students and start building rapport and
motivation
• Authors randomly assigned students within one class
and were blind to which students received the email
• Measured multiple, relevant DVs, at 3 time periods
Student Attitude Change in a
Prejudice Course
• Reducing prejudice is often a goal of psychology
courses
• Reducing prejudice should be assessed over time, as
done superbly by Kernahan and Davis (2010)
– The utility of the classroom for ideas in basic research
Student Attitude Change in a
Prejudice Course
• Compared a Psychology of Prejudice and Racism
course & a Statistics course
• Utilized a pretest-posttest design with a control
group, plus a one-year follow-up
• Design allows for examination and comparison of
attitude changes from beginning to end of
semester, and beyond
Student Attitude Change in a
Prejudice Course
• Results showed that by the end of the semester, those
in the Prejudice and Racism course showed greater:
– Awareness of white privilege, white guilt, noticing of racism,
and responsibility for taking action
• At one year follow-up, some effects:
– Plateaued (e.g., awareness of white privilege)
– Waned (e.g., responsibility for taking action)
– Increased (e.g., comfortability in mixed-race interactions)
Kudos to Kernahan and Davis (2010)!
• Comparison between a group with an anticipated
effect and a control group
• Effects examined across 3 time periods, including
one year later
• High external validity, and the target outcome of
interest for basic and pedagogical research!
Learning or Living-Learning Communities
• Designed to create a coherent educational
experience for a group of students centered on a
particular topic
• Can improve student performance and retention
while fostering relationships
• Buch and Spaulding (2008) spectacularly examined
these important outcomes utilizing a longitudinal
design with a matched comparison group
Learning or Living-Learning Communities
• Compared GPA, retention, co-curricular
involvement in psychology courses/activities over 7
semesters of college
• Utilized a longitudinal design with a control course
with students matched (n total = 40) on important
variables (SAT score, gender, ethnicity, generation
status)
Learning or Living-Learning Communities
• Cumulative GPA better in the Learning Community
group during the 1st, 2nd, and 4th semester of college
• Retention at the school and progress within the
major were better among Learning Community
group (some significant findings, some nonsignificant but in the expected direction)
• Co-curricular involvement higher in the Learning
Community group
– Involvement in psychology club, internship, research
Kudos to Buch and Spaulding (2008)!
• Longitudinal design (see also Buch & Spaulding,
2011), with a comparison group matched on
important variables
• On a hot topic in higher education
•
•
•
•
•
7
Semesters
Of
Data
!
Laptop Multitasking and Performance
• Multitasking divides attention and can lead to poor
memory of course material and poor performance
• Laptops in the classroom can provide distractions
for the users and others in view of laptop screens
• Sana and colleagues (2013) put distractions from
laptop use to the test (literally!)
Laptop Multitasking and Performance
• Two studies on how well students retained information
during lectures for a multiple choice quiz
• One study examined student performance when they
were randomly assigned to multitask during the lecture
(given some online tasks to perform)
• A second study examined student performance when
they were in view of confederates’ laptop screens on
which they were multitasking
Laptop Multitasking and Performance
• In the first study, students assigned to multitask
performed worse on a quiz of the information of
the lecture (with no difference between simple factbased items or application items)
• In the second study, students who were in view of a
research confederate who was multitasking
performed worse on the quiz
Kudos to Sana and Colleagues (2013)!
• Examining a topic extremely relevant to teaching
and learning
• Including two studies, one on distraction from the
self and one on distraction from others
• Using an experimental design that simulated
classroom experiences
– High external validity and mundane realism
Time-of-Day Preference and
Grade-Point-Average (GPA)
• Preckel et al. (2013) examined whether high school
students’ (N = 272) time-of-day preference was
predictive of their GPAs in a variety of subjects
• At the college level, we discuss the “best” times to
offer classes from a students perspective
Time-of-Day Preference and
Grade-Point-Average (GPA)
• Preckel et al. (2013) statistically controlled for
variables suggested to be predictive of academic
performance (e.g., conscientiousness, cognitive
ability, achievement motivation, gender)
• Included measures of other-report as well as selfreport
Time-of-Day Preference and
Grade-Point-Average (GPA)
• Entered time-of-day preference on the last step of
three hierarchical linear regressions
• Used three criterion:
1. overall GPA
2. math-science GPA
3. language GPA
Time-of-Day Preference and
Grade-Point-Average (GPA)
• Found that students with an evening preference
had a lower overall GPA, lower math-science GPA,
and language GPA than students with a morning
preference
Time-of-Day Preference and
Grade-Point-Average (GPA)
• Use of statistical control, a commonly-used
technique in personality research
• Use of multiple predictors, including self- and otherreport data
Academic Dishonesty and
Narcissism
• Much has been made about the “rise of narcissism”
in Western society during the past 2-3 decades
• Teachers have lamented (likely since the beginning
of time) academic dishonesty among students
Academic Dishonesty and
Narcissism
• Brunnell et al. (2011) randomly assigned 199
Introductory Psychology students to one of
two questionnaire conditions:
– Questions referred to the Self
– Questions referred to the Typical Student on
Campus
Academic Dishonesty and
Narcissism
• Questions referred to:
1. Guilt experienced for cheating if:
• Exam was overly difficult
• Classmates did not help them study
• Friends pressured them to cheat
2. Prevalence of Academic Dishonesty
• The number of times they (others) cheated on a test in
the past 12 months
• Predicted how many times the typical student on
campus would cheat on a test
Academic Dishonesty and
Narcissism
• 40-item Narcissism Personality Inventory
(Raskin & Terry, 1988)
– Contains 3 subscales
1. Power
2. Exhibitionism
3. Special Person
Academic Dishonesty and
Narcissism
• Low exhibitionists displayed less guilt when
they cheated than did high exhibitionists; no
differences were found when judging others
who cheated
• High exhibitionists reported more academic
dishonesty when judging themselves than
others; no differences were found when
judging others
Academic Dishonesty and
Narcissism
• Within the self condition, mediational
analyses revealed that the relationship
between exhibitionism and reported
academic dishonesty was fully mediated by
guilt
Academic Dishonesty and
Narcissism
• Use of both experimental and
nonexperimental measures
• Analysis of narcissism measure at both the
factor and subscale levels
• Examined interactive effects of experimental
condition and narcissism
• Mediational analyses to hint at the
underlying mechanism
Overview
• Scholarly teaching, the scholarship of
teaching and learning (SoTL), and the
learning sciences
• Issues of control in pedagogical research
• Effective methodology exemplars from SoTL
and the learning sciences
• Take-home messages for effective
pedagogical research
A happy medium?
• SoTL and the learning sciences have
somewhat different approaches to studying
similar outcomes
• Effective pedagogical research utilizes the
strengths of both areas
Highlighted strengths of my research
(as a gracious author)
• Strengths highlighted in reviews of my work:
–
–
–
–
–
–
–
Novelty of topic and research design
Building in theory to support findings in an intuitive way
Meaningful, stringent control/comparison groups
Using established materials from basic and pedagogical research
Large-scale studies
Pressing, relevant, generalizable, and current topics
Clear, logical writing
“Though this be madness, there is a method in it”
• Instead of ghosts of kings past, we have ghosts in
the form of confounds and noise
• But fear not! “Though this may be madness, there is
a method in it”
– We just need to use the right tools to separate the
method from the madness
(or the meaningful effects
from the confounds and noise)
Thank you for your attention!
Questions please!