Transcript Make observations

SoTL Fellows 2005-2006
Project Design and Data
Collection Methods:
A quick and dirty introduction to
classroom research
Margaret Waterman
September 21, 2005
Where are we in
the Action Research Cycle?
Identify Innovation
Develop Action Plan
Interpret Data
Decide when
and how to
collect data
☻
Gather Data
Analyze Data
Design vs. Method

Design is about when and from whom you are
going to collect data so that the data can be
interpreted.
On Tuesday
With Colonel Mustard

Methods are about where and how you are
going to collect that data.
In the Library
With a candlestick
With an interview
Designs: Experimental



Experimental: strongest for drawing
inferences about cause and effect.
Control vs. experimental groups, randomly
assigned, one variable at a time
Diagram as follows:
O1 X1 O2
O1
O2
O = observation, X = treatment; Each row is a group
Experimental designs

Problems:

Not always possible in education to have
two or more matched groups,

Random assignment usually not feasible,

May not be suitable for small-scale projects

Usually not for classroom research
problems
Designs: Pre-Experimental

Weakest designs: Post test only
X1 O1
Slightly stronger: Post-test with comparison
group
X1 O1
O1

NOT RECOMMENDED
Pre-Experimental: Post Test Only
Problems:




Too many other reasons for observed conditions
Cannot draw valid inferences about causality
because you don’t know what pre-treatment
condition is.
In comparison group: may have differences in
incoming ability in the two groups
Good for: preliminary observation and development of
hypotheses, not showing cause and effect
Designs: Pre-Experimental

One group, Pre-test Post-test
O1 X1 O2

Uses: Can make some inferences about
cause and effect, but only when discuss
possible intervening variables.
Useful in classrooms when trying new
techniques or materials.

Pre-Experimental: Pre-test, Post-test

Problems:



Can’t discount all intervening variables: e.g.,
history, awareness of being in an
experiment, selection effects on group
membership.
Generalizability is limited
May sensitize students with the pre-test
Designs: Quasi-Experimental

Uses an experimental design, but groups are not
randomly assigned.
O1 X1 O2
O1
O2 Effect of treatment isolated
X1 O2 Use two more groups to
O2 isolate effect of pre-test.
Can make inferences with more confidence than
with one-group designs
Problem: intervening variables again.

Design: Time Series

One group, multiple observations before and after
treatment. Can be two groups or two treatments.
O1 O2 O3 O4 X1 O5 O6 O7 O8
OR
O1 O2 O3 O4 X1 O5 O6 O7 X2 O8 O9 O10
Look for BIG differences after treatment as compared
to before treatment.
Problem: The intervening variables
Linking Design and Methods
Suppose the design is one-group, pretest, post-test:
O1 X1 O2
How, where, and exactly when will the
data be gathered at the two points of
observation?
Action Research Often Uses These
Kinds of Methods to Gather Data
Observations
Surveys
Interviews
Artifact
Analysis
Methods of Gathering Data



Make observations

Structured observations

Unstructured observations
Ask questions

Surveys

Interviews
Examine and Score Artifacts

Tests, portfolios, treatment plans, student comments

Must design scoring guide
Structured Observations

Examples:
 Use a class map to keep track of who asks
questions, answers questions, makes comments.

Have a list of behaviors. Make simple hash
marks for each type of behavior when it occurs.

Make a list, from a recorded class discussion, of
the kinds of questions you asked.
Methods: Unstructured Observations

Examples

A journal that a teacher keeps to record personal
impressions of how a class is going.

A written remembrance of interactions with one or
a few students that are being tracked over time.

A written set of impressions made while watching
a videotape of a class.
Methods: Unstructured Observations

Think of these as similar to ethnographic “field
notes,” a place to record reactions, impressions,
planned actions, responses and thoughts.

Can be useful to see trends in how YOU are
responding to the innovation you are attempting.

Can be useful when studying one or a few
students, again, over time to see trends.
Methods of Gathering Data



Make observations

Structured observations

Unstructured observations
Ask questions

Surveys

Interviews
Examine and Score Artifacts

Tests, portfolios, treatment plans, student comments

Must design scoring guide
Methods: Questioning
(survey/interview)






Uses a prepared set of questions.
May have open-ended or close-ended
questions, or a combination.
Good for getting background info.
Can be used pre and post treatment.
Can address things that may not be visible,
such as attitudes, motivation.
Use with individuals or groups.
Methods: Questioning with Surveys

Examples:

A survey of attitudes toward science

Rankings of importance

Student ratings of instruction

May include open-ended questions, e.g., what
element of this course most helped you to learn?

May include some content, but if entirely content,
it’s a test and is an artifact of the course.
Sample from an Survey
Methods: Questioning with Interviews


Advantage over surveys: can ask follow-ups,
more personal contact.
Examples:




Sort readings by usefulness, telling why
Solve a genetics problem aloud, explaining
thinking
A focus group
Interview of team members in small groups
Sample from Interview
Methods of Gathering Data



Make observations

Structured observations

Unstructured observations
Ask questions

Surveys

Interviews
Examine and Score Artifacts

Tests, portfolios, treatment plans, student comments

Must design scoring guide
Method: Examine and Score Artifacts

Examples of artifacts:








Diagrams of cells before and after instruction
Wear on computer keys to see which are hit most
Answers to a test question
Portfolios
Term papers
Case analyses
Management plans
Treatment plans
Methods: Artifact Analysis

Decide what you kinds of materials you want to collect

Justify why the artifact you are choosing is a good choice
given your research question

Create a scoring rubric (guide) to assign points

Good for pre-post designs
A KEY TO SUCCESS:
Pilot Test Your Instruments



Give a small group of people (not in your
class if you can) your survey or interview
Collect sample artifacts to see if your grading
scheme works
Try out your observation scheme to see if it
needs to be tweaked.
Designs and Methods: Summary





Many of these approaches are familiar to we
who teach
Much of this is used in research in many
disciplines.
Talk about your design and instruments with
Teaching Associates, Fellows, other
colleagues.
Keep your data collection focused.
Try to keep the project reasonable for the
time you have available.
To create your own plans answer the
questions below:

From whom are you gathering data?
More than one class, subgroups?

When will you gather data?
All one term, over several courses, in three weeks?

How will you gather data?
Questions, artifacts, observations?

Where will you gather data?
Classroom, online forum, mailed survey, telephone?