Transcript Please cite as - University of Hawaii

Please cite as:
 Ortega, L. (2009). A sociology of replication and replicability in applied
linguistics. Paper presented in the invited colloquium “Encouraging
replication research in the field of AL and SLA,” Graeme Porte
convener. The American Association for Applied Linguistics Annual
Conference, Denver CO, March 23.
Copyright © Lourdes Ortega, 2009
A Sociology of Replication and
Replicability in Applied
Linguistics
Lourdes Ortega
University of Hawai‘i at Mānoa
AAAL colloquium, Graeme Porte Convener
Denver CO, March 23, 2009
Sociology of knowledge: “study of the
relationship between human thought
and the social context within which it
arises”
(Wikipedia)
Replicability: conditions that improve a
field’s ability to support replication
activities
(McCullough, 2007, inter alii)
In art,
as in science...
“Original”
1656
“Replica/tion”
1950s
Fidelity?
Value?
In
science,
as in art...
Fidelity
Verification
Value
Worthy Knowledge,
Worthy Purpose
Difference…?
Value…?
contested
understandings…
“how different/similar”?
Rosetta Stone
(c. 203 BC) in
British Museum
… these
Rosetta
Stone
bookends?
uncertain
understandings of
“replication”…
Polio & Gass (1997):
“many more studies than claim to be are
actually some type of replication […]
clearly, a fine line exists between
replication and extending research”
(p. 501)
Neuliep & Crandall (1993):
Everyone was wrong: There are lots of replications
out there. Journal of Social Behavior and Personality,
8(6), 1-8.
As much as 78% of studies were (conceptual)
replications that were not labeled by authors
as such.
Sociological perspective:
Norms for replication
Incentives for replicability
Replication,
the norm
“doing it again”
Based on Rosenthal (1991)
T
Y
P
E
of
R
E
P
L
I
C
A
T
I
O
N
“doing it right”
“contributing to a
valued dialogue”
Result of replication
Same
Exact
Conceptual,
etc
Value
Different
Supports the
theory (&
investigator)
Damages the
theory (&
impugns
investigator)
Internal validity,
results easy to
interpret, no
“novelty”
Extends the
theory (&
supports
investigator)
Limits the
theory (&
impugns the
investigator very
little)
External validity,
greater “novelty,”
any difference in
results difficult
to interpret
Replication as dialogue
1998, TQ
Original:
2001, Chapter
Replication +1:
Bardovi-Harlig Niezgoda &
& Dörnyei
Roever
Same:
Different:
2006, LL
Replication +1+
extension:
Schauer
RQs & focus, instrument, analyses
EFL sample from a
purposefully different
population
Validation of instrument
via L1 baseline & interviews
Longitudinal component
Verification of findings?
Pragmatic awareness of ESL vs. EFL learners
Barvodi-Harlig &
Dörnyei (1998)
Niezgoda &
Roever (2001)
ESL (n=173, Indiana, college):
pragmatic > grammatical
EFL (n=370, Hungary, coll & hs):
pragmatic < grammatical
Schauer (2006)
“ESL” (n=16, Germans in UK, college):
pragmatic > grammatical
(and over 9 months close to L1 baseline
EFL (n=17, Germany, college):
pragmatic < grammatical
ESL (n=48, Hawaii, college):
pragmatic > grammatical
EFL (n=124, Czech Rep, college):
pragmatic = grammatical
(& better than ESL sample in both)
Contributions to knowledge?
Pragmatic awareness & Context
Barvodi-Harlig &
Dörnyei (1998)
CONTEXT: a powerful
influence on L2 pragmatic
development
Niezgoda &
Roever (2001)
Schauer (2006)
CONTEXT: Always worthwhile to
apply longitudinal, developmental
lens
[context+proficiency?]
CONTEXT: powerful,
but not deterministic
[program evaluation to shed light
on optimal context+instruction design?]
Knowledge construction
as multi-study effort
“The thinking presented in a single
study may turn the movement of the
literature, but the results in a single
study are important primarily as
one contribution to a mosaic of
study effects.”
(Wilkinson & APA Task Force on Statistical Inference, 1999, p. 602)
“doing it
right”
it
“doing
The
more
polemic
a
research
area
is,
again”
the more important the choice
becomes between verificationist or
value-oriented and value-aware
approaches to replicating
“contributing to a valued
dialogue”
Yet… is the
replication norm
only an unrealistic
ideal?
“Economists treat replication the way
teenagers treat chastity – as an ideal
to be professed but not to be
practised”
Hamermesh (2007, p. 715)
Sociologists
…
Medical
researchers…
Educational
researchers
…
Replication, the
“incentives”
What things could be
done?
Graduate programs
Journal editors
Authors (replicated & replicating)
Things graduate training
programs could do
Make replication central to
graduate training experiences
cf. papers in this
colloquium by Abbuhl,
Fitzpatrick
Things journal editors
could do
Foster within-study replication
activities
To encourage within-study exact and
conceptual replication (or “systematic”
replication, Hendrick, 1991; also Hamermesh,
2007, pp. 731-732)
“multi-trait multi-method
validation
e.g., Long, Inagaki,
&
(Campbell & Fiske,Ortega
1959)(1998)
 “sensitivity analyses” (Saltelli et al., 2000)
Devote dedicated space to replication
articles
 Several journals across fields have featured
special replication sections at various times
SSLA, Valdman (1993)
LT, Porte (2008)
Commission replication by
senior researchers
“… journal editors [could] commission leading senior
empirical researchers to undertake a replication study
of a paper of their choice, one that had previously
been published in the journal. If editors of each of the
three leading general journals commissioned two
replication studies per year, with publication
possible
in our
guaranteed subject to refereeing
(NOT by
the author
of the original study) to assure some field?
minimum quality
level, more replications would be undertaken. Original
authors would be expected to write a short reply to
the final version of the replication study”
(Hamermesh, 2007, p 726)
Make data-sharing mandatory
for publication in a journal
“Data distributed with an article should become
accessible to the scientific community without
having to obtain permission from the author for
each use. Having to obtain permission from an
author to read a published article, such as by
agreeing ex ante not to criticize it in print, is so
obviously unacceptable it no longer occurs. The
same should be true for data. Science requires
the transmission of information through public
means, not private agreements.”
(Gary King, 2007, p. 177)
 “information about replication [should be]
part of the price of admission to competitive
journals, rather than as an act of individual
honor or graciousness following
possible publication”
in our
field? should be
 Mandatory data- and code-sharing
“akin to the expectation of a full citation for
all references”
(Freese, 2007, p. 155 & 156)
Things (replicated and replicating)
authors could do
Design studies with the
forethought of replication
“…think at the onset of their work about
data sharing and opportunities for
replication.”
Schneider (2004, p. 1477)
How often/regularly do
we do this in our
field?
Enhance the quality of our
studies with replication elements
include by design:
 within-study exact and conceptual
replication elements (Hendrick, 1991)
“multi-trait multi-method validation
(Campbell & Fiske, 1959)
 “sensitivity analyses” (Saltelli et al., 2000)
Make your instruments/data/code
publicly available (don’t wait to be asked!)
 In the pre-Internet era, complaints about
journal limited space may have justified lower
reporting standards and lower levels of
methodological transparency (cf. Polio & Gass, 1997)
 In our Internet-dominated era, full public
availability of instruments/data/code is made
easier through personal webpages, professional
archiving, and public repositories
(Freese, 2007; King, 2007)
Make your instruments/data/code
publicly available (don’t wait to be asked!)
Skehan’s (1989)
longitudinal follow-up
study of Wells’ (1985)
Bristol project
Don’t wait to be asked!
CHILDES (McWhinney, 2000)
FLLOC & SPLLOC (Myles,
2008; Mitchell et al., 2008)
Don’t wait to be asked!
IRIS: Marsden & Mackey’s
UK-US collaborative grant
proposal to create a Digital
Database of Instruments for
Research Into SecondLanguages
Overcome our own prejudices
about data sharing:
 The freeloading problem: Why should I go to the effort to
obtain grants and collect my own data if I am then required
to share my data with others?
 The ‘‘I-might-be-scooped’’ problem: Not only will there be
freeloaders, but they might become famous at my expense
by publishing key results before I am able to.
 […]
 Too much work: The extra work for authors […] would
be onerous.
(Firebaugh, 2007, p. 207)
Be mindful of standards of
collegiality
e.g., Contact the to-be-replicated researcher
when planning a replication, for a collaborative
(not oppositional) footing if possible.
e.g., Respond (positively or negatively but promptly and
with courtesy)
to requests for
instrument/code/data sharing.
e.g., Make collegial and measured discourse
choices in published exchanges (replicating or
replicated)
Conclusion:
Where do we
go from here?
The promised benefits of
heightened replication
activity are many…
…but will they be
fulfilled?
Fidelity
Verification
Value
Recognizing
knowledge & purpose,
contributing to
dialogue
Thank You
[email protected]
References:

Bardovi-Harlig, K., & Dörnyei, Z. (1998). Do Language Learners Recognize
Pragmatic Violations? Pragmatic Versus Grammatical Awareness in Instructed L2
Learning. TESOL Quarterly, 32, 233-262.

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by
the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.

Child Language Data Exchange System (CHILDES): http://childes.psy.cmu.edu/

Firebaugh, G. (2007). Replication Data Sets and Favored-Hypothesis Bias: Comment
on Jeremy Freese (2007) and Gary King (2007). Sociological Methods & Research, 36,
200-209.
Freese, J. (2007). Replication Standards for Quantitative Social Science. Sociological
Methods & Research, 36, 153-172 .


French Learner Language Oral Corpora (FLLOC):
http://www.flloc.soton.ac.uk/index.php

Hamermesh, D. S. (2007). Viewpoint: Replication in economics. Canadian Journal of
Economics, 40, 715-733.

Hendrick, C. (1991). Replications, strict replications, and conceptual replications: Are
they important? In J. W. Neuliep (Ed.), Replication research in the social sciences (pp.
41-49). Newbury Park, CA: Sage.
References:

King, G. (2007). An Introduction to the Dataverse Network as an Infrastructure for
Data Sharing. Sociological Methods & Research, 36, 173-199.

Language Teaching Review Panel. (2008). Replication studies in language learning
and teaching: Questions and answers. Language Teaching, 41, 4-14.

Long, M. H., Inagaki, S., and Ortega, L. (1998) The role of implicit negative feedback
in SLA: Models and recasts in Japanese and Spanish. Modern Language Journal, 82,
357–71.

MacWhinney, B. (2000). The CHILDES Project: Tools for Analyzing Talk. Volume 1:
Transcription format and programs. Volume 2: The database. Mahwah, NJ: Lawrence
Erlbaum Associates.

McCullough, B. D. (2007). Got replicability? The Journal of Money, Credit, and
Banking archive. Eco Journal Watch, 4, 326-337.

Mitchell, R., Dominguez, L., Arche, M. J., Myles, F. and Marsden, E. 2008. “SPLLOC:
A new database for Spanish second language acquisition research.” EuroSLA Yearbook
8, 287-304. Muñoz, C. (Ed.). (2006). Age and the rate of foreign language learning.
Clevedon, UK: Multilingual Matters.
References:

Myles, F.(2008). Investigating learner language development with electronic
longitudinal corpora: Theoretical and methodological issues. In Ortega, L. & Byrnes,
H. (Ed.), The longitudinal study of advanced L2 capacities (pp. 58-72). New York:
Routledge.

Neuliep, J. W., & Crandall, R. (1993). Everyone was wrong: There are lots of
replications out there. Journal of Social Behavior and Personality, 8(6), 1-8.

Niezgoda, K., & Roever, C. (2001). Pragmatic and grammatical awareness. In K. R.
Rose & G. Kasper (Eds.), Pragmatics in language teaching (pp. 63–79). New York:
Cambridge University Press.

Polio, C., & Gass, S. M. (1997). Replication and reporting: A commentary. Studies in
Second Language Acquisition, 19, 499-508.

Rosenthal, R. (1991). Replication in behavioral research. In J. W. Neuliep (Ed.),
Replication research in the social sciences (pp. 1-30). Newbury Park, CA: Sage.

Rosenthal, R. (1966). Experimenter effects in behavioral research. New York:
Appleton-Century Crofts.

Saltelli, A, Chan, K., & Scott, E. M. (Eds.).(2000). Sensitivity Analysis. Malden, MA:
Wiley.
References:

Schauer, G. A. (2006). Pragmatic Awareness in ESL and EFL Contexts: Contrast and
Development. Language Learning, 56, 269–318.

Schneider, B. (2004). Building a scientific community: The need for replication.
Teachers College Record, 106, 1471–1483.

Skehan, P. (1989). The relationship between native and foreign language learning
ability: Educational and linguistic factors. In H. W. Dechert (Ed.), Current trends in
European second language acquisition research (pp. 83-106). Clevendon, UK: Multilingual
Matters.

Spanish Learner Language Oral Corpus (SPLLOC):
http://www.splloc.soton.ac.uk/index.html

Valdman, A. (1993). Replication study (editorial introduction). Studies in Second
Language Acquisition, 15, 505.

Valdman, A. (1997). Editorial.Studies in Second Language Acquisition, 19, 67.

Wells, G. (1985) Language development in the pre-school years. Cambridge, UK:
Cambridge University Press.
Image copyright credits:
 Velazquez’s Meninas:
http://www.richardsonandgray.com/Theme%20Tours/
Art%20Tours/Spanish%20Treasure.htm
 Picasso’s Meninas:
http://www.flickr.com/photos/32357038@N08/328636
5519/
 Rosetta Stone bookends:
http://www.egyptianmarketplace.com/index.php?act=vi
ewProd&productId=1490
Please cite as:
 Ortega, L. (2009). A sociology of replication and replicability in applied
linguistics. Paper presented in the invited colloquium “Encouraging
replication research in the field of AL and SLA,” Graeme Porte
convener. The American Association for Applied Linguistics Annual
Conference, Denver CO, March 23.
Copyright © Lourdes Ortega, 2009