Transcript Linguistic indicators of L2 proficiency levels Some conceptual

L2 communicative levels meet
linguistic profiling
Plenary paper presented at the Estonian Association of
Applied Linguistics Conference, Tallinn, Estonia,
24-25 April 2008
Florencia Franceschina
[email protected]
1. Introduction
Question 1
What stages do L2 learners go through as
they develop their L2 knowledge and skills?
Question 2
What are the linguistic indicators that a
learner is at a particular stage on his/her way
to becoming a proficient L2 speaker?
Increasing L2 proficiency can be thought of
in terms of:
a) Increasing ability to do things with language
(eg: write notes, ask the time, talk about
your family, understand announcements at a
railway station, etc.)
b) Increasing linguistic knowledge (eg: larger
vocabulary, greater range of known syntactic
structures, higher level of grammatical
accuracy, etc.)
c) Increasing ability to use linguistic knowledge
in real time (eg: fluent spontaneous speech,
good listening or reading comprehension,
etc.)
L2 proficiency scales
L2 proficiency scales used in language
teaching and testing tend to focus on a
mixture of these levels.
The original CEFR scales have little
information about linguistic knowledge typical
of each level (among other reasons, because
they have been designed to be languageneutral).
CEFR global scale
C2
• Can understand with ease virtually everything heard or read.
• Can summarise information from different spoken and written sources, reconstructing
arguments and accounts in a coherent presentation.
• Can express him/herself spontaneously, very fluently and precisely, differentiating
finer shades of meaning even in more complex situations.
C1
…
B2
B1
…
…
A2
…
A1
• Can understand and use familiar everyday expressions and very basic phrases
aimed at the satisfaction of needs of a concrete type.
• Can introduce him/herself and others.
• Can ask and answer questions about personal details such as where he/she lives,
people he/she knows and things he/she has.
• Can interact in a simple way provided the other person talks slowly and clearly and is
prepared to help.
CEFR general linguistic range
illustrative scale
C2
…
C1
…
B2
B1
…
…
A2
• Can produce brief everyday expressions in order to satisfy simple needs of
a concrete type: personal details, daily routines, wants and needs, requests
for information.
• Can use basic sentence patterns and communicate with memorised
phrases, groups of a few words and formulae about themselves and other
people, what they do, places, possessions etc. Has a limited repertoire of
short memorised phrases covering predictable survival situations; frequent
breakdowns and misunderstandings occur in non-routine situations.
A1
CEFR grammatical accuracy
illustrative scale
C2
…
C1
…
B2
…
B1
• Uses reasonably accurately a repertoire of frequently used ‘routines’ and patterns
associated with more predictable situations.
A2
• Uses some simple structures correctly, but still systematically makes basic mistakes
– for example tends to mix up tenses and forget to mark agreement; nevertheless, it is
usually clear what he/she is trying to say.
A1
Finnish National Curriculum
scales for L2s/FLs
C1.1
…
B2.2
…
B2.1
…
B1.2
B1.1
…
…
A2.2
…
A2.1 • Can manage in the most routine everyday situations in writing.
• Can write brief, simple messages (personal letters, notes), which are related to everyday needs,
and simple, enumerated descriptions of very familiar topics (real or imaginary people, events,
personal or family plans).
• Can use concrete vocabulary related to basic needs, basic tenses and co ordinate sentences
joined by simple connectors (and, but).
• Can write the most simple words and structures with reasonable accuracy, but makes frequent
basic errors (tenses, inflection) and uses many awkward expressions in free writing.
A1.3
…
A1.2
…
A1.1
…
Our challenge
To find reliable linguistic indicators that
a learner is at a particular stage on
his/her way to becoming a proficient L2
speaker when proficiency level has
been established using communicative
scales such as the CEFR scales.
This presentation
• Some answers to Q1 and Q2 we can glean
from the existing literature
• How CEFLING are going about trying to find
full answers to these questions
– Selection and management of relevant datasets
(learner corpora)
– Methodology for tracking linguistic indicators in the
data (DEMfad)
– General methodological suggestions
2. What we already know
SLA generalizations
about L2 development
1. Factors such as type and amount of L2
instruction, type and amount of naturalistic
exposure to the L2, language aptitude, age, L1
background, among others, are likely to have
an effect on the rate of L2 development
but do not seem to affect the route of L2
development in a dramatic way.
Dulay and Burt (1974)
*Dulay and Burt (1974)
** Bailey et al. (1974)
Dulay and Burt (1973)
Tino
Marcello
Andrea
Ilhami
Mahmut
Berta
Mari
Alfonso
Nora
Lavinia
Abdelmalek
Mohamed
Subject
Zahra
Fernando
Leo
Madan
Fatima
Ergun
Cevdet
Ravinder
Allwood (1993)
MLU per subject in Cycle 1 (blue) and Cylce 3 (red)
18
16
14
12
10
8
6
4
2
0
S1
MLU
Banerjee et al. (2007)
Mean lexical density - Task 2
Mean lexical density score
0.85
0.8
0.75
0.7
L1 Chinese
L1 Spanish
0.65
0.6
0.55
0.5
Band 3
Band 4
Band 5
Band 6
Band level
Band 7
Band 8
TLU score
TLU: SV agreement on copula - Tasks 1&2
1.00
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
L1 Chinese
L1 Spanish
Level 3
Level 4
Level 5
Level 6
Band level
Level 7
Level 8
TLU score
TLU: SV agreement on main V - Tasks 1&2
1.00
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
L1 Chinese
L1 Spanish
Level 3
Level 4
Level 5
Level 6
Band level
Level 7
Level 8
TLU score
TLU: Passives - Tasks 1&2
1.00
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
L1 Chinese
L1 Spanish
Level 3
Level 4
Level 5
Level 6
Band level
Level 7
Level 8
2. There is a set of features F1 that is used
productively by learners in their early stages of
development (e.g., case contrasts in English
pronouns) and there is another set F2
comprising features that appear later in
development (e.g., passivization).
There is some relevant evidence for this, but the
information is complex and no clear detailed
picture has emerged yet. This is part of what we
are trying to sort out.
3. What linguistic indicators of L2
proficiency levels have been
investigated so far? And how?
Linguistic indicators
from the SLA literature
Examples of existing surveys or collections of
papers on L2 development, often with a
predominance of L2 English data
Dulay and Burt (1982)
Felix (1980)
Harley et al. (1990)
Hatch (1974)
Hatch (1978)
Hawkins (2001)
Klein (1986)
Perdue (1993)
White (2003)
Examples of surveys exclusively on the L2
development of languages other than
English:
L2 French: Carlo et al. (2006)
L2 Italian: Giacalone Ramat (2003)
L2 Spanish: Montrul (2004)
Type of indicators discussed
in the existing SLA literature
• Semantic categories (Perdue, 1993)
• Morphosyntactic categories (Dulay and
Burt, 1982)
• Functional grammar categories
(Giacalone Ramat, 2003; Carlo et al.,
2006)
• Generative grammar categories
(Hawkins, 2001)
Examples of indicators of L2 proficiency
level from the SLA literature
Perdue (1993)
• Word formation
–
–
–
–
–
N-N word formation
Derivation and word formation
Connective elements in N-N compositions
Kingship reference
Reference to possession
• Temporality
• Reference to space
Examples of indicators of L2 proficiency
level from the SLA literature
•
Giacalone Ramat (2003)
•
Noun morphology
– Gender
– Number
– Agreement
• Definite article
• Indefinite article
• NP-internal adjectives
• Predicative adjectives and
past participles
• 3PS pronouns
• Clitics
•
•
•
•
The verb
– Verb morphology and verb
placement
– Tense
– Mood
– Aspect
– Voice
Simple phrases
– Negation
– Adverbs
Subordination and coordination
– Adverbials
– Completives
– Relatives
Text-level phenomena
– Anaphoric reference
– Connectives
Marked word order
Examples of indicators of L2 proficiency
level from the SLA literature
Hawkins (2001)
• Grammatical morphemes
• Negation
– Sentential negation
– Negation and verb movement
• Word order
– Location of verbs
– Basic word order
• Questions
• Relative clauses
• Unaccusative verb constructions
• Null subjects and objects
• Articles
• Number
• Wh-movement
• Anaphors
Examples of indicators of L2 proficiency
level from the SLA literature
Carlo et al (2006)
• NP
– Pronouns
– N agreement
– Clitics
• Negation
– Anaphoric negation
– Negation in verbless
constituents
– Pre-verbal negation
– Post-verbal negation
• Mood and modality
• VP
– Finiteness
– SV agreement
– Temporal marking
• From simple to complex
constructions
– Early morphological inflection
– Subordination
Examples of indicators of L2 proficiency
level from the SLA literature
Dulay and Burt (1982)
•
•
•
•
•
•
•
•
•
•
•
Pronoun case
Article the, a
Progressive –ing
Contractible copula ‘s
Regular short plural -s
Contractible auxiliary ‘s
Regular past –ed
Irregular past
Long regular plural –es
Possessive ‘s
3PS -s
Type of indicators mentioned
by policy makers
Suggested contents for the reference level descriptions (RLDs) by the
Language Policy Division, DG IV, Council of Europe (November 2005),
as minimum common features of RLDs for specific languages:
• inventories of the linguistic realisation of:
– general notions (e.g., existence, quantity, space, time, qualities of things
and people, logical relations)
– acts of discourse or functions (e.g., imparting and seeking information,
expressing and finding out attitudes, deciding on and managing course of
action, socialising, structuring discourse, assuring and repairing
communication)
– specific notions*
– lexical elements*
– morphosyntactic elements*
• indication as to whether the proposed forms should be known for reception
only or also for production
*No specific examples of these categories have been provided in the ‘Guide for production of RLDs’ (Nov 2005).
Other indicators are being
explored by:
• SLA-informed linguistic profiling systems. Eg:
– Rapid Profile (Pienemann, Kessler, Lenzing and Hoffmann)
– Direkt Profil (Grandfeldt, Kostanidov, L. Persson, E. Persson
and Nugues)
• LT-informed NLP systems. Eg:
– E-rater (ETS)
• Corpus linguistics error tagging systems. Eg:
– Cambridge Learner Corpus’ (Nichols, 2003)
– Chinese Learner English Corpus’ (Gui and Yang, 2003)
– UC Louvain’s (Dagneaux et al., 1998)
– FALKO’s (Lüdeling et al., 2005)
Summary
We can make a start at answering Q1
and Q2 based on the existing literature.
However, the generalizations we can
make based on this body of knowledge
are still too vague for our purposes, as
they don’t allow us to establish precise
links between communicative
competence levels and linguistic
indicators of those levels.
4. Tracing DEMfad
Method for the empirical description of
linguistic indicators of development
We have adopted a method of
processing the empirical data from the
literature and from our own studies that
consists of 3 main elements:
4.1. Principled selection of D
4.2. DEMfad
4.3. Operators , >> and >
4.1. Principled section of
linguistic domains (D)
The selection of D is inevitably theoryspecific. The Ds selected will depend on
one’s chosen:
• Theory of language
• Theory of learning
There are also likely to be practical
considerations that influence our choice of D
that are not related to these theoretical
assumptions.
4.2. DEMfad
D
D=domain
f
E
a
E=emergence
M
M=mastery
f=frequency
d
a=accuracy
d=distribution
frequency
accuracy
distribution
4.3. Operators
The following operators describe the possible
relationships among relevant phenomena in
language development:
• X >> Y = X takes place before Y
• X  Y = if Y then X (i.e., Y does not occur without X also
occurring)
• X > Y = the value of X is larger than the value of Y (on
measure m)
Accuracy (SOC) orders in Dulay & Burt (1974)
Noun -related morphemes
pronoun case
>
article (a, the)
>
reg. plural -s
>
long plural -es
>
possessive-s
>
3PS -s
Verb-related morphemes
prog. -ing
>
contr. cop. Ф
s
>
contr. aux. Ф
s
>
past
Emergence (E)
• Caveat: emergence criteria and their
operationalisation vary across studies.
For example:
Min. no. of occurrences in productive
use (Rapid Profile, Pienemann, 2005
http://groups.uni-paderborn.de/rapidprofile/)
• Pallotti (2006) has discussed the advantages and
also some practical difficulties of using emergence
criteria for investigating L2 development. Also see
Meisel et al (1981) for an early discussion of these
issues.
• Unfortunately, some studies (typically early case
studies) do not even define their emergence criterion
and simply report on the first recorded occurrence of
specific forms. This is of very limited use for our
purposes.
Mastery (M)
Caveat: ‘Acquisition criteria’ and their
operationalisation vary across studies.
Example from FLA research:
Brown (1973) used 90% criterion
• 90% suppliance
• in three successive samples
• each sample contains 5 or more obligatory
contexts
Examples from SLA research:
Andersen (1978) used 80% criterion
Ellis (1988) used 75% criterion
Vainikka and Young-Scholten (1994) used
60% criterion
It has also been found that certain forms
become fully accurate at different times in
different functions/contexts.
For example, Wode’s (1978) longitudinal study of his four L1
German-speaking children during their 6 month-stay in the US
found that while the children were fully target-like from the start
regarding the distribution of the regular plural allophones, the
same forms were often used incorrectly in possessives.
Mxa reg plural -s >> Mxa possessive ‘s
Caveat: Mastery needs to be considered in relation to
all of fad, as mastery may be reached in one area but
not another.
For example, learners may be fully accurate in their
production of X, but the frequency of use and/or the
distribution of use of X may not be target-like.
See, for example, the literature on avoidance
strategies, for some concrete examples. Also see the
development of demonstratives in Banerjee et al.
(2006).
Frequency of use (f)
Example: Ringbom (1998)
D = verbs
f = occurrences of the most frequent
verbs per 10,000 words
f be > f have > f do > f can
Accuracy (a)
Example 1: Ellis (1982)
D= verb TO BE
M= 75% accuracy
a= SOC
Mxa cop ‘be’ >> Mxa aux ‘be’
Example 2: Andersen (1978)
D= grammatical morphemes
M= 80% accuracy
a= SOC scores
Mxa cop ‘be’ >> Mxa aux ‘be’ >> Mxa irreg past >> Mxa ’have’ aux
Distribution (d)
Example 1: Ellis (1988)
D= copula BE
a= SOC
d= pronoun subject contexts vs NP subject
contexts
axd pron S+cop BE > axd NP S+cop BE
Example 2: Cancino et al (1978)
D=negation
E=
d=VP contexts in which negation is used
This study of the acquisition of negation found that
the first verbal forms to be negated post-verbally
were cop ‘is’ and ‘can’, followed by other auxiliaries.
This took place at their proposed stage 3, which
comes before the stage at which negative devices
are productively combined with auxiliaries in a range
of tense and agreement forms.
Exd pres cop+neg, can+neg >> Exd other aux+neg
Advantages of this method
a. DEMfad can be used with different theories
of language.
b.
The combined use of the three operators (, >>
and >) in the description of the relationship
between different phenomena over time and when
compared to other phenomena allows the
researcher to capture both the stage-like and
continuous characteristics of the L2 developmental
route.
c.
The combined use of the three operators makes
the the comparison of cross-sectional and
longitudinal data easier that other approaches.
d.
DEMfad allows us to account for more of the
varation/systematicity in the data than approaches
looking at E, M, f, a or d independently.
e.
The three operators allow us to produce
implicational hierarchies. These are very costeffective tools for diagnosis.
5. CEFLING
CEFLING
Linguistic Basis of the Common European Framework
for L2 English and L2 Finnish
• Project funded by the Academy of
Finland 2007-2009
• Based at the University of Jyväskylä,
but with collaborators at a number of
universities in Europe (SLATE network)
• Homepage: http://www.jyu.fi/cefling
Research questions
1. What combinations of linguistic features
characterise learners’ performance at the
proficiency levels defined in the Common
Framework and its Finnish adaptations?
2. To what extent do adult and young learners
who engage in the same communicative
tasks, at a given level, perform in the same
way linguistically? To what extent are the
adult-oriented CEFR levels and their Finnish
adaptations for young learners equivalent?
3. To what extent are the pedagogical tasks
found in the teaching materials in the
Finnish comprehensive school comparable
with the tasks defined in the CEFR and the
new curriculum?
4. What are the linguistic and communicative
features that teachers (or National
Certificates raters) pay attention to when
assessing learners with the help of the
Finnish adaptations of the CEFR scales?
How do these features relate to the linguistic
and communicative analysis of the same
performances?
Tasks
• From all learners:
–
–
–
–
–
Email to a friend or colleague
Email to your teacher
Email to a store
Opinion piece
Narrative piece
• From subsets of learners:
– Translation (L1 Finnish --> L2 English)
– Word formation tasks
Structure of the dataset
Transcription and coding
• We are adapting CHILDES tools for use
with writing performances
http://childes.psy.cmu.edu/
• The coding system is being developed.
Some of it can be done semiautomatically (e.g., MOR analysis for
English) but other codes have to be
entered manually.
Planned analyses
DnEMfadt6
C2
DnEMfadt5
C1
DnEMfadt4
B2
DnEMfadt3
B1
DnEMfadt2
A2
DnEMfadt1
A1
Ongoing and planned analyses
A number of Ds are currently being
explored. See the list of current
subprojects at
http://www.jyu.fi/hum/laitokset/kielet/cefli
ng/en/Subprojects
Examples of initial findings
• L2 Finnish:
– It’s not just emergence, mastery or frequency of
relative clause use that are particularly informative
(RCs appear at all levels), but the frequency x
distribution of RCs (some types of RCs are typical
at specific levels).
• L2 English:
– Number marking on N is not a discriminating
feature in terms of emergence (E is clear even at
A1) or mastery (accuracy is not 100% even at C2),
but relative accuracy in plural contexts allows us
to conclude that if pl TLU<90%, then the
performance is below B1 level.
6. Final thoughts
Some methodological suggestions
1. Collaboration is better than
competition
•
•
Get in touch with other teams doing this
type of work (e.g., the SLATE network,
your country’s RLD team, etc.)
Aim to open up your work to a range of
experts and stake holders (e.g., SLA
researchers, language testers, teachers,
policy makers, etc.)
2. Datasets
•
•
•
•
The following variables should ideally be
controlled for: task, topic, conditions under which
data are produced by learners (e.g.,
timed/untimed, high-/low-stakes, etc.)
The NS comparison may be helpful (e.g., to help
us understand M better)
We may need to explore some data from levels
below A1 to understand E better
Format of transcription and coding should ideally
be done following some generally accepted
standard in the field (e.g., CHILDES)
3.Rating of performances
Assigning performances to
communicative levels should be done
with great care (or else you risk the
linguistic analyses that follow not being
as reliable or informative as they could
be).
4. Linguistic analyses
• There is no such thing as an a-theoretical linguistic
analysis of data. It’s potentially problematic to work
with unprincipled selections of Ds.
• Overlap between levels is common (esp. between
adjacent levels). This suggests that communicative
levels tend to yield linguistic indices that are
probabilistic rather than categorical. A single feature
(e.g., number) is unlikely to be a completely reliable
index of level, but groups of features may be.
• Given the typically high levels of individual variation
(within-level and within-learner) in L2 development,
group scores should be used and interpreted with
caution.
• Combined, multivariate, factorial analyses (e.g.,
DEMfad) are likely to explain more than analyses of
independent aspects of development.
Much of the work reported here was possible thanks to the
financial support provided by the following institutions:
http://www.ielts.org/
http://www.britishcouncil.org/
http://www.cambridgeesol.org/
http://www.aka.fi
http://www.jyu.fi/en/