Transcript 10 Golden Rules for Writing Multiple Choice Questions

Item Development
Presentation
June 11, 2012
1
Item Developer’s Tasks
 To develop high-quality, standards-based items that will
adhere to the Florida Department of Education Item
Specification and will be used on three formative
district test forms to evaluate students’ strengths and
weaknesses on the Next Generation Sunshine State
Standards (NGSSS) in Social Studies
2
DOK I
DOK II
DOK III
3
DOK Sample Question 1
What was the immediate cause of the outbreak of World War I (1914)?
A. Formation of secret alliances.
B. Competition over Asian and African colonies.
C. Accumulation of arms and weapons.
D. Assassination of Archduke Ferdinand of Austria.
4
DOK Sample Question 2
Use the table to answer the question.
1860
North
South
Industrial
Agricultural
More cities
Mainly rural
Many factories
Few Factories
Slavery widely opposed
Slavery widely supported
Which conclusion related to the Civil War can be drawn from the information in
this chart?
A. The South had more large cities than the North.
B. Sectionalism was based on social and economic differences.
C. The South was well prepared to win a war against the North.
D. Agriculture was no longer important to the United States economy.
5
DOK Sample Question 3
Which late 19th-century event is portrayed in this cartoon?
A.
B.
C.
D.
economic tariffs on European goods
the incident leading to war with Spain
colonial expansion into Latin America
the capture of the Philippine Islands
6
Cognitive Complexity Levels
Low Complexity
Recall and Reproduction
Moderate Complexity
Skills & Concepts
High Complexity
Strategic/ Extended Thinking
Low-complexity items require students to recall, observe, question or
represent basic facts. For a low complexity item, the student would
be expected to demonstrate simple skills or abilities. A lowcomplexity item requires only basic understanding – often verbatim
recall or simple understanding of a term or process.
Moderate-complexity items require two steps: identification and
subsequent processing of that information. Students are expected to
make inferences and may encounter items that include words such
as classify, organize and compare. Depending on the objective a
particular moderate-level item, students may also be required to
explain, describe, or interpret.
High-complexity items make heavy demands on student thinking.
Students may be asked to explain, generalize, or make multiple
connections. High complexity items require several steps involving
abstract reasoning and planning.
7
8
Terminology





Item - the entire multiple choice question.
Stem – the first, sentence-like portion of the multiplechoice question.
Alternatives – all possible multiple-choice response
options.
Key response – correct answer.
Distracters - the wrong answers. They are called
distracters because they should be written to be plausible
and, ideally, reflect common misconceptions, thus
distracting students with only superfluous knowledge of
the tested material.
9
Example
What is the process of elimination in a multiple choice
question? (stem)
A. Skipping the entire question (distracter)
*B. Eliminating the wrong answers (key)
C. Eliminating all extreme answers
(distracter)
D. Just guessing (distracter)
10
Example (poor)
In World War II the Allied strategy, agreed upon by
the U.S. and Great Britain was to
concentrate on defending Japan before turning to
Germany.
B. fight only against Japan, leaving the Russians to fight
Germany alone.
C. take a passive role and limit operations to reacting to
Axis moves.
D. concentrate on defeating Germany first before turning on
Japan.
A.
11
Example (poor)
Use a direct question (not an incomplete sentence) in
the stem
In World War II the Allied strategy, agreed upon by
the U.S. and Great Britain was to
concentrate on defending Japan before turning to
Germany.
B. fight only against Japan, leaving the Russians to fight
Germany alone.
C. take a passive role and limit operations to reacting to
Axis moves.
D. concentrate on defeating Germany first before turning on
Japan.
A.
12
Example (better)
What was the Allied strategy, agreed upon by the U.S.
and Great Britain, in World War II?
Concentrate on defending Japan before turning to
Germany
B. Fight only against Japan, leaving the Russians to fight
Germany alone
C. Take a passive role and limit operations to reacting to
Axis moves
D. Concentrate on defeating Germany first before turning
on Japan
A.
13
Example (poor)
Which of the following events abolished the institution
of slavery in the United States?
A. Ratification of the 13th amendment in 1865
B. Creation of the Freedmen’s Bureau of 1865
C. Compromise of 1850
D. Emancipation Proclamation of 1863
14
Example
Arrange numerical alternatives (such as dates) in
ascending order.
Which of the following events abolished the institution
of slavery in the United States?
A. Ratification of the 13th amendment in 1865
B. Creation of the Freedmen’s Bureau of 1865
C. Compromise of 1850
D. Emancipation Proclamation of 1863
15
Example (better)
Which of the following events formally abolished the
institution of slavery in the United States?
A. Compromise of 1850
B. Emancipation Proclamation of 1863
C. Creation of the Freedmen’s Bureau of 1865
D. Ratification of the 13th amendment in 1865
16
Example (poor)
Which of the following groups of citizens of the
United States were not allowed to cast their votes in
elections for President of the United States until the
23rd Amendment was passed?
17
Example
Do not use negative words, such as “not” or “except” in
a stem.
Which of the following groups of citizens of the
United States were not allowed to cast their votes in
elections for President of the United States until the
23rd Amendment was passed?
18
Example (better)
Passage of the 23rd Amendment gave presidential
election voting privilege to which group of citizens?
19
Example (poor)
Which of the following is an example of a success of the
“right-to-life” movement ?
A. It created a constitutional amendment banning
abortions.
B. Achieving restrictions on abortions
C. Getting the Stewart v. Freeman case to the Supreme
Court
D. All of these choices are correct.
20
Example
Do not use “all of the above” or “none of the
above”
Which of the following is an example of a success of
the “right-to-life” movement ?
A. It created a constitutional amendment banning
abortions.
B. Achieving restrictions on abortions
C. Getting the Stewart v. Freeman case to the Supreme
Court
D. All of these choices are correct.
21
Example (better)
Which of the following is an example of a
success of the “right-to-life” movement ?
A. Creating a constitutional amendment banning
abortions
B. Achieving restrictions on abortions
C. Getting the Stewart v. Freeman case to the Supreme
Court
D. Reversing Roe v. Wade
22
Writing Good Items
Part I. Emphasis on the Stem
 Write questions that examine important concepts;
avoid questions addressing minute details.
 Make the stem clear and unambiguous.
 Use simple grammar and sentence structure.
 Do not use vocabulary more advanced than required.
 Include in the stem any word(s) that otherwise would
be repeated in each alternative.
23
Example (poor)
How many grievances against the king were
named in the Declaration of Independence?
24
Example
How many grievances against the king were
named in the Declaration of Independence?
Write questions that examine important
concepts; avoid questions addressing
minute details.
25
Example (better)
What major grievance against the king was
named in the Declaration of Independence?
26
Example (poor)
As the level of fertility approaches its nadir,
what is the most likely ramification for the
citizenry of a developing nation?
27
Example
As the level of fertility approaches its nadir,
what is the most likely ramification for the
citizenry of a developing nation?
Make the stem clear and unambiguous.
Do not use vocabulary more advanced than
required.
28
Example (better)
What would be the likely result of a decline in
fertility in a developing nation?
29
Example (poor)
I believe that it must be the policy of the United
States to support free peoples who are resisting
attempted subjugation by armed minorities or
outside pressures.
This statement was used
A. to support the Baruch Plan.
B. to support the Monroe Doctrine.
C. to support the Good Neighbor Policy.
D. to support the Truman Doctrine.
30
Example
I believe that it must be the policy of the United States
to support free peoples who are resisting attempted
subjugation by armed minorities or outside pressures.
This statement was used
A. to support the Baruch Plan.
B. to support the Monroe Doctrine.
C. to support the Good Neighbor Policy.
D. to support the Truman Doctrine.
Include in the stem any word(s) that otherwise
would be repeated in each alternative.
31
Example (better)
I believe that it must be the policy of the United States
to support free peoples who are resisting attempted
subjugation by armed minorities or outside pressures.
This statement was used to support which of the
following?
A. The Baruch Plan
B. The Monroe Doctrine
C. The Good Neighbor Policy
D. The Truman Doctrine
32
Writing Good Items
Part II. Emphasis on Alternatives
 Do not let any portion of the stem provide unintended






clues.
Do not let the length of alternatives provide unintended
clues.
Make all distracters plausible (use common errors).
Make the alternatives mutually exclusive.
Keep the grammar of each alternative consistent.
Be careful to keep items independent, so that the answer to
one question does not provide a clue to solving another.
Make the position of the key response (correct answer)
random.
33
Example (poor)
What was one of the most important domestic
economic results of World War II?
A. Complete employment for minority Americans
B. The end of discrimination for African Americans
C. Change in social status for women
D. The swift ending of the Great Depression
34
Example
What was one of the most important domestic
economic results of World War II?
A. Complete employment for minority Americans
B. The end of discrimination for African Americans
C. Change in social status for women
D. The swift ending of the Great Depression
Do not let any portion of the stem provide
unintended clues.
35
Example (better)
What was one of the most important domestic results
of World War II?
A. Complete employment for minority Americans
B. The end of discrimination for African Americans
C. Change in social status for women
D. The swift ending of the Great Depression
36
Example (poor)
“Reaganomics” was based upon which of the following
views?
A. The government must cut taxes to stimulate investment,
which would, in turn, increase employment and tax revenue.
B. The government must commit to large expenditures.
C. The government must institute wage and price controls.
D. The government must increase taxes on corporations and
individuals.
37
Example
“Reaganomics” was based upon which of the following
views?
A. The government must cut taxes to stimulate investment,
which would, in turn, increase employment and tax revenue.
B. The government must commit to large expenditures.
C. The government must institute wage and price controls.
D. The government must increase taxes on corporations and
individuals.
Do not let the length of alternatives provide
unintended clues.
38
Example (better)
“Reaganomics” was based upon which of the following
views?
A. The government must cut taxes to stimulate investment
into the economy.
B. The government must commit to large expenditures.
C. The government must institute wage and price controls.
D. The government must increase taxes on corporations and
individuals.
39
Example (poor)
The English based their claim to territorial rights in
North America on the expeditions of which of their
explorers?
A. Henry Hudson
B. Ferdinand Magellan
C. Sir Walter Raleigh
D. Amerigo Vespucci
40
Example
The English based their claim to territorial rights in North
America on the expeditions of which of their explorers?
A. Henry Hudson
B. Ferdinand Magellan
C. Sir Walter Raleigh
D. Amerigo Vespucci
Make all distracters plausible (use common errors).
41
Example (better)
The English based their claim to territorial rights in
North America on the expeditions of which of their
explorers?
A. Henry Hudson
B. Francis Drake
C. Sir Walter Raleigh
D. John Rolfe
42
Example (poor)
Which of the following was a circumstance of the
election of 1789?
A. Southern Federalists supported Thomas Jefferson.
B. The Federalist Party did not unite behind Adams.
C. George Washington took an active role.
D. The Federalist party divided over the support of Adams.
43
Example
Which of the following was a circumstance of the
election of 1789?
A. Southern Federalists supported Thomas Jefferson.
B. The Federalist Party did not unite behind Adams.
C. George Washington took an active role.
D. The Federalist party divided over the support of Adams.
Make the alternatives mutually exclusive.
44
Example (better)
Which of the following was a circumstance of the
election of 1789?
A. Southern Federalists supported Thomas Jefferson.
B. The Federalist Party united behind Adams.
C. George Washington took an active role.
D. The Federalist party divided over the support of Adams.
45
Example (poor)
Why did President Kennedy decide to remove Diem
from the presidency of South Vietnam?
A. Diem launched attacks on the country's Buddhists.
B. Massacring a large number of Viet Cong
C. Refusing to allow American soldiers to engage in
combat
D. Shooting his own brother for treason
46
Example
Why did President Kennedy decide to remove Diem
from the presidency of South Vietnam?
A. Diem launched attacks on the country's Buddhists.
B. Massacring a large number of Viet Cong
C. Refusing to allow American soldiers to engage in
combat
D. Shooting his own brother for treason
Keep the grammar of each alternative consistent.
47
Example (better)
Why did President Kennedy decide to remove Diem
from the presidency of South Vietnam?
A. Diem launched attacks on the country's Buddhists.
B. Diem massacred a large number of Viet Cong.
C. Diem refused to allow American soldiers to engage in
combat.
D. Diem had his own brother shot for treason.
48
Example
Which of the following events
occurred as a direct result of the
Townshend Acts?
A. The Committees of
Correspondence were formed.
B. The First Continental
Congress was convened.
C. The principle of "salutary
neglect" ended.
D. The colonists resented
mercantilist economic policy.
What was the purpose of the
colonial Committees of
Correspondence created as a
result of the Townshend Acts?
A. Keep colonial intellectuals in
contact with each other
B. Publicize grievances against
England
C. Improve the writing skills of
young gentlemen
D. Correspond with English
radicals who supported the
American cause
Be careful to keep items independent, so that the answer to one
question does not provide a clue to solving another.
49
Example
Which of the following events
occurred as a direct result of the
Townshend Acts?
A. The Committees of
Correspondence were formed.
B. The First Continental
Congress was convened.
C. The principle of "salutary
neglect" ended.
D. The colonists resented
mercantilist economic policy.
What was the purpose of the
colonial Committees of
Correspondence created in
response to the Townshend
Acts?
A. Keep colonial intellectuals in
contact with each other
B. Publicize grievances against
England
C. Improve the writing skills of
young gentlemen
D. Correspond with English
radicals who supported the
American cause
50
Practice Exercise
As a group you will develop one multiple choice question on the following
benchmark associated with your course.
Use
the template to guide you in creating the question.
.
51