Transcript Medical Examinations

Assessing Medical Students’
Clinical & Scientific
Recalling, Reasoning & Decision Making
Abilities
Objectives
 At the end of this workshop you will be able to:
•
determine which tests & techniques best help students
develop clinical and scientific reasoning skills
• and which simply assess & reinforce their ability to
memorise factual information
•
compose selected-response items that measure clinical
& scientific reasoning abilities
Overview (cont’d):
•
Highlight pertinent research
• evaluate selected-response to other assessment formats
• compare T/F and Best One of ‘N’ formats
•
Review basic do’s and don’ts in writing items
•
Clinical vignettes
• useful aid in assessing clinical reasoning
• recommended format
• useful templates for clinical student assessments
• helpful templates for integrating clinical and basic science
student assessments
Overview (cont’d):
 Practice in writing items that measure
• clinical reasoning
• integration of clinical and relevant basic science content
 Practice in critiquing items
 Understand Extended Matching format (Day 1)
 Develop & use Extended Matching Items (Day 2)
 Understand Pick n Best Choices from N Options
format (Day 2)
Important:
 Feel warmly welcomed & very comfortable
•
to challenge, question, suggest, seek clarification
•
at any point throughout the workshop
What’s the relevant research?
1. Key Research Finding:
 What is best learned is not what’s taught
 What is best learned is what’s tested
Examinations drive learning among medical undergraduates
 If one teaches clinical & scientific reasoning,
but
•
tests for only recall of scientific & clinical facts,
then
•
students will learn primarily facts
• and very few reasoning or problem solving skills
Thus, the key criterion for good exams
 Create tests that assess what you want
your students to learn
2. Research:Two components of a test
…determine its validity:
 Content
 Format
Both influence if one obtains a valid measure
e.g., of either clinical & scientific reasoning or
an ability in memorising clinical & scientific facts
Related Research Findings:
 Some test formats are regarded by students to
tap certain skills better than other formats
•
Research: this is not necessarily true
•
But the perception by students remains / is real
Perceived purposes of test formats
 Essays
 Reasoning, synthesis, organisation
 Short answer  Facts: Recall (not recognition)
 Bedside orals,  Clinical skills (non standardised), inViva
& depth probing (in any direction)
 OSCE
 Standardised clinical & practical
skills
Thus, 2nd criterion for good exams
 Use a test format that appears to the student to
tap the skill that you want to be learned
•
in assessment jargon: ensure there is “face validity”
•
if students think test format requires reasoning skills,
their study efforts become directed at these skills
Additional Research Findings:
 Examination results improve as predictors of
future clinical competence if
•
based on a representative sampling of pertinent
clinical skills and scientific knowledge (validity)
•
based on a large sample of these attributes
(precision)
• sample size can be increased by using long tests, or by
using more short tests (latter has some advantages)
Thus, 3rd criterion for good exams
 Test large samples of the skills and knowledge
that you want the students to learn
• important for separating those who do & don’t meet the bar
of minimally acceptable performance (e.g., pass/fail)
Thus, 4th criterion for good exams
 Enable representative sampling
•
Write a large pool of items from which good test
items can be sampled
• validity in predicting future clinical competency improves
Review: The “Musts”
 1. Test for the ability that needs to be acquired
 2. Use a test format that appears to students to
test what you asked/want them to learn
 3. Test a representative sample of the skills
 4. Test a lot of these skills
Problems
Each test format has some shortfalls
 Essays
 Poor sampling; poor reliability; long time
needed to mark
 Short answer
 Often only facts tested; reliability and time
for marking improved (not best);
 Orals, Viva
 Poor sampling; poor reliability; not
equivalent across students; time needed for
administration is longer
 OSCE
 Better sampling and reliability (not best);
much preparation time in relation to sampling
One additional test format: MCQ
 Not suitable for testing practical or clinical skills
 Came into wide spread use because of some
shortfalls in essays, orals/viva
•
easier to administer, easier to mark
 Repeatedly criticised
•
doesn’t measure even memorisation, just recognition
•
students can develop their own item banks
Research findings on the MCQ
 If composed poorly, criticisms of MCQs are valid
Research findings on the MCQ
 If properly written and developed
•
can test clinical reasoning & problem solving abilities
•
best reliability among of all test formats
•
easiest to get more of a representative sample
•
easiest to get a more precise measure
•
best predictor of clinical competency
• (as rated by clinical supervisors)
Two Common Types of Selected-response
questions (MCQ’s)
 True/False or Multiple T/F: categorical (type ‘X’)
 One Best Answer: probabilistic
(type- ‘A’ & ‘R’)
Types of MCQ’s
Among selected-response items (MCQs)
• probabilistic format has become most prevalent
in medical schools & licensing bodies
True / False
T/F (X - type) MCQ’s
 Often assumed that:  Research has found that:
•
simple to prepare
•
difficult to write well
•
easy to understand
•
more likely to be misinterpreted
•
little ambiguity
•
more subject to ambiguity
•
more content sampled
•
often sample less content
•
widely adaptable
•
more likely to be discarded by
medical test committees after review
NBME, p.15
To answer T/F ( X-type) items:
 1. Examinee needs to know the content
 2. May also have to decide to what extent a choice is
‘true’
•
if options are not completely true or completely
false, the students have to guess what the item writer
had in mind
• this latter judgement can be unrelated to their clinical or
scientific expertise
NBME, p.16
Criteria for T/F MCQ’s
 Stems must not have

imprecise phrases
• ‘is associated with’
• ‘is useful for’
•
cues
• ‘may be’
• ‘could be’
•
vague terms
• ‘usually’
• ‘frequently’
 Options must be:
•
absolutely true or false
NBME, p.18
A word of caution:
(regarding trying to write good T/F questions)
Trying to avoid ambiguity in T/F items, medical item writers
often find themselves being pushed toward assessing
the recall of isolated facts -
Something normally to be avoided
Caution:
A true/false decision is not necessarily a trivial exercise
( e.g., deciding if a verdict of guilty or innocent is correct decision)
But writing a T/F question to measure the complex skills
of why one arrived at a T/F decision is very difficult.
Illustrations of these principles in writing
T/F Items
(“X”-type constructed-response)
Acknowledgement
• Portions of the following are reprinted, with permission, from
Constructing Written Test Questions for the Basic and Clinical
Sciences, copyright © 2001 by the National Board of Medical
Examiners®.
[see web site at http://www.nbme.org/PDF/2001iwg.pdf ]
• Other illustrative items are taken from IDEAL’s bank.
Acknowledgement
Hong Kong International Consortium for Sharing Medical
Student Assessment Bank recognises the Swanson & Case
text as a recommended reference standard for item writers
Swanson DB & Case SM, Constructing Written
Test Questions for the Basic and Clinical
Sciences. (3rd Edition, NBME: Philadelphia, 2001)
Example:
A Good T/F Item
O&T
(Multiple T/F item)
The following are autosomal dominant conditions.
A.
B.
C.
D.
E.
Totally False
A
C
D
fibrous dysplasia
osteogenesis imperfecta
osteoporosis
Duchenne’s muscular dystrophy
achondroplasia
Totally True
B
E
A Flawed T/F Item
Example:
Which of the following statements about
Osteoarthritis are True and which are False?
A. there is a genetic pre-disposition
B. males are more affected than females
C. hip joints are more affected than knee
D. obesity is a risk factor in its development
False
A
B
C
True
D
A,B,C are not absolutely true or false.
Experts would only unanimously agree that D is true.
Examine 2nd major type of MCQs
Best One of ‘N’ Choices
( “A”-type Selected-response)
A-type writers also face problems in
composing a good item
If you can avoid making these
common mistakes, students’ test
wiseness will not be a major issue
NBME, p.19?
1. Avoid grammatical cues:
• e.g., one or more distracters may not
follow grammatically from the stem
The following question is not necessarily in your own
specialization & thus the content may not be readily familiar.
If so, this will help illustrate the point . . .
much like a typical medical student you too can
figure out that some options could not possibly be the
correct answer; in so doing, you have a better chance
selecting correct answer without knowing the answer.
NBME, p.19
A 60-year-old alcoholic derelict is brought to
the emergency department by police. After
ascertaining that the airway is open, the first
step in management should be intravenous
administration of
A
B
C
D
E
examination of cerebrospinal fluid
glucose with B1 (thiamine)
CT scan of the head
phenytoin
diazepam
As options “A” & “C” don’t fit grammatically with stem,
neither would be the correct answer
NBME, p. 22-26 ?
2. Avoid irrelevant techniques for making an item
more difficult
 options are sometimes made unnecessarily complicated

- numeric data are not stated in a consistent unit


A. 20 mg
B. 40 g
C. 45 oz
D. 50 oz
- options are in non logical (or non sequential) order

A. 120 ml
B. 100 ml
C. 150 ml
D. 115 ml
In valid tests, difficulty level depends on the level of
clinical & scientific reasoning
NBME, p. 22-26 ?
3. Avoid using these imprecise terms in the
stem and/or choices:





Most of the time
Usually
Frequently
Likely to occur
Probably




Commonly
Associated with
Rarely
Almost never
Consider the variation writers have in what these terms mean.
NBME, p. 22-26 ?
4. Avoid other common item writer mistakes:
a. logical cues (subset of options is collectively exhaustive)
b. absolute terms (‘always’, ‘never’)
c. long correct answer (phrased to be qualified)
d. word repeats (in stem and the correct choice)
e. convergence strategy
(correct answer includes the most elements that are in
common with other options)
Example items for each of these errors in writing questions
are illustrated in your Appendix 1
Having seen what not to do,
examine what should be done
Criteria to meet in writing A-type items
 Stem asks a question & a knowledgeable student can
answer question without looking at set of answer choices
 A common continuum underlies all choices of answers

thereby enabling a “best” answer to be found
 All distracters are plausible
 All distracters are relatively same length as the answer
 Question deals with important concept(s)
 i.e., does not deal with trivial fact(s)
Practice in Diagnosing Quality
of
Type-A Items
Example of a poor
Type A item:
NBME, p.17
Which of the following is true about
pseudogout?
A. Occurs more frequently in women.
B. It is seldom associated with acute pain in a joint.
C. It may be associated with a finding of
chondrocalcinosis.
D. It is clearly hereditary in most cases.
E. It responds well to treatment with allopurinol.
What’s wrong with this question?
Flaw #1
 The Stem is poor
•
After reading stem, student has little idea of
what the question is asking
NBME, p.17
Which of the following is true about
pseudogout?
A. Occurs more frequently in women.
B. It is seldom associated with acute pain in a joint.
C. It may be associated with a finding of
chondrocalcinosis.
D. It is clearly hereditary in most cases.
E. It responds well to treatment with allopurinol.
Flaw #2:
 Options are not on a single dimension
NBME, p.17
Which of the following is true about
pseudogout?
A. Occurs more frequently in women.
B. It is seldom associated with acute pain in a joint.
C. It may be associated with a finding of
chondrocalcinosis.
D. It is clearly hereditary in most cases.
E. It responds well to treatment with allopurinol.
Flaw #2:
 Options are not on a single dimension
•
how can one determine if the frequency of pseudogout’s occurrence in women is more true than the low
frequency of concomitant pain in a joint?
•
is the distinction of frequency of one thing vs the
infrequency of another of any importance?
• asking for a comparison between apples and oranges is not
important & probably always irrelevant for MB ChB students
Examples of a good A-type item
Please note:
An item from Medicine is illustrated now; 3 additional examples
(Surgery, OBGyn, & Orthopaedics) are provided in the handout
NBME, p.16
Example: good A-type item
(Clinical Vignette)
A 32-year-old male has a 4-day Hx of progressive weakness in his
extremities. He has been healthy except for an upper respiratory
tract infection 10 days ago. His temperature is 37.8 oC, blood
pressure is 130/80 mm Hg, pulse is 94/min and respirations are
42/min and shallow. He has symmetric weakness of both sides of
the face and the proximal and distal muscles of the extremities.
Sensation is intact. No deep tendon reflexes can be elicited; the
plantar responses are flexor.
(Lead-in)
Which of the following is the most likely diagnosis?
NBME, p.16
Which of the following is the most likely diagnosis?
(Alternatives):
 A.
 B.
 C.
 D.
 E.
Acute disseminated encephalomyelitis
Guillain-Barre syndrome (*)
Myasthenia gravis
Poliomyelitis
Polymyositis
NBME, p.17
Even though the incorrect answers are not completely
wrong, they are less correct than the ‘keyed’ answer.
D
C A
E
B
_____________________________________________
Least
Correct
Most
Correct
Good features
 Experts would agree on the most likely diagnoses

- or certainly agree that the other diagnoses are less likely
 Item measures more than recall of isolated facts
 Lead-in is clear and unambiguous
 All options can be laid out on a single continuum
 Even if previous question was only somewhat
familiar (not necessarily something you would test
in your own teaching)
•
you probably still could tackle the item
• if so, you sensed that which students feel --- the question, not
familiar, required one to reason & problem solve
• with some known facts and reasoning you could solve it -- i.e.,
the question was fair, you did not guess, at worst guesstimated
• ‘guesstimation’ can be & estimation is a desirable skill to
measure in medical students as few real clinical decisions are
on based on certainty
– measure of ability to make decisions under uncertainty
These same principles can apply for
assessments in any of the clinical or
basic sciences . . .
(additional clinical science examples are included in
your handout, a couple of basic science items follow)
Anatomy
A 42-year-old male laborer fell off a structure and is
taken to hospital. A fracture of the middle third of
the shaft of the humerus is diagnosed and damage
to the radial nerve is suspected by the examining
doctor.
If the doctor's suspicion is indeed correct, the patient probably
will be unable to:
A. Oppose the thumb of the middle finger.
B. Extend the wrist against gravity.
(*)
C. Feel a pinprick on the lateral side of the forearm.
D. Contract the triceps muscle.
Physiology, NBME, p.44
A 34-year-old woman has had severe watery
diarrhea for the past four days. Two months
earlier she had infectious mononucleosis. She
abuses drugs intravenously and has antibodies
to HIV in her blood. Physical examination
shows dehydration and marked muscle
weakness.
Laboratory studies are most likely to show
A. decreased serum K+ concentration.
B. decreased serum Ca2+ concentration.
C. increased serum HCO3- concentration.
D. increased serum Na+ concentration.
E. increased serum pH.
(*)
NBME, p. 33
Review: basic rules for writing A-type items
 Focus on important concept
•
e.g., a common or potentially catastrophic clinical problem
 Include items that assess application of knowledge
•
not only recall of isolated facts
 Pose a clear question
•
student can arrive at an answer with options covered
 Make all distractors homogeneous

common continuum exists for all choices
 Avoid technical flaws in composition

see previous slides & Appendix 1 re errors to avoid
Thus: A-type items are also difficult to write
(not unlike T/F)
But when well written, A-type less often forces the
writer to deal with isolated facts
This does not mean A-type questions are a
panacea
•
Good multiple T/F questions can be also written
• but medical test writers usually find it more difficult
to write them well
Additional research in comparing X and A type items
is included in your handout
Considerations:
 If development of problem solving and clinical reasoning
skills is wanted, one has to test for these skills (and do so
with many questions/tasks)
•
May wish to consider using some A-type items to help do so
•
May wish to involve all teaching staff in writing items so a large
bank of current, clinical reasoning items are available
May want to use IDEAL’s item banks
 10,000+ questions are now available
 Consortium provides a windows-based computer program for

convenient storage, retrieval, test creation, printing, updating
and item response analysis

versions for hard copy high stakes & web-based administered
examinations
 Created by pooling & sharing items among medical schools at:

Hong Kong (CUHK, HKU), British Columbia, Sydney, Otago,
Flinders, Witwatersrand, and recently Peninsula)
Considerations:
 With any item bank . . .
•
Important to regularly review & modify items as suitable
measures
• importance & currency of content, clarity, lack of writing errors
• statistical evidence from previous assessments to guide
revisions
– will discuss more of this later in the workshop
What questions do you have?
(mini break)
Look at the course objectives
(particularly the verbs in the objectives)
 If the objectives state the students should
•
recall, recognise, remember
•
these verbs indicate the students need to memorise facts
 If objectives state the students should
•
analyse, assess, apply, compare, conclude, contrast,
diagnose, derive, differentiate, discriminate, evaluate,
estimate, extrapolate, generalise, predict, relate, validate
•
these verbs indicate the students need to do 3 things:
• recall what they know, interpret the info, & then draw
some conclusion
• this process is reasoning or problem solving
The secret to assessing more than
factual information
 Consider the following two items
O&T
Osteogenesis imperfecta is the result of a
defect in the biosynthetic pathway for
A.
B.
C.
D.
E.
fatty acids
elastin
gelatin
collagen (*)
osteonectin
and then this question . . .
O&T
An 8-year-old child is admitted to hospital with a
fracture of the femur following a fall at home. The
child has been admitted many times before with
fractures affecting both the femora, humeri and
clavicles. Physical examination shows he has a blue
sclera, rib cage deformity shortening and malunion
of the arm and lower limb.
These findings suggest a defect in the biosynthetic pathway for
A.
B.
C.
D.
E.
fatty acids
elastin
gelatin
collagen (*)
osteonectin
The pair of items measure the same topic
 The choices are identical in each, but
•
•
the 1st item tests knowledge (memorised facts)
the 2nd item tests application of knowledge to a
clinical situation
 To achieve the second, develop a clinical vignette
NBME, p.38 or p.56
A vignette includes a patient description & a clinical
scenario with some or all of the following information:
Age, Gender (e.g. A 45-year-old man)
Site of Care (e.g. comes to the emergency department)
Presenting Complaint (e.g. because of annoying pain in his knee)
Duration (e.g. that has continued for 2 days;)
Patient History (e.g., inside desk job, active in athletics;)
Physical Findings
+/- results of diagnostic workup
+/- initial treatment, subsequent findings, etc.
NBME, p. 36 ?
Advantage in using a clinical vignette
 Face validity goes up  Students regard items with
clinical vignettes as more
clinically relevant
 Students see that they have to
apply their knowledge to make a
decision or solve a problem
Additional advantage of a clinical vignette
 One can create more than one question from one
vignette
 In an integrated curriculum this is particularly
advantageous
•
as related clinical & basic science questions can be
asked under same vignette
(An item that draws on knowledge from Physiology)
A 44-year-old housewife is admitted to
the ward with generalized bone pain
especially over the spine and pelvis. She
has a history of breast cancer, and had a mastectomy 3
years earlier. A plain radiograph of the spine reveals
collapse of the L4 and L5 vertebrae.
Laboratory studies are most likely to show
A.
B.
C.
D.
E.
decreased
increased
increased
increased
decreased
Se Ca
Se Ca (*)
Se Creatinine
Se Albumin
Se Creatinine
(And this item draws on coverage from Radiology & Orthopaedics)
Additional question that could be asked
using the same vignette:
What is the most appropriate investigation for this patient at
this stage?
A.
B.
C.
D.
E.
bone scan (*)
Dexa study
bone biopsy
MRI examination
Se PTHRP measurement
(And this item also draws on coverage in Orthopaedics)
A third question using the same vignette
A skeletal survey was performed revealing
multiple metastastic lesions in the pelvis,
cervical dorsal and lumbar spine.
What is the next most appropriate step in management?
A.
B.
C.
D.
E.
(*)
intravenous Pamidronate
spinal stabilization
calcium and Vit D supplements
whole body irradiation
treatment with Tamoxifen
Additional examples of multi disciplinary item sets
(i.e., that share a common clinical vignette)
are provided in your handout
The most important advantage in using
a clinical vignette:
If your objective is to measure the application
of knowledge (rather than testing for factual
knowledge) . . .
the use of a clinical vignette greatly helps.
Advantage in using clinical vignette
 Improves item’s
discrimination power
 The poorer and better students
perform differently
 Students who only memorize
often cannot answer an
application question as well
Med
A 54-year-old female has frank aortic regurgitation,
with a murmur graded six out of six, a diastolic thrill
along the left sternal border and egg shell
calcification of the aortic arch shown on x-ray. Her
main complaint is angina progressing over several
years. She is now in congestive failure.
The most informative laboratory examination would
be a/an:
A.
B.
C.
D.
E.
Positive LE cell preparation.
Elevated antistreptolysin O titer.
Elevated LDH and heart muscle isoenzymes.
Positive rheumatoid factor in the serum.
Positive VDRL. (*)
Medicine
Are there pedagogical reasons to structure
questions to test more than recall of isolated
facts . . .
particularly when it’s so important to
have a good factual base?
The answer (generally) is Yes:
 But not because:
• the application question is more difficult
 Not because facts are thought, inappropriately,
to be unimportant
The answer (generally) is Yes:
 Because:
•
study preparation for answering application of facts is
different, helping student remember more of the facts being
used in the application
•
for medical students facts are easier to remember if they
become organised in a clinical, logical & relevant fashion
Providing a CONTEXT is the secret
Some evidence to show that context is important &
needs to be provided
 Have you ever gone from one room to another, looking for
something, and then when you got there you couldn’t
remember what it was you were intending to get?
 Did you go back to your room & then recalled what it was
that you needed?
 This was not co-incidental.
• your own room provided the context
• your room cognitively trigged recall of the object because it
was there that you needed to use it
Some evidence to show that context is important &
needs to be provided
 Have you ever gone from one room to another, looking for
something, and then when you got there you couldn’t
remember what it was you were intending to get?
 Did you go back to your room & then recalled what it was that
you needed?
 This was not co-incidental.
• your own room provided the context
• your room cognitively trigged recall of the object
because it was there that you needed to use it
Experimental evidence to support thesis that
context is important & needs to be provided
 Classical experiment with scuba diver qualification tests
•
results on written tests (re decompression) for students who
took the test underwater were better
• in comparison to those who were tested in a classroom
• given nitrogen toxicity build-up that occurs during a
dive, opposite result would be expected
Still need further experimental proof about
importance of providing context?
… maybe it will help if we do a small, but soundly
designed, scientific experiment right now that
hopefully will prove the point
Two members in this audience are needed
 We will randomly select two members
 Both will leave the room, after randomly assigning
which will come in back first and take a little test
 Second person will then come in and take the test
 Both are requested to try to do as well as possible
Randomly select
Randomly assign
Candidates please leave the room;
will call each of you when to return
Although we’re using a sample of two (i.e., my statistical power for
this experiment is minimized),
the effect size is so large that I’m still confident of the
experimental outcome --- so confident that I will make a wager.
The Wager
•
I bet I can predict which of the two will do better on the test
•
To make it even more difficult for me to win
• I will give my choice significantly less time to prepare for
the test
• Some of you will score each candidate on the test, not me
•
If I am proved to be wrong, I will give this group 500 Rand
(buy a coffee or tea or a coke)
•
If I am right, I get to keep my money, but hopefully you will
remember a most important principle
First Candidate
You have one minute to study the following material.
Please try to remember as many points as you can.
Standing sometimes helps. Rain adds danger. After awhile
you may get thirsty. Using cement is better than grass and
grass is better than rocks. Getting geared up makes things
harder. Using one hand is optional but only when seated.
Chains work but ropes do not. Leaning forward helps a lot,
but leaning a lot in other directions can be a hindrance.
Using sidewalks makes people unhappy. Using both legs is
easier than one. Wind can hinder but having good air
pressure is helpful. You can’t go backwards easily. It is quite
easy to learn and difficult to forget. Tipping is not a gratuity
and often hurts more. Nails are obstacles.
Second Candidate
I will give ¾ minute to study the following material on
How to Ride a Bike.
Please try to remember as many points as you can.
How to Ride a Bike
Standing sometimes helps. Rain adds danger. After awhile
you may get thirsty. Using cement is better than grass and
grass is better than rocks. Getting geared up makes things
harder. Using one hand is optional but only when seated.
Chains work but ropes do not. Leaning forward helps a lot,
but leaning a lot in other directions can be a hindrance.
Using sidewalks makes people unhappy. Using both legs is
easier than one. Wind can hinder but having good air
pressure is helpful. You can’t go backwards easily. It is quite
easy to learn and difficult to forget. Tipping is not a gratuity
and often hurts more. Nails are obstacles.
Let's see who remembered the most points
Remembering details is much more onerous
if . . .
a meaningful context is not available
Examine the nature of a meaningful context
i.e., it will elicit cognitive skill of application (not only recall of facts),
& at the same time actually helps one to remember facts
 For MB ChB students
• as their interests/orientation is clinical, patient-centered
• using a clinically oriented vignette is certainly helpful
• but only describing some patient (or a disease or trauma, or
some Hx or Px findings) doesn’t necessarily make the task
meaningful
 To develop a meaningful context requires task to
mimic real life
Nature of Meaningful context
 A clinically oriented question that still only measures recall
•
begins by citing a disease & then asks which patient findings are
to be expected
 Such a sequence is flawed: i.e., it is backwards to what
happens in real life . . .
• a patient rarely tells a doctor what disease s/he has, after which
the doctor then asks for her/his signs and symptoms
• the sequence is however (& unfortunately) similar to how many
textbooks do cover material
Here’s an illustration of an item that lacks
meaningful context
(note that the question is clinically oriented,
but it doesn’t provide a meaningful context)
Which of the following diseases is an
indication for doing HLA B-27 testing
in a 28-year-old male?
A.
B.
C.
D.
E.
Ankylosing Spondylitis
Reiter’s Disease
Rhuematoid Arthritis
Psoriasis
Scleroderma
What’s wrong ?
In clinical practice, doctors don’t go around
searching for the most appropriate patient in
order to do some HLA B-27 testing.
The logical clinical task (meaningful
context) would be something such as:
A 28-year-old clerk has been complaining of
pain and stiffness of the spine & hip joints
for a year. He has now developed a
kyphosis and requires a cane for support.
Radiographs show scaroilitis.
Which of the following investigations is the most appropriate
to undertake at this time?
A.
B.
C.
D.
E.
test for presence of Bence Jones Protein
do a VDRL test
do a HLA B-27 test
test for presence of C-Reactive Protein
test for presence of anti-nuclear antibody
Thus, best guideline for composing good
questions is:
structure items in a clinically realistic
manner
such as,
- give some findings, then
- ask student to indicate the underlying disease
You may find that a template helps you
to keep this sequence as you develop
items
See Appendix 2 for the clinical sciences;
and Appendix 3 for the basic sciences &
for integrating clinical & basic sciences
Item templates can help you compose items
- easier, quicker
- more than one item can be developed from the
same template.
- some illustrative items are included in your
handout
- a few shown now; when we break into
groups to develop items (using these
templates) you might find the other
examples useful
Acknowledgement
• These templates are drawn from:
Constructing Written Test Questions for
the Basic and Clinical Sciences
D. B. Swanson and S.M. Case
(3rd Ed., NBME: Philadelphia, 2001)
See web site: http://www.nbme.org/PDF/2001iwg.pdf
Clinical Sciences (Appendix 2)
NBME, p. 61
If writing items for
Health and Health Maintenance Tasks:
 Assess ability to
•
evaluate risk factors
•
understand epidemiological data
•
apply preventive measures
NBME, p. 61
Health and Maintenance items usually fall
into one of the following categories:
 screening tests
 emergency intervention
 immunisations / travel medicine
NBME, p. 61
Basic Strategy:
begin with a vignette that describes the
patient & include a relevant clinical scenario
 History and physical examination
 Information about immunisation history, risk
factors, and family history
 Relevant information about the community
 Question should focus on individual patient
NBME, p. 61
After the scenario, these lead-ins are useful for
assessing health and maintenance topics:
 Which of the following is the most appropriate screening
test?
 Which of the following tests would have predicted these
findings?
 Which of the following is the most appropriate
intervention?
 For which of the following conditions is the patient at
greatest risk?
NBME, p. 61
Lead-ins for health and maintenance (cont’d)
 Which of the following is most likely to have prevented
this condition?
 Which of the following is the most appropriate next step in
management to prevent [morbidity/mortality/disability]?
 Which of the following should be recommended to prevent
disability from this injury/condition?
 Early treatment with which of the following is most likely
to have prevented this patient’s condition?
Radiology & Surgery
A 2-year-old child presents with a
midline cystic neck nodule which
moves on tongue protrusion and
swallowing. A clinical diagnosis of a
thyroglossal duct cyst was made.
A ultrasound scan is indicated to evaluate:
A.
B.
C.
D.
E.
The exact size of the cyst.
The exact delineation of the associated tract.
The presence and status of the thyroid gland. (*)
The presence/absence of the hyoid bone.
The thickness of the cyst wall.
NBME, p.63
Templates for testing Mechanisms of Diseases
 These test students’ knowledge of --•
aetiology, pathogenesis, natural history, clinical
course, associated findings, complications,
severity of illness, and intended or unintended
effects of therapeutic intervention
NBME, p.63
Basic strategy:
Begin a mechanism item with a clinical vignette of
a patient and her symptoms, signs, history, lab
study findings, etc. . . .
then ask a question such as one of the
following:
NBME, p.63
Questions for testing mechanisms:
 Which of the following is the most likely explanation for
these findings?
 Which of the following is the most likely location of the
patient’s lesion?
 A biopsy is most likely to show which of the following?
 Which of the following is most likely to be
increased/decreased?
 Which of the following is the most likely pathogen?
Surgery
A 70-year-old man has just returned from the
operating theatre after a transurethral resection
of the prostate. The duty House Officer noticed
there was no urine flow from the urethral
catheter.
Which of the following is the most likely cause?
A.
B.
C.
D.
E.
Hypovolumic shock.
Blood clot retention. (*)
Inadequate resection of the prostate.
Hyponatraemia.
Urethral stricture.
NBME, p.64
Templates related to Diagnosis
 Require the students to interpret a history, physical
findings, results of laboratory, imaging, and other
studies in order to determine
•
the most likely diagnosis (or differential diagnosis);
•
the next step in diagnosis (or, in diagnostic testing).
NBME, p.64
Basic strategy for items designed to diagnostic
ability begins with a patient description:




Age
Symptoms
History
Lab studies




Gender
Signs (duration)
Physical findings
Diagnostic findings
 Ends with one of the following questions
NBME, p.64
Questions related to diagnosis
 Which of the following is the most likely diagnosis?
 Which of the following is the most appropriate next step
in diagnosis?
 Which of the following is most likely to confirm the
diagnosis?
Surgery
A 29-year-old woman has had a tender mass
in the upper outer quadrant of the right
breast for six weeks. The mass became
somewhat smaller during her last menstrual
period but then increased in size again. The mass is now
3 cm in diameter, smooth in contour, and is slightly
tender.
Which of the following is the most appropriate next
step in diagnosis?
A. A mammogram.
B. An excisional biopsy.
C. Ultrasonography of the mass.
D. A needle aspiration. (*)
E. Re-examination in four weeks.
NBME, p.65
Templates related to Management
 Assess principles of chronic and acute care in
inpatients and outpatient settings
 Focus on aspects of care relevant to the level of
practice expected of the student
•
•
•
•
supervised (according to most accreditation criteria)
limited supervision (maybe ??)
independent practice (very unlikely)
subspecialist (never)
NBME, p. 65
Management questions include:
 Which of the following is the most appropriate initial or
next step in patient care?
 Which of the following is the most effective
management?
 Which of the following is the first priority in caring for
this patient
•
e.g., in emergency department
 Which of the following is the most appropriate
pharmacotherapy?
NBME, p.66
Writing Items on Difficult Topics:
Medical Ethics and Jurisprudence
 Common belief is that certain topics do not
lend themselves to a selected-response
format.
Difficult Topics:
NBME, p.66
 Consent and informed consent to treatment
•
full disclosure, alternate therapies, risks and benefits
 Physician-patient relationship
•
truth-telling, confidentiality, privacy, autonomy, public reporting
 Death and dying
•
diagnosing death, life-support, autopsy, organ donation,
euthanasia, suicide
 Birth-related issues
•
prenatal diagnosis, abortion, maternal-foetal conflict
 Research issues
•
consent, placebos, conflict of interest, vulnerable populations
NBME, p.66
Poor Question
(that attempts to test difficult topics)
The legal basis for an easing of restrictions on
abortions in S.Africa can be traced most closely to
A.
B.
C.
D.
S. African legislation
S. African court ruling
Practice in other countries
Rulings made by the S. Africa Medical Association
Tests for recall of some historical fact

To avoid the foregoing type of mistake, focus on what
the surgeon or physician should do
for example . . .
- evaluate the appropriateness of a doctor’s
actions that is indicated in a clinical vignette.
NBME, p.67
A nurse is hospitalised, for repair of a torn knee
ligament, at the medical centre where she is employed.
One week after discharge, the assistant hospital
administrator asks her orthopaedic surgeon what the
final diagnosis was.
Which of the following is the most appropriate response on the part
of the surgeon?
A.
B.
C.
D.
E.
Provide the answer, as it will expedite handling of insurance issues
at the medical centre
Provide the answer, since the administrator has access to
information about the patients
Provide the answer, in order to avoid the possibility of spreading
misinformation about the patient
Decline to answer, as the administrator is not a medical doctor
Decline to answer, as the information is confidential (*)
Writing items for Basic Sciences
& an
Integrated Curriculum
See Appendix 3
Assessment templates to help vertically integrate
Basic and Clinical Sciences
Appendix 3
templates for use in earlier years of medical
programme
& also in later years if integration with basic
science is a desired focus of assessment
These templates may prove helpful if you are
trying to elevate the clinical relevance of some basic
science questions
Curriculum revisions that seek to achieve greater vertical
integration also require that:
Clinical instructors become more involved in and
with coverage of basic science material (& vice
versa).
This logically leads to revised student assessments --those that reflect this desired vertical integration.
If students are assessed in this fashion, they must also
be taught in this fashion.
As students progress use a short vignette to ask about both
clinical and basic science aspects that underlie a particular
patient problem
 Describe a laboratory task or a patient condition in
which a problem or question is inherent
 Then ask a question related to this task, patient or
situation
. . . see the following templates
[Short vignette]
A (insert here a description of patient) is unable to
(insert here the patient’s functional disability).
[Lead-in question]
Which of the following (bones, muscles, ligaments,
cartilage) is most likely to have been damaged or
injured?
[Provide a list of optional injury sites]
A.
B.
C.
D.
E.
Anatomy
A 10-yr-girl demonstrates an inability to hold a
piece of paper firmly between the extended
middle and extended ring fingers.
Her inability to do so probably indicates an
injury to the:
A. median nerve above the wrist.
B. deep branch of the ulnar nerve. (*)
C. superficial branch of the ulnar nerve.
D. dorsal branch of the ulnar nerve.
E. posterior interosseous nerve.
NBME, p. 39
 A (patient description) has a (type of injury and location).
•
Which of the following structures is most likely to be
affected?
Provide a list of optional structures
Anatomy
A 18-year-old male patient has an awkward gait,
characterized mainly by dragging his toes on the
ground during the "swing phase" and slapping his
foot on the ground at "heel strike".
This young man likely has a lesion affecting the:
A. nerve to tibialis anterior.
B. superficial peroneal nerve.
C. tibial nerve.
D. common peroneal nerve.
E. medial plantar nerve.
(*)
NBME, p. 39
 A (patient description) has (abnormal findings).
•
Which (additional finding) would suggest a
diagnosis of (injury 1) rather than (injury 2)?
Provide list of optional findings
NBME, p. 39
 A (patient description) has (abnormal findings).
•
Which of the following (positive laboratory
results) would be expected?
Provide a list of optional lab results
NBME, p. 39
 Following (a procedure), a (patient description) develops
(symptoms and signs). Laboratory findings show
(findings).
•
Which of the following is the most likely cause?
Provide a list of optional causes
NBME, p. 39
 A patient has (abnormal findings) but (normal findings).
•
Which of the following is the most likely diagnosis?
Provide list of optional diagnoses
NBME, p. 39
 A (patient description) dies of (trauma/injury).
•
Which of the following is the most likely
finding on autopsy?
Provide list of possible findings
NBME, p. 39
 A patient has (symptoms and signs).
•
Which of the following is the most likely
explanation for the (findings)?
Provide a list of possible causes
NBME, p. 39
 A (patient description) has (symptoms and signs).
•
These observations suggest that the (disease) is
a result of the (absence or presence) of which
of the following (enzymes, mechanisms)?
NBME, p. 39
 A (patient description) has (history findings) and is
taking (list her medications).
•
Which of the following medications is the most likely
cause of her (one of: history, PE or lab findings)?
NBME, p. 39
 A (patient description) follows a (specific dietary regime).
•
Which of the following conditions is most likely to
occur?
NBME, p. 39
 A (patient description) has (symptoms, signs, or
specific disease) and is being treated with (drug or
drug class).
•
The drug acts by inhibiting which of the
following (functions, processes)?
NBME, p. 39
 (Time period) after a (event such as trip or meal with
certain foods), a (patient or group description) became
ill with (symptoms and signs).
•
Which of the following (organisms, agents) is most
likely to be found on analysis of (food).
NBME, p. 39
 A (patient description) has (symptoms and signs).
•
Exposure to which of the (toxic agents) is the
most likely cause?
NBME, p. 39
 Which of the following is the most likely mechanism
of the therapeutic effect of this (drug class) in
patients with (disease)?
Options are sometimes difficult to develop
 Alternatives to the correct  Options to use for various
answer
lead-in questions
•
should be seductive
•
diagnostic of students typical
“mis-understanding” or “mislearning”
•
enable you (the teacher) to
provide helpful feedback to
the student
•
see last page of Appendix 3
& the following
After your vignette, if your lead-in question is . . .
NBME, p. 40
Which of the following ________ is (or is most likely
to be) (present / normal / abnormal)?
Options sets could include:
-sites of lesions
-list of nerves
-list of muscles
-list of enzymes
-list of hormones
-types of cells
-list of neurotransmitters
-list of toxins,
-list of molecules
. . . or
If your lead-in question is . . .
NBME, p. 40
Which of the following findings is most
likely?
Options sets could include:
-list of laboratory results
-list of additional physical signs
-DNA analysis results
-serum levels
-results of microscopic examination of fluids,
muscle or joint tissue
.
. . . or
NBME, p. 40
Which of the following _______is
(defective or deficient or nonfunctioning)?
Options sets could include
-list of enzymes;
-list of feedback mechanisms;
-list of endocrine structures, dietary elements,
vitamins.
. . .or
NBME, p. 40
Which of the following (mechanisms, medications,
drugs, agents) is the most likely cause?
Options sets could include
-list of underlying mechanisms of the disease;
-medications that might cause side effects;
-drugs or drug classes;
-toxic agents;
-hemodynamic mechanisms, viruses, metabolic
defects.
. . . or
NBME, p. 40
Which of the following _______ should be
administered?
Options sets could include:
-drugs;
-vitamins;
-amino acids;
-enzymes;
-hormones.
Let’s recap the pertinent issues
if using any of these
clinical or integrative templates
Clinical vignettes help to establish a realistic context
within which the students’ knowledge base becomes
stored, organised & more easily recalled & used:
 Help focussing on important information, not trivia
•
the student while studying (and the item writer)
 Help identifying students who have only memorised
factual information
•
those who are unable to use the information effectively
 Help improving relevance of material
•
examinees required to ‘solve’ clinically-oriented, patientcentred problems
NBME, p.37
Remember to
 Focus items on key concepts and principles that are
essential information for all examinees to understand
 Test material that is relevant to learning in clinical
clerkships
 Focus on common or potentially catastrophic
problems
•
avoid esoteric or interesting topics that are not essential
• Mimic clinical practice by describing a patient then
asking the student to take some course of action
 Pose the type of decision-making tasks that are
expected of a successful examinee
•
those that must be able to be undertaken at the next
stage of training in a MB ChB programme
NBME, p.38
The patient description within the vignette should include
some or all of the following:
Age, Gender (e.g. A 45-year-old man)
Site of Care (e.g. comes to the emergency department)
Presenting Complaint (e.g. because of headache)
Duration (e.g. that has continued for 2 days)
Patient History (with/without Family History)
Physical Findings
+/- results of diagnostic workup
+/- initial treatment, subsequent findings, etc.
A lead-in question follows the vignette, such as:
 Determine the most likely hypothesis or diagnosis
 Indicate what additional laboratory studies should be
done
 Formulate the next step in managing this patient or in
conducting this experiment
 Predict the most likely additional finding
NBME, p.41
When writing your options, make sure that they are:
 Homogeneous in content (e.g. all are diagnoses; all are
next steps in patient care)
 Incorrect or inferior to the correct answer
 Plausible and attractive to the uninformed
 Similar to the correct answer in construction and length
 Grammatically consistent and logically compatible with
the stem
What questions do you have at this
point --or concerns, needed clarifications,
issues to debate?
Practice in writing & critiquing
MCQs
Using clinical vignettes
Break into groups of five
 Each group is expected to include both clinical and basic science
teachers and the group is to write at least two items
•
may involve using only one or two clinical vignettes
•
develop one of the items so it integrates clinical & basic science material
•
you will have 25 minutes
 Copy one of the items onto an overhead
 Group rep will present the item to the total group
 Group will critique
 Revise the item based on group’s feedback
•
review the other item that you did not present to the group, to see if
revisions are also needed
 Submit final versions of both items to your Block Assessment
Groups for use in this next year’s student assessments
Additional relevant information not as
yet covered . . .
for writing items to assess students’
Clinical Problem Solving Ability
NBME, p.55
Some Don’ts in using clinical vignettes:
 Avoid clinical situations handled by post graduates
and specialists
NBME, p.57
Some Don’ts (cont’d):
 Avoid or least minimise use of negatively phrased
questions:
•
e.g., avoid using “except” or “not” in the lead-in
• why test for something is not applicable?
– who cares?
– certainly not the students, nor the patients
General
The following conditions could have caused
carbon dioxide retention, EXCEPT:
A.
B.
C.
D.
E.
Myasthenia gravis.
Pentobarbital overdosage.
Severe chronic airflow obstruction.
Severe metabolic alkalosis.
(*)
Renal tubular acidosis.
General
Which of the following is NOT a proper
treatment for akathisia ?
A.
B.
C.
D.
E.
Beta blockers.
Benzodiazepines.
Lithium carbonate. (*)
Antihistamines.
Anticholinergics.
Avoid using a vignette from a textbook or one
already discussed in class
•
if the vignette describes a patient identical to one
the student has seen before, you’ll measure only
recall
NBME, p.58
Issues re additional information in the vignette
 Avoid verbosity
•
keep it succinct; extra wording takes more time, adds no validity
 Can include some “window dressing” to enhance realism
•
•
“incidental findings”
but tends to make vignette long, extra reading & time
 Avoid including “red herrings”
•
•
i.e., information that is intended to lead examinees away from
correct answer
this is usually only suitable for postgraduates’ or specialists’
examinations
NBME, p.58
NBME’s research on use of vignettes:
 None, short and long vignettes
•
Items became more difficult as patient findings were
presented in a less interpreted form
• Discrimination increased (not statistically significant)
• Regarded generally more appropriate because vignettes tested
application of knowledge to patient situations, and posed
appropriate clinical challenges (face validity)
For the lower-performing students,
the following 3 items were not identical
NBME, p.59
Non Vignette:
The most likely renal abnormality in
children with nephrotic syndrome and
normal renal function is:
A.
B.
C.
D.
acute poststreptococcal glomerulonephritis
hemolytic-uremic syndrome
minimal change nephrotic syndrome (*)
nephrotic syndrome due to focal and segmented
glomerulosclerosis
E. Schönlein-Henoch purpura with nephritis
A B C D E
hi 1 0 99 0 0
lo 8 1 90 1 0
NBME, p.59
Short Vignette:
A 2-year-old boy has a 1-week history of oedema. His
blood pressure is 100/60 mm Hg, and there is a
generalised oedema and ascites. Serum concentrations
are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569
mg/dL. Urinalysis shows 4+ protein and no blood.
The most likely diagnosis is
A.
B.
C.
D.
E.
acute poststreptococcal glomerulonephritis
hemolytic-uremic syndrome
minimal change nephrotic syndrome (*)
nephrotic syndrome due to focal and segmented glomerulosclerosis
Schönlein-Henoch purpura with nephritis
A B C D E
hi 0 0 98 2 0
lo 5 2 82 8 1
NBME, p.60
Long Vignette:
A 2-year-old black child developed swelling of his eyes
and ankles over the past week. Blood pressure is 100/60
mm Hg, pulse 110/min, and respiration 28/min. In addition
to swelling of his eyes and 2+ pitting oedema of his ankles, he has
abdominal distention with a positive fluid wave. Serum concentrations are:
creatinine 0.4mg/dL, albumin 1.4g/dL, and cholesterol 569 mg/dL.
Urinalysis shows 4+ protein and no blood.
The most likely diagnosis is
A.
B.
C.
D.
E.
acute poststreptococcal glomerulonephritis
hemolytic-uremic syndrome
minimal change nephrotic syndrome (*)
nephrotic syndrome due to focal and segmented glomerulosclerosis
Schönlein-Henoch purpura with nephritis
A B C D E
hi 0 1 98 1 0
lo 10 9 66 10 5
Helpful Statistics
What are the typical statistics
 If a reasonably well written clinical vignette
is used?
General
Example
A 63-year-old man complains of burning
numbness of his feet and weakness of his legs.
Examination shows reduction of the tendon
reflexes and superficial sensation in the legs.
The muscles are weak and tender.
The diagnosis is likely to be:
A.
B.
C.
D.
E.
Polymyositis.
Sympathetic reflex dystrophy.
Amyotrophic lateral sclerosis.
Peripheral neuropathy. (*)
Multiple sclerosis.
General
However, use of a clinical vignette doesn’t solve all
problems. Notice the following question and the pattern
of the students’ responses at the University of Alberta
???
A 29-year-old woman has had a tender mass
in the upper outer quadrant of the right breast
for six weeks. The mass became somewhat
smaller during her last menstrual period but
then increased in size again. The mass is now
3 cm in diameter, smooth in contour, and is
slightly tender.
Which of the following is the most appropriate next
step in diagnosis?
A.
B.
C.
D.
E.
A mammogram.
An excisional biopsy.
Ultrasonography of the mass.
A needle aspiration. (*)
Re-examination in four weeks.
???
To minimize odds of guessing correctly
when only five options are given
 Extended Matching format has come into
more use by medical test committees
 Referred to as R-type questions
Extended Matching (R-type)
Theme: Nerves
A.
B.
C.
D.
E.
F.
Median Nerve
Ulnar Nerve
Posterior Interosseous Nerve
Anterior Interosseous Nerve
Axillary Nerve
Radial Nerve
G.
H.
I.
J.
K.
L.
Accessory nerve
Musculoscutaneous Nerve
Long Thoracic Nerve
Deep Branch of Median Nerve
Deep Branch of Ulnar Nerve
Sensory Branch of Radial Nerve
For each of the following patients with motor/sensory impairment,
select the nerve that is most likely to be involved.
•
A construction site worker fell and landed on his arm. He was initially
treated by a bone setter but is now complaining of an inability to abduct
the arm.
E
Extended Matching (R-type)
Theme: Nerves
A.
B.
C.
D.
E.
F.
Median Nerve
Ulnar Nerve
Posterior Interosseous Nerve
Anterior Interosseous Nerve
Axillary Nerve
Radial Nerve
G.
H.
I.
J.
K.
L.
Accessory nerve
Musculoscutaneous Nerve
Long Thoracic Nerve
Deep Branch of Median Nerve
Deep Branch of Ulnar Nerve
Sensory Branch of Radial Nerve
For each of the following patients with motor/sensory impairment,
select the nerve that is most likely to be involved.
•
A kitchen worker accidentally injured himself at work, sustaining a
penetrating injury over the forearm. Repair was done in the A&E
department. He now complains of inability to actively flex the IP joint of
the thumb and the DIP joint of the index finger.
D
Extended Matching (R-type)
Theme: Nerves
A.
B.
C.
D.
E.
F.
Median Nerve
Ulnar Nerve
Posterior Interosseous Nerve
Anterior Interosseous Nerve
Axillary Nerve
Radial Nerve
G.
H.
I.
J.
K.
L.
Accessory nerve
Musculoscutaneous Nerve
Long Thoracic Nerve
Deep Branch of Median Nerve
Deep Branch of Ulnar Nerve
Sensory Branch of Radial Nerve
For each of the following patients with motor/sensory impairment,
select the nerve that is most likely to be involved.
An 18 year old martial-arts student blocked an opponent's kick with his
forearm, during a kung-fu tournament. He now complains of an inability to
extend his fingers & thumb.
C
A cyclist was knocked down while riding. He sustained a mid-shaft fracture
of the humerus. Physical examination reveals he is unable to extend his wrist,
F
thumb & fingers.
NBME, p. 74
If you want to start using R-types
a template is provided in Appendix IV
 Start by selecting your theme
•
select from the list of topics provided or pick
another
NBME, p. 74
Step Two in writing R-type questions
 Compose a lead-in
•
"For each patient described below, select the most likely
(diagnosis / abnormality / injury site / treatment / etc.
NBME, p. 74
Step Three
 Generate a list of Options fitting the theme
•
options should be single words or very short phrases
•
the list is specified in alphabetical order
• unless there is another inherently logical order
Criteria for a good set of options
NBME, p. 75
 Use single words or very short phrases
 Make all homogeneous (e.g., all are diagnoses, or management
steps, or anatomical sites, or vitamins, or enzymes)
 If desired, can be labeled areas in a graph or a picture, or
expressed in tabular form
 Include all relevant options that are appropriate for the examinees'
stage of learning
 Avoid subtle distinctions and uncommon diagnoses if assessing
undergraduates (although might be acceptable for postgraduates)
 Use as few as three options if it is only appropriate to do so for
some questions, while for others, use a list of up to 26 if needed
See Appendix IV
NBME, p. 75
Some Topics Used as a Basis for Option Lists
Arteries
Nerves
Muscles
Amino Acids
Peptides
Hormones
Enzymes
Cell Components
Cell Types
Blood Components
Molecules
Karyotypes
Proteins
Lipids
Pathogens/Bacteria/Fungi
Viruses
Cytokines
Toxins
Vitamins/Minerals
Diagnoses
Connective Tissue Types
Anatomic Structures
Endocrine Structures
Neurotransmitters
Metabolic Defects
Immune Disorders
Motor System Components
Cardiac Structures
Organelles
Congenital Anomalies
Segments of the Spinal Cord
Central Nervous System Components
Secretory Products
Extracellular Matrix Components
Management Alternatives
Drugs/Drug Classes
Pathologic Processes
Pathophysiologic States
Electrolyte Abnormalities
Diagnostic Tests
NBME, p. 74
Step 4
 Write the items
•
All items should be similar in structure
•
Short patient vignettes are very appropriate
NBME, p. 74
Step 5
 Review the items
•
is there a single "best" answer in the option list?
•
are there at least four others in the option list
that are reasonably seductive?
•
does a colleague agree with your conclusions?
Wrap Up to First Section
If your learning objectives are clinical
reasoning & decision making skills . . .
you might wish to use some of these
ideas.
What & How you assess will influence
 how students study
 what students learn
 how well students develop clinical reasoning
& problem solving skills
Recall the workshop’s objectives
 “At the end of this workshop you will be able to:
•
determine which tests & techniques best help students
develop clinical and scientific reasoning skills
• and which simply assess & reinforce their ability to
memorise factual information
•
compose MCQs that measure clinical & scientific
reasoning abilities”
It is Apparently Clear
 You accomplished these objectives.
 Congratulations!
 Thank you for attending
Second Section
More on R-type items
(Extended Matching)
@ 12 September 2002
Relationship of A-type to R-type Items
 Can change an A-type into a R-type
 See the following illustration
NBME, p. 97
A 5-option A-type item
A patient with the classic phenotypic features of trisomy 21
(Down syndrome) has 46 chromosomes on each of 100
metaphase karyotypes.
Which of the following is the most likely explanation for this
finding?
A.
B.
C.
D.
E.
Deletion
(*)
Mosaicism
Somatic mutation
Translocation
Undetected trisomy
NBME, p. 97
To transform this A-type into a R-type (extended-matching) item:
A.
B.
C.
D.
E.
Deletion
Mosaicism
Somatic mutation
Translocation
Undetected trisomy
For each patient with genetic abnormalities, select the genetic
pattern that is most likely to be involved.
A patient with the classical phenotypic features of trisomy 21
(Down syndrome) has 46 chromosomes on each of 100
metaphase karyotypes.
NBME, p. 97
Additional options make it more challenging
A.
B.
C.
D.
E.
Deletion
Genomic imprinting
Mosaicism
Pleiotropy
Reduced penetrance
F. Somatic mutation
G. Translocation
H. Undetected trisomy
I. Variable expressivity
For each patient with genetic abnormalities, select the genetic
pattern that is most likely to be involved.
Four Components for R-type items




Theme
Option List
Lead-in
2 or more items (preferably in vignette form)
NBME, p. 71
Theme:
Fatigue
Options:
A.
B.
C.
D.
E.
Acute leukemia
Anemia of chronic disease
Congestive heart fatigue
Depression
Epstein-Barr virus infection
F. Folate deficiency
G. Glucose 6-phosphate
dehydrogenase deficiency
H.
I.
J.
K.
L.
Hereditary spherocytosis
Hypothyroidism
Iron deficiency
Lyme disease
Microangiopathic hemolytic
anemia
M. Miliary tuberculosis
N. Vitamin B12 (cyanocobalamin) deficiency
Lead-in:
For each patient with fatigue, select the most likely diagnosis.
NBME, p. 71
Lead-in:
For each patient with fatigue, select the most likely diagnosis.
Stems:
• A 19-year-old woman has had fatigue, fever, and a sore throat for the
past week. She has a temperature of 38.3 C (101 F), cervical
lymphadenopathy, and splenomegaly. Initial laboratory studies show
a leukocyte count of 5000/mm3 (80% lympho-cytes, with many
lymphocytes exhibiting atypical features). Serum aspartate
aminotransferase (AST, GOT) activity is 200 U/L. Serum bilirubin
concentration and serum alkaline phosphatase activity are within
normal limits.
Answer: E
NBME, p. 71
2. A 15-year-old girl has a two-week history of fatigue and back
pain. She has widespread bruising, pallor, and tenderness
over the vertebrae and both femurs. Complete blood count
shows hemoglobin concentration of 7.0 g/dL, leukocyte count
of 2000/mm3, and platelet count of 15,000/mm3.
Answer: A
Sample Lead-ins & Option Lists
NBME, p. 40
For each of the following patients, select the ________
that is (or is most likely to be) (present / normal /
abnormal)?
Options sets could include:
-sites of lesions
-list of nerves
-list of muscles
-list of enzymes
-list of hormones
-types of cells
-list of neurotransmitters
-list of toxins,
-list of molecules
. . . or
NBME, p. 74
For each of the following patients, select the ______ that
would be expected to be found?
Options sets could include:
-list of laboratory results
-list of additional physical signs
-DNA analysis results
-serum levels
-results of microscopic examination of fluids,
muscle or joint tissue
. . . or
NBME, p. 74
For each of the following patients, which _______ is
most likely (defective or deficient or nonfunctioning)?
Options sets could include
-list of enzymes;
-list of feedback mechanisms;
-list of endocrine structures, dietary elements,
vitamins.
. . .or
NBME, p. 74
For each of the followin patients, select the (mechanisms,
medications, drugs, agents) that is the most likely cause?
Options sets could include
-list of underlying mechanisms of the disease;
-medications that might cause side effects;
-drugs or drug classes;
-toxic agents;
-hemodynamic mechanisms, viruses, metabolic
defects.
. . . or
NBME, p. 40
For each of the following patients, select the _____ that
should be administered
Options sets could include:
-drugs;
-vitamins;
-amino acids;
-enzymes;
-hormones.
NBME, p. 40
For each of the following patients, select the most
appropriate next step in management
Options sets could include list of:
-pharmacologic therapies;
-lab studies;
-or a mix of treatment sets;
-or a mix of laboratories studies;
-or a combo of the above two.
NBME, p. 40
For each of the following patients with (chief complaint),
select the most likely diagnois
Options sets could include a list of
diagnoses:
Microbiology NBME, p. 77
Example good & poor item stems using the same option list.
Theme: Pathogens (viruses and bacteria)
A.
B.
C.
D.
E.
F.
G.
H.
Adenovirus
Aspergillus fumigatus
Bacillus anthracis
Candida albicans
Chlamydia psittaci
Coccidioides immitis
Coronavirus
Corynebacterium
diphteriae
I. Coxiella burnetii
J. Coxsackievirus
K. Epstein-Barr virus
L. Haemophilus influenzae
M. Histoplasma capsulatum
N. Mycobacterium tuberculosis
O. Mycoplasma pneumoniae
P. Neisseria gonorrhoeae
Q. Neisseria meningitidis
R. Pneumocystis carinii
S. Rhinovirus
T. Streptoccus pneumoniae
U. Streptococcus pyogenes
(group A)
NBME, p. 77
For each patient with fever, select the pathogen most likely to
have caused his/her illness.
1. A 7-year-old girl has a high fever and a sore throat. There is
pharyngeal redness, a swollen right tonsil with creamy exudate,
and painful right submandibular lymphadenopathy. Throat
culture on blood agar yields numerous small β-hemolytic
colonies that are inhibited by bacitracin.
Answer: U (Question requires application of knowledge)
2. An encapsulated gram-positive organism that usually grows
in pairs or short chains
Answer: T (Question assesses recall of a questionably useful
fact for medical students)
NBME, p. 78
Theme: vitamins and minerals
Options.
A.
B.
C.
D.
E.
F.
G.
H.
Vitamin A
Vitamin B1
Vitamin B2
Vitamin B6
Vitamin C
Vitamin D
Vitamin E
Vitamin K
I.
J.
K.
L.
M.
N.
O.
P.
Biotin
Copper
Folate
Iodine
Iron
Magnesium
Niacin
Zinc
Lead-in:
For each patient with clinical features caused by metabolic abnormalities,
select the vitamin or mineral that is most likely to be involved.
NBME, p. 78
For each patient with clinical features caused by metabolic
abnormalities, select the vitamin or mineral that is most likely
to be involved
1. A 70 yr old widower has ecchymoses, perifollicular
petechiae, and swelling of the gingiva. His diet consists
mostly of cola and hot dogs.
Answer: E
(requires a diagnosis, a high order ability)
2. Involved in clotting factor synthesis.
Answer: H
(requires recall of a fact)
Example of Flawed Extended-Matching
A. is motion sickness
B. have no effects on people
C. indirectly increase CO2
D. cause death
E. increased odor sensitivity
F. is a reduction in visibility
G.esthetics, economics, health
H. products of fossil fuel
combustion
Microbiology
NBME, p. 72
I. are completely controlled
J. cause plant and eye damage
K. are negligible
L. increase risk of skin cancer
M. cannot be controlled
N. excess acute respiratory
illness among children
O. contrary to public opinion
1. Factors that people consider when evaluating air quality
2. A principal effect of particulate matter in air
3. The products of photochemical smog
Problems:
What is the common theme? What common lead-in could be used?
NBME, p. 80
Birth Gender
A. 1980
B. 1975
C. 1970
D. 1965
E. 1960
F. 1955
G. 1950
H. 1945
I. 1940
J. 1935
Male
Female
Female
Female
Male
Female
Female
Male
Male
Female
Occupation
Pregnant Childhood
Diseases
History
Medical
Allergy
Student
Painter
Teacher
Lawyer
Painter
Clerk
Nurse
Executive
Driver
Homemaker
—
Yes
Yes
No
—
No
No
—
—
No
None
None
None
None
None
Diabetes
None
Hypertension
Splenectomy
Gastritis
Egg products
Penicillin
None
Gelatin products
Tetanus toxoid
None
None
None
None
Sulfonamides
None
None
None
None
Measles
Mumps
Varicella
Measles
Rubella
Unknown
For each vaccination, select the patient profile that represents its most appropriate use.
1. Measles vaccine
2. Meningococcal vaccine
Problem:
This is clinically backward; given a vaccine, pick the patient
NBME, p. 82
Example R-type for Anatomy
Theme: Arteries
A. Left anterior cerebral artery
B. Right anterior cerebral artery
C. Left middle cerebral artery
D. Right middle cerebral artery
E. Left posterior cerebral artery
F. Right posterior cerebral
artery
G. Left lenticulostriate arteries
H. Right lenticulostriate arteries
Lead-in:
For each patient with neurologic abnormalities, select the artery
that is most likely to be involved.
NBME, p. 82
1. A 72-year-old right-handed man has weakness and hyperreflexia of the right lower limb, an extensor plantar response
on the right, normal strength of the right arm, and normal
facial movements.
Answer: A
2. A 68-year-old right-handed man has right spastic hemiparesis, an extensor plantar response on the right, and
paralysis of the lower two-thirds of his face on the right. His
speech is fluent, and he has normal comprehension of verbal
and written commands.
Answer: G
NBME, p. 83
Example: R-type for Pharmacology
Theme: Drugs
A. Acetaminophen
B. Amiodarone
C. ACE inhibitors
D. Aspirin
E. Atenolol
F. Bleomycin
G.Cytosine arabinoside
H. Furosemide
I. Metronidazole
J. Nalidixic acid
K. Nitrofurantoin
L. Penicillin
M. Prednisone
N. Procainamide
O. Propranolol
P. Sulfasalazine
Q. Tetracycline
R. Verapamil
Lead-in:
For each patient, select the drug most likely to have caused
the adverse effect.
NBME, p. 83
1. A 56-year-old man with recurrent ventricular arrhythmias
began taking an antiarrhythmic drug 5 months ago. He now has
progressive dyspnea, cough, and low-grade fever. Erythrocyte
sedimentation rate is increased. X-ray film of the chest shows a
diffuse interstitial pneumonia. Pulmonary function tests show
that diffusing capacity for carbon monoxide is decreased.
Answer: B
2. A 62-year-old man with chronic obstructive pulmonary
disease begins therapy with an antihypertensive drug. Two
weeks later, he has marked worsening of dyspnea and
clearly audible wheezing.
Answer: O
NBME, p. 84
Example: R-type items for Physiology
Theme: Arterial Blood Gas Findings
A.
B.
C.
D.
E.
F.
G.
pH
7.15
7.15
7.30
7.40
7.50
7.50
7.50
PO2 mm Hg
98
98
56
100
100
100
56
PCO2 mm Hg
33
24
80
40
33
24
33
HCO3 – mEq/L
11
8
38
25
25
18
25
Lead-in:
For each patient described below, select the most likely arterial
blood gas finding.
NBME, p. 84
For each patient described below, select the most likely arterial
blood gas finding.
1. A 22-year-old man with a 3-week history of polyuria and
polydipsia has had nausea, vomiting, and decreased responsiveness for the past 12 hours. Urinalysis (dipstick) shows 4+
glucose and 4+ ketones.
Answer: B
2. A 25-year-old woman is brought to the emergency department
12 hours after a suicide attempt. She took approximately 100
500-mg aspirin tablets.
Answer: F
NBME, p. 85
Example: R-type items for Diagnosis
Theme: Back Pain
A. Ankylosing spondylitis
B. Intervertebral disc infection
C. Multiple myeloma
D. Myofacial pain
E. Osteoporosis
F. Spinal stenosis
G. Spondylolysis
H. Tuberculosis of the spine
Lead-in:
For each patient with back pain, select the most likely
diagnosis.
NBME, p. 85
1. A 26-year-old man has insidious onset of low back pain and
early morning stiffness. The pain alternates from side to side and
occasionally radiates into the buttocks and back of the thighs,
but not below the knees. The patient has acute anterior uveitis,
diffuse low back and sacroiliac tenderness, and restricted range
of motion at the hips. His erythrocyte sedimentation rate is 40
mm/h; latex fixation test is negative; and mild hypoproliferative
anemia is present.
Answer: A
2. Twelve hours after being struck from the rear while driving her
car, a 28-year-old woman has vague cervical and lumber pain
associated with headache and restricted cervical mobility. She is
now very anxious. Rope-like bands of muscle are present in the
lumbar area and over the left buttock; the bands are painful.
Answer: D
NBME, p. 88
Example: R-type items for Management:
Theme: diagnostic testing
A. Test of the stool for occult blood
B. Fasting serum glucose level
C. Hemoglobin level
D. Prostate-specific antigen level
E. Serum cholesterol level
F. Serum iron level
G. Thyroid function test
H. Exercise tolerance test
I. Digital prostate examination
J. ECG
K. Spirometry
L. X-ray film of the chest
M. Sigmoidoscopy
Lead-in:
For each patient who comes to the physician for a health
maintenance examination, select the most appropriate
diagnostic study.
NBME, p. 88
1. A 22-year-old man who weighs 89 kg (196 lb) and is 175 cm
(69 in) tall has smoked one pack of cigarettes daily for 8 years;
he does not exercise. His last examination was 5 years ago.
His father had a myocardial infarction at the age of 48 years.
Physical examination shows no abnormalities.
Answer: E
2. A 28-year-old woman who weighs 70 kg (154 lb) and is
173 cm (68in) tall has smoked one pack of cigarettes daily for
12 years; she does not exercise. Her last examination was 5
years ago, though she had a Pap smear 9 months ago that
showed normal results. Her father had a myocardial infarction
at the age of 48 years. Her grandmother was diagnosed with
colon cancer at the age of 62 years. Physical examination
showed no abnormalities.
Answer: E
NBME, p. 81
Review: the steps for writing extended-matching items
1. Identify the common theme for the set.
2. Write the common lead-in for the set.
3. Prepare a list of options that fit the theme.
4. Write the items, preferably in vignette format.
5. Review the items
a. place emphasis on application not recall
b. remember to avoid the same writing mistakes
that apply to A-type questions
Pick n best choices from N options
(Pick n)
If the next best step or best option is not a
singular case (or is not known to singular)
 To simulate clinical situations one must allow
for instances


when multiple next steps are realistic and optimal
where is no unanimity among experts as to what is
the best or most likely
 To accommodate for these instances use another
format

i.e., Pick n choices from N options
NBME, p. 100
Example of Pick n
A. Diabetic polyneuropathy
B. Huntington’s disease
C. Lateral medullary syndrome
D. Lead encephalopathy
E. Medulloblastoma
F. Multiple sclerosis
G. Parkinson’s disease
H. Pontine glioma
I. Tabes dorsalis
J. Wernicke’s encephalopathy
A 50-year-old man has the gradual onset of mental confusion,
disorientation, and loss of short-term memory. He has a left
footdrop. A blood smear shows microcytosis and basophilic
stippling of erythrocytes.
Select the two most likely diagnoses
NBME, p. 100
With the previous item, there would be disagreement about which
diagnoses are likely, but the task becomes clearer if the
examinee is told to select the two most likely diagnoses.
The options can be laid out along a diagnostic continuum
something like the following:
F
C
E D
A G
H
B
__________________________________________________
Least likely diagnosis
Most likely diagnosis
Even if disagreement exists about H & B, there is agreement that both
are more likely than all the others.
NBME, p. 100
Additional examples for Pick n
A. Calcium
B. Fluoride
C. Folic acid
D. Iron
E. Vitamin A
G. Vitamin B6
H. Vitamin B12 (cyanocobalamin)
I. Vitamin C
J. Vitamin D
K. Vitamin E
For each child select the appropriate vitamin or mineral
supplements.
1. A 1-month-old infant is brought to the physician for a wellchild examination. He has been exclusively breast-fed, and
examination shows normal findings. (SELECT 2 SUPPLEMENTS)
Answer: B, J
NBME, p. 100
2. A 6-year-old girl has cystic fibrosis. She has been taking no
Medications. (SELECT 3 SUPPLEMENTS)
Answer: E, J, K
NBME, p. 101
Additional examples of Pick n
A. Analysis and culture of cerebrospinal fluid
B. Blood culture
C. Complete blood count
D. Examination of the stool for leukocytes
E. Measurement of serum electrolyte levels
F. Urinalysis
G.Urine culture
H. X-ray film of the abdomen
I. X-ray film of the chest
For each child with fever, select the appropriate initial diagnostic
studies.
NBME, p. 101
1. A previously healthy 1-year-old girl is brought to the emergency department because of fever for 1 day. Her temperature is
41 C (105.8 F). She is otherwise asymptomatic. Physical
examination shows no abnormalities. (SELECT 4 STUDIES).
Answer: B, C, G, I
2. A previously healthy 10-day-old newborn is brought to the
emergency department because of fever for two hours. He was
born at term after an uncomplicated pregnancy. His temperature
is 39 C (102.2 F). Physical examination shows no abnormalities.
(SELECT 6 STUDIES)
Answer: A, B, C, E, G, I
NBME, p. 101
3. A 7-year-old boy with sickle cell disease is brought to the
emergency department because of fever for 1 day and chest
pain for 1 hour. His temperature is 39.5 C (103.1 F). Breath
sounds are slightly decreased in the right lower lung; he is not
in respiratory distress. (SELECT 3 STUDIES)
Answer: B, C, I
PATIENT CHART
Patient History
Sex:
Current age:
NBME, p. 102
male
28 years
Chief complaint:
health maintenance examination
Social history:
Marital status:
Occupation:
Alcohol:
Smoking:
Exercise prgm:
single
computer programmer
2-4 beer/weekend
5-10 cigarettes daily from age 16-24
sedentary
Medical history:
Childhood:
Immunizations:
Screening:
obese since grade school
all childhood immunizations; last tetanus toxoid age 15;
no immunizations since childhood.
no physician visits since college
Family history:
Parents:
Siblings:
Children:
father age 57; hypertensive mother age 55; obese & hypertensive
none
none
Current meds:
none
Allergies:
none
NBME, p. 102
Physical Examination
Height:
Weight:
178 cm (70 in)
134 kg (295 lb)
Vital signs:
Blood pressure:
Pulse:
Respirations:
148/86 mm Hg
90/min
16/min
Skin:
erythematous rash in groin
Abdominal:
obese
Laboratory studies:
none ordered
NBME, p. 103
1. For the patient whose chart is shown, select the conditions for
which he is at increased risk.
SELECT 4 CONDITIONS
A. Alcoholism
B. Colon cancer
C. Coronary artery disease
D. Diabetes mellitus
E. Hemochromatosis
F. HIV
G. Hypertension
H. Hypothyroidism
I. Osteoarthritis
J. Skin cancer
K. Thyroid cancer
L. Urinary tract infection
NBME, p. 103
2. For the patient whose chart is shown, select the most
appropriate health maintenance interventions.
SELECT 4 INTERVENTIONS
A. Complete blood count
B. Dietetic counseling
C. Exercise counseling
D. Fasting serum lipid profile
E. Hepatitis B immunization
F. HIV testing
G. Influenza immunization
H. Serum urea nitrogen (BUN)
& creatinine level measmnts
I. Tetanus toxoid vaccine
J. Thyroid function tests
K. Urinalysis
L. X-ray film of the chest
Summary
These are the common selected-response
formats that medical test committees use
Following the guidelines for their items’ development
(provided in the foregoing), one can better meet the
principle criteria for all assessments:
1. Objective
(correct or keyed response is indeed correct)
2. Reliable
(consistently reproducible)
3. Valid
(adequately measures intended trait/ability)
4. Defensible
(when challenged by examinees/society)
5. Efficient
(time and cost is justified)
6. Practical
(usable within educational/assessment context)
7. Educational
(aids in driving learning in desired direction)
There are additional assessment formats
 Including other selected-response formats & a selection of
constructed-response formats
 The Hong Kong Consortium for Sharing Medical Student
Assessment Banks & its IDEAL system platform


supports use of selected-response items discussed in this presentation
but also:
• Key Features (special case of Pick n)
• OSCE
• short answer & modified essay
 In the future, other presentations will be prepared on these
additional supported formats
For now, the Consortium hopes these guidelines
& presentation prove to be useful to each of you.
We again gratefully acknowledge the use of Swanson, Case &
NBME’s materials in the preparation of this training package