Transcript X A R MR Item Writing S Case C Hazlett

Workshop Aid for Writing Student Assessments in Medicine
This PowerPoint presentation is prepared as an aid for trainers in faculty enhancement workshops,
specifically assessment writing workshops within institutions that are members of the Hong Kong
International Consortium for Sharing Medical Student Assessment Banks.
The primary focus of the presentation is to develop writing skills for A-type and R-type items that can
assess not only students’ recall abilities but also their clinical reasoning, problem solving and decision
making skills.
Principles suggested follow those identified in S M Case & D B Swanson text Constructing Written Test
Questions For the Basic and Clinical Sciences, 3rd Ed., NBME: Philadelphia, 2002. With the kind
permission of the authors and the National Board of Medical Examiners, selected items and templates have
been included and then supplemented with additional examples drawn from the IDEAL (International
Database for Enhanced Assessments and Learning). All items extracted from the above text are so noted.
In providing example items to illustrate item writing errors or recommended item writing principles, this
presentation frequently provides more than one example item for each point. The additional items relate to
various disciplines (e.g. Anatomy, Physiology, Medicine, Psychiatry, Surgery).
Trainers using this presentation will wish to hide some of the extra examples, the choice being a function of
the disciplinary orientation of the registrants in one’s workshop. To hide a slide click ‘Slide Show’, then
click ‘Hide Slide’ in order to skip the slide during a particular PowerPoint presentation.
Workshop Aid for Writing Student Assessments in Medicine (cont’d)
The presentation has suggested breaks for one’s workshop. It also provides effective examples of how one
can convince registrants that providing a context enables the student to remember more in less time. Once
this is proven to the registrant (by running the simple experiment provided), the need for and the advantages
of providing a context (i.e. by embedding a clinical vignette into the item) becomes clear. [The author has
run the experiment as provided in this presentation (see slides 125 to 141) in many workshops in different
countries; its utility in convincing registrants of the importance of setting a question within a meaningful
context is clearly evident. For added flexibility alternate tasks are provided for the experiment, so that one
can repeat the experiment successfully in more than one workshop within the same locale.]
The total presentation includes 320 slides covering the following topics:
1.
Pertinent research findings that have established the key components of quality items.
2.
Nature, strengths and or limitations of X type and A type items.
3.
Typical item writing errors and basic requirements for writing quality items
4.
Advantages of using vignettes to provide a clinical context
5.
Nature, strengths & limitations of R type items & an introduction to Multiple Response items.
6.
NBME templates for writing items.
On behalf of the Hong Kong International Consortium, I am indebted to Drs Case & Swanson and the
NBME in giving me permission for this presentation. If you wish to use this PowerPoint file in your own
faculty enhancement workshops, please ensure that all slides acknowledging the foregoing colleagues
and the National Board of Medical Examiners are always included, in particular slides 36-37.
Clarke Hazlett, CUHK
CPC Chair, Hong Kong International Consortium
Composing Selected Response Items
Assessing Medical Students’ Clinical & Scientific
Recall & Reasoning & Decision Making Abilities
C. B. Hazlett
Based on Swanson, DB & Case SM Constructing Written Test Questions
for the Basic and Clinical Sciences, 3rd Ed. NBME, Philidelphia, 2001.
Objectives
 At the end of this workshop you will be able to:
• determine which tests & techniques best help students
develop clinical and scientific reasoning skills
• and which simply assess & reinforce their ability to
memorise factual information
• compose selected-response items that measure clinical
& scientific reasoning abilities
Overview:
• Highlight pertinent research
• evaluate selected-response to other assessment formats
• compare T/F and Best One of ‘N’ formats
• Review basic do’s and don’ts in writing items
• Clinical vignettes
• useful aid in assessing clinical reasoning
• recommended format
• useful templates for clinical student assessments
• helpful templates for integrating clinical and basic science
student assessments
Overview (cont’d):
 Practice in writing items that measure
• clinical reasoning
• integration of clinical and relevant basic science content
 Practice in critiquing items
 Understand Extended Matching format
 Develop & use Extended Matching Items
 Understand Pick n Best Choices from N Options
format
Important:
 Feel warmly welcomed & very comfortable
• to challenge, question, suggest, seek clarification
• at any point throughout the workshop
What’s the relevant research?
1. Key Research Finding:
 What is best learned is not what’s taught
 What is best learned is what’s tested
Examinations drive learning among medical undergraduates
 If one teaches clinical & scientific reasoning,
but
• tests for only recall of scientific & clinical facts,
then
• students will learn primarily facts
• and few reasoning or problem solving skills
Thus, the key criterion for good exams
 Create tests that assess what you want your
students to learn
2. Research: components affecting test’s validity:
 Content
 Format
Both influence if one obtains a valid measure
of either clinical & scientific reasoning or
an ability in memorising clinical & scientific facts
Related Research Findings:
 Some test formats are regarded by students to
tap certain skills better than other formats
• Research: this is not necessarily true
• But the perception by students remains / is real
Perceived purposes of test formats
 Essays
 Reasoning, synthesis, organisation
 Short answer  Facts: Recall (not recognition)
 Bedside orals,  Clinical skills (non standardised), inViva
& depth probing (in any direction)
 OSCE
 Standardised clinical & practical
skills
Thus, 2nd criterion for good exams
 Use a test format that appears to the student to
tap the skill that you want to be learned
• in assessment jargon: ensure there is “face validity”
• if students think test format requires reasoning skills,
their study efforts become directed at these skills
Additional Research Findings:
 Examination results improve as predictors of future
clinical competence if
• based on a representative sampling of pertinent
clinical skills and scientific knowledge (validity)
• based on a large sample of these attributes
(precision)
• sample size can be increased by using long tests, or by
using more short tests (latter has some advantages)
Thus, 3rd criterion for good exams
 Test large samples of the skills and knowledge
that you want the students to learn
• important for separating those who do & don’t meet the bar
of minimally acceptable performance (e.g., pass/fail)
Thus, 4th criterion for good exams
 Enable representative sampling
• Write a large pool of items from which good test
items can be sampled
• validity in predicting future clinical competency improves
Review: The “Musts”
 1. Test for the ability that needs to be acquired
 2. Use a test format that appears to students to
test what you asked/want them to learn
 3. Test a representative sample of the skills
 4. Test a lot of these skills
Problems
Each test format has some shortfalls
 Essays
 Poor sampling; poor reliability; long time
needed to mark
 Short answer
 Often only facts tested; reliability and time
for marking improved (not best);
 Orals, Viva
 Poor sampling; poor reliability; not
equivalent across students; time needed for
administration is longer
 OSCE
 Better sampling and reliability (not best);
much preparation time in relation to sampling
One additional test format: MCQ
 Not suitable for testing practical or clinical skills
 Came into wide spread use because of some
shortfalls in essays, orals/viva
• easier to administer, easier to mark
 Repeatedly criticised
• doesn’t measure even memorisation, just recognition
• students can develop their own item banks
Research findings on the MCQ
 If composed poorly, criticisms of MCQs are valid
Research findings on the MCQ
 If properly written and developed
• can test clinical reasoning & problem solving abilities
• best reliability among of all test formats
• easiest to get more of a representative sample
• easiest to get a more precise measure
• best predictor of clinical competency
• (as rated by clinical supervisors)
2 Common Types of Selected-response questions (MCQ’s)
 True/False or Multiple T/F: categorical (type ‘X’)
 One Best Answer: probabilistic
(type - ‘A’ & ‘R’)
Types of MCQ’s
Among selected-response items (MCQs)
• probabilistic format has become most prevalent in
medical schools & licensing bodies
True / False
T/F (X - type) MCQ’s
 Often assumed that:  Research has found that:
• simple to prepare
• difficult to write well
• easy to understand
• more likely to be misinterpreted
• little ambiguity
• more subject to ambiguity
• more content sampled
• often sample less content
• widely adaptable
• more likely to be discarded by medical
test committees after review
To answer T/F ( X-type) items:
NBME, p.15
 1. Examinee needs to know the content
 2. May also have to decide to what extent a choice is
‘true’
• if options are not completely true or completely
false, the students have to guess what the item writer
had in mind
• this latter judgement can be unrelated to their clinical or
scientific expertise
Criteria for T/F MCQ’s
 Stems must not have
 imprecise phrases
• ‘is associated with’
• ‘is useful for’
• cues
• ‘may be’
• ‘could be’
• vague terms
• ‘usually’
• ‘frequently’
NBME, p.16
 Options must be:
• absolutely true or false
A word of caution:
NBME, p.18
(regarding trying to write good T/F questions)
Trying to avoid ambiguity in T/F items, medical item writers
often find themselves being pushed toward assessing the
recall of isolated facts -
Something normally to be avoided
Caution:
A true/false decision is not necessarily a trivial exercise
( e.g., deciding if a verdict of guilty or innocent is correct decision)
But writing a T/F question to measure the complex skills
of why one arrived at a T/F decision is very difficult.
Illustrations of these principles in writing
T/F Items
(“X”-type constructed-response)
Acknowledgement
• Portions of the following are reprinted, with permission, from
Constructing Written Test Questions for the Basic and Clinical
Sciences, copyright © 2002 by the National Board of Medical
Examiners®.
[ see web site at
http://www.nbme.org/PDF/ItemWriting_2003/2003IWGwhole.pdf ]
• Other illustrative items are taken from IDEAL’s bank.
Acknowledgement
Hong Kong International Consortium for Sharing Medical
Student Assessment Bank recognises the Swanson & Case
text as a recommended reference standard for item writers
Swanson DB & Case SM, Constructing Written Test
Questions for the Basic and Clinical Sciences. 3rd
Edition Revised, NBME: Philadelphia, 2002.
Example:
A Good T/F Item
NBME, p.14
Which of the following is/are
X-linked recessive conditions?
A. Hemophilia A (classic hemophilia)
B. Cystic fibrosis
C. Duchenn’s muscular dystrophy
D. Tay-Sachs disease
B
D
Totally Wrong
(False)
A
C
Totally Correct
(True)
O&T
(Multiple T/F item)
The following are autosomal dominant conditions.
A.
B.
C.
D.
E.
Totally False
A
C
D
fibrous dysplasia
osteogenesis imperfecta
osteoporosis
Duchenne’s muscular dystrophy
achondroplasia
Totally True
B
E
A Flawed T/F Item
Example:
Which of the following statements about
Osteoarthritis are True and which are False?
A. there is a genetic pre-disposition
B. males are more affected than females
C. hip joints are more affected than knee
D. obesity is a risk factor in its development
False
A
B
C
True
D
A,B,C are not absolutely true or false.
Experts would only unanimously agree that D is true.
NBME, p.14
True statements about cystic fibrosis
(CF) include:
A. The incidence of CF is 1:2000
B.
C.
D.
Children with CF usually die in their teens
Males with CF are sterile
CF is an autosomal recessive disease
A, B, and C are not absolutely true or false; experts can’t agree
Examine 2nd major type of MCQs
Best One of ‘N’ Choices
( “A”-type Selected-response)
A-type writers also face problems in
composing a good item
If you can avoid making these
common mistakes, students’ test
wiseness will not be a major issue
NBME, p.19?
1. Avoid grammatical cues:
• e.g., one or more distracters may not
follow grammatically from the stem
The following question is not necessarily in your own
specialization & thus the content may not be readily familiar.
If so, this will help illustrate the point . . .
much like a typical medical student you too can
figure out that some options could not possibly be the
correct answer; in so doing, you have a better chance
selecting correct answer without knowing the answer.
NBME, p.19
A 60-year-old alcoholic derelict is brought to
the emergency department by police. After
ascertaining that the airway is open, the first
step in management should be intravenous administration
of
A
B
C
D
E
examination of cerebrospinal fluid
glucose with B1 (thiamine)
CT scan of the head
phenytoin
diazepam
As options “A” & “C” don’t fit grammatically with stem,
neither would be the correct answer
NBME, p. 22-26 ?
2. Avoid irrelevant techniques for making an item
more difficult
 options are sometimes made unnecessarily complicated

- numeric data are not stated in a consistent unit


A. 20 mg
B. 40 g
C. 45 oz
D. 50 oz
- options are in non logical (or non sequential) order

A. 120 ml
B. 100 ml
C. 150 ml
D. 115 ml
In valid tests, difficulty level depends on the
level of clinical & scientific reasoning
NBME, p. 22-26 ?
3. Avoid using these imprecise terms in the
stem and/or choices:





Most of the time
Usually
Frequently
Likely to occur
Probably




Commonly
Associated with
Rarely
Almost never
Consider the variation writers have in what these terms mean.
NBME, p. 22-26 ?
4. Avoid other common item writer mistakes:
a. logical cues (subset of options is collectively exhaustive)
b. absolute terms (‘always’, ‘never’)
c. long correct answer (phrased to be qualified)
d. word repeats (in stem and the correct choice)
e. convergence strategy
(correct answer includes the most elements that are in
common with other options)
Example items for each of these errors in writing
questions are illustrated in your Appendix 1
Having seen what not to do,
examine what should be done
Criteria to meet in writing A-type items
 Stem asks a question & a knowledgeable student can
answer question without looking at set of answer choices
 A common continuum underlies all choices of answers

thereby enabling a “best” answer to be found
 All distracters are plausible
 All distracters are relatively same length as the answer
 Question deals with important concept(s)

i.e., does not deal with trivial fact(s)
Practice in Diagnosing Quality
of
Type-A Items
Example of a poor A-Type item
NBME, p.17
Which of the following is true about
pseudogout?
A. Occurs more frequently in women.
B. It is seldom associated with acute pain in a joint.
C. It may be associated with a finding of
chondrocalcinosis.
D. It is clearly hereditary in most cases.
E. It responds well to treatment with allopurinol.
What’s wrong with this question?
Flaw #1
 The Stem is poor
• After reading stem, student has little idea of what
the question is asking
NBME, p.17
Which of the following is true about
pseudogout?
A. Occurs more frequently in women.
B. It is seldom associated with acute pain in a joint.
C. It may be associated with a finding of
chondrocalcinosis.
D. It is clearly hereditary in most cases.
E. It responds well to treatment with allopurinol.
Flaw #2:
 Options are not on a single dimension
NBME, p.17
Which of the following is true about
pseudogout?
A. Occurs more frequently in women.
B. It is seldom associated with acute pain in a joint.
C. It may be associated with a finding of
chondrocalcinosis.
D. It is clearly hereditary in most cases.
E. It responds well to treatment with allopurinol.
Flaw #2:
 Options are not on a single dimension
• how can one determine if the frequency of pseudogout’s occurrence in women is more true than the low
frequency of concomitant pain in a joint?
• is the distinction of frequency of one thing vs the
infrequency of another of any importance?
• asking for a comparison between apples and oranges is not
important & probably always irrelevant for MB ChB students
Examples of a good A-type item
Please note:
An item from Medicine is illustrated now; 3 additional examples
(Surgery, OBGyn, & Orthopaedics) are provided in the handout
Example: good A-type item
NBME, p.16
(Clinical Vignette)
A 32-year-old male has a 4-day Hx of progressive weakness in his
extremities. He has been healthy except for an upper respiratory tract
infection 10 days ago. His temperature is 37.8 oC, blood pressure is
130/80 mm Hg, pulse is 94/min and respirations are 42/min and
shallow. He has symmetric weakness of both sides of the face and the
proximal and distal muscles of the extremities. Sensation is intact. No
deep tendon reflexes can be elicited; the plantar responses are flexor.
(Lead-in)
Which of the following is the most likely diagnosis?
NBME, p.16
Which of the following is the most likely diagnosis?
(Alternatives):
A.
B.
C.
D.
E.
Acute disseminated encephalomyelitis
(*)
Guillain-Barre syndrome
Myasthenia gravis
Poliomyelitis
Polymyositis
NBME, p.17
Even though the incorrect answers are not completely
wrong, they are less correct than the ‘keyed’ answer.
D
C A
E
B
_____________________________________________
Least
Correct
Most
Correct
Surgery
Example of Good A-type item:
(Clinical Vignette)
A 10-month-old infant takes his evening feeding normally and
falls asleep. Two hours later he awakens, cries for 10 minutes,
and vomits a small amount of thin greenish fluid before falling
asleep again, Two hours later he awakens again, cries harder,
then has a bloody bowel movement.
(Lead-in)
Which of the following is the most likely diagnosis?
Surgery
Which of the following is the most likely diagnosis?
A. Bleeding Meckel’s diverticulum.
(*)
B. Intussusception.
C. Midgut volvulus.
D. Intestinal obstruction due to ingested foreign body.
E. Gastroenteritis.
Surgery
Again even though incorrect answers are not completely
wrong, they are less correct than the ‘keyed’ answer.
E
A D
C
B
____________________________________________
Least
Correct
Most
Correct
OBG
Example of a good A-type item:
(Vignette)
A young healthy primigravida whose pelvis is normal to clinical
examination has been in labour for 12 hr. Painful contractions of
moderate intensity have been occurring at intervals varying between
5 and 10 min with a duration of 30-40 sec. The vertex is at station 0,
the position is L.O.T. and the estimated foetal weight is 3200 g. The
cervix is 3 cm dilated and well effaced. The membranes were
artificially ruptured 6 hours ago to augment labour.
(Lead-in)
Which of the following conclusions is most likely?
OBG
Which of the following conclusions is most
likely?
A.
B.
C.
D.
E.
Contractions will become regular and forceful in 2 to 3 hrs
This is protective inertia
There is unsuspected inlet disproportion
This is uterine inertia in the latent phase.
Delivery must be effected by caesarean section
B C E
A
D
_____________________________________________
(least likely)
(most likely)
O&T
(Clinical Vignette)
An 82-year-old man has neck, hip, and knee
pain with joint swelling. The rheumatoid factor
test is weakly positive at 1:80.
(Lead-in)
Which of the following is the most likely
diagnosis?
O&T
Which of the following is the most likely
diagnosis?
A. Osteoarthritis.
B. Late onset rheumatoid arthritis.
C. Polyarteritis nodosa.
D. Polymyalgia rheumatica.
E. Metastatic malignancy.
C D E
B
A
_____________________________________________
(least likely)
(most likely)
Good features
 Experts would agree on the most likely diagnoses

- or certainly agree that the other diagnoses are less likely
 Item measures more than recall of isolated facts
 Lead-in is clear and unambiguous
 All options can be laid out on a single continuum
 Even if previous question was only somewhat familiar (not
necessarily something you would test in your own
teaching)
• you probably still could tackle the item
• if so, you sensed that which students feel --- the question, not
familiar, required one to reason & problem solve
• with some known facts and reasoning you could solve it -- i.e.,
the question was fair, you did not guess, at worst guesstimated
• ‘guesstimation’ can be & estimation is a desirable skill to
measure in medical students as few real clinical decisions are
on based on certainty
– measure of ability to make decisions under uncertainty
These same principles can apply for
assessments in any of the clinical or
basic sciences . . .
(additional clinical science examples are included in
your handout, a couple of basic science items follow)
Anatomy
A 42-year-old male laborer fell off a structure and is
taken to hospital. A fracture of the middle third of
the shaft of the humerus is diagnosed and damage
to the radial nerve is suspected by the examining
doctor.
If the doctor's suspicion is indeed correct, the patient probably
will be unable to:
A. Oppose the thumb of the middle finger.
B. Extend the wrist against gravity.
(*)
C. Feel a pinprick on the lateral side of the forearm.
D. Contract the triceps muscle.
Physiology, NBME, p.44
A 34-year-old woman has had severe watery
diarrhea for the past four days. Two months
earlier she had infectious mononucleosis. She
abuses drugs intravenously and has antibodies
to HIV in her blood. Physical examination
shows dehydration and marked muscle
weakness.
Laboratory studies are most likely to show
A. decreased serum K+ concentration.
B. decreased serum Ca2+ concentration.
C. increased serum HCO3- concentration.
D. increased serum Na+ concentration.
E. increased serum pH.
(*)
Review:
NBME, p. 33
Basic Rules for Writing A-type Items
 Focus on important concept
• e.g., a common or potentially catastrophic clinical problem
 Include items that assess application of knowledge
• not only recall of isolated facts
 Pose a clear question
• student can arrive at an answer with options covered
 Make all distractors homogeneous
 common continuum exists for all choices
 Avoid technical flaws in composition
 see previous slides & Appendix 1 re errors to avoid
Thus: A-type items are also difficult to write
(not unlike T/ F)
But when well written, A-type less often forces
the writer to deal with isolated facts
This does not mean A-type questions are a panacea
• Good multiple T/F questions can be also written
• but medical test writers usually find it more difficult
to write them well
Additional research in comparing X and A type items
is included in your handout
Further Comparisons
of
X-type & A-type items
Issue of Sampling: A- vs. X-type items
 If the number of questions is kept constant in two different
exams
• A-type will normally sample more broadly the skills of interest
 But remember:
• 100 A-type items take longer to answer than 20 stems each with 5
T/F choices
• average of 0.75 min per question, if English is mother tongue;
• for other students who are examined in English, but whose 1st
language is not English, then the time needed is ???
A- & X-type compared on medical speciality exams
(S. Downing, R. Baranowski, L. Grosso, J. Norcini, 1995)
 The Number of Exams (ABIM)
• 1 qualifying exam,
• 2 initial certifying exams,
• 5 subspecialty exams
 The Number of Items
• 1,011 A-type items
• 2,432 X-type items
 The Number of Candidates:
• 21,346
Downing, SM, Baranowski, RA, Grosso, LJ, & Norcini, JJ (1995). Item Type and Cognitive Ability Measured: The Validity Evidence
for Multiple True-False Items in Medical Specialty Certification. Applied Measurement in Education, 8, 87-97.
Results: (S. Downing, et al)
 X-type slightly more internally consistent than A-type
 X- and A-type exam results were highly correlated
 Average A-type items were more difficult
• i.e.. more examinees got the correct answers on T/F items than
on one of best of five choices
 A-type had better validity
• higher correlation with overall clinical competency ratings
assigned by residency directors
Findings: 1997 HK licentiate exam (Hazlett, 1998):
 Confirmed others’ findings
• A- & X-type are correlated well, even across disciplines
• A-type more difficult
 Also found:
• A-type better in measuring ability to make decisions
under uncertainty (many omits with X-type)
• A-type more stable to application of penalty score
Considerations:
 If development of problem solving and clinical
reasoning skills is wanted, one has to test for these
skills (and do so with many questions/tasks)
• May wish to consider using some A-type items to help do so
• May wish to involve all teaching staff in writing items so a
large bank of current, clinical reasoning items are available
May want to use IDEAL’s item banks
 10,500+ questions are now available
 Consortium provides a windows-based computer program for
• convenient storage, retrieval, test creation, printing, updating and item
response analysis
• versions for hard copy high stakes & web-based administered examinations
 Created by pooling & sharing items among medical schools at:
• Hong Kong (CUHK, HKU), Alberta, British Columbia, Sydney, Flinders,
Otago, Witwatersrand, Peninsula Medical School, Arabian Gulf Univ, UAE
Univ, Australian National Univ
Considerations:
 With any item bank . . .
• Important to regularly review & modify items as suitable
measures
•
importance & currency of content, clarity, lack of writing errors
•
statistical evidence from previous assessments to guide revisions
– will discuss more of this later in the workshop
What questions do you have?
(mini break)
Look at the course objectives
(particularly the verbs in the objectives)
 If the objectives state the students should
• recall, recognise, remember
• these verbs indicate the students need to memorise facts
 If objectives state the students should
• analyse, assess, apply, compare, conclude, contrast,
diagnose, derive, differentiate, discriminate, evaluate,
estimate, extrapolate, generalise, predict, relate, validate
• these verbs indicate the students need to do 3 things:
• recall what they know, interpret the info, & then draw some
conclusion
• this process is reasoning or problem solving
The secret to assessing
more than factual information
 Consider the following two items
O&T
Osteogenesis imperfecta is the result of a
defect in the biosynthetic pathway for
A.
B.
C.
D.
E.
fatty acids
elastin
gelatin
(*)
collagen
osteonectin
and then this question . . .
O&T
An 8-year-old child is admitted to hospital with a
fracture of the femur following a fall at home. The
child has been admitted many times before with
fractures affecting both the femora, humeri and
clavicles. Physical examination shows he has a blue
sclera, rib cage deformity shortening and malunion
of the arm and lower limb.
These findings suggest a defect in the biosynthetic pathway for
A.
B.
C.
D.
E.
fatty acids
elastin
gelatin
collagen (*)
osteonectin
NBME, p.37
Acute intermittent porphyria is the result of a defect
in the biosynthetic pathway for
A.
B.
C.
D.
E.
collagen
corticosteroid
fatty acid
glucose
(*)
heme
NBME, p.37
An otherwise healthy 33-year-old man has mild weakness and
occasional episodes of steady, severe abdominal pain with some
cramping but no diarrhoea. During an episode, his abdomen is
distended, and bowel sounds are decreased. Neurological
examination shows mild weakness in the upper arms.
These findings suggest a defect in the biosynthetic pathway for:
A.
B.
C.
D.
E.
collagen
corticosteroid
fatty acid
glucose
(*)
heme
The pair of items measure the same topic
 The choices are identical in each, but
• the 1st item tests knowledge (memorised facts)
• the 2nd item tests application of knowledge to a clinical
situation
 To achieve the second, develop a clinical vignette
NBME, p.38 or p.56
A vignette includes a patient description & a clinical
scenario with some or all of the following information:
Age, Gender (e.g. A 45-year-old man)
Site of Care (e.g. comes to the emergency department)
Presenting Complaint (e.g. because of annoying pain in his knee)
Duration (e.g. that has continued for 2 days)
Patient History (e.g., inside desk job, active in athletics)
Physical Findings
+/- results of diagnostic workup
+/- initial treatment, subsequent findings, etc.
Advantage in using a clinical vignette
 Face validity goes up
NBME, p. 36 ?
 Students regard items with
clinical vignettes as more
clinically relevant
 Students see that they have to
apply their knowledge to make a
decision or solve a problem
Additional advantage of a clinical vignette
 One can create more than one question from one
vignette
 In an integrated curriculum this is particularly
advantageous
• as related clinical & basic science questions can be asked
under same vignette
(An item that draws on knowledge from Physiology)
A 44-year-old housewife is admitted to
the ward with generalized bone pain
especially over the spine and pelvis. She
has a history of breast cancer, and had a mastectomy 3
years earlier. A plain radiograph of the spine reveals
collapse of the L4 and L5 vertebrae.
Laboratory studies are most likely to show
A.
B.
C.
D.
E.
decreased
increased
increased
increased
decreased
Se Ca
Se Ca (*)
Se Creatinine
Se Albumin
Se Creatinine
(And this item draws on coverage from Radiology & Orthopaedics)
Additional question that could be asked
using the same vignette:
What is the most appropriate investigation for this patient at
this stage?
A.
B.
C.
D.
E.
(*)
bone scan
Dexa study
bone biopsy
MRI examination
Se PTHRP measurement
(And this item also draws on coverage in Orthopaedics)
A third question using the same vignette:
A skeletal survey was performed revealing
multiple metastastic lesions in the pelvis,
cervical dorsal and lumbar spine.
What is the next most appropriate step in management?
A.
B.
C.
D.
E.
intravenous Pamidronate (*)
spinal stabilization
calcium and Vit D supplements
whole body irradiation
treatment with Tamoxifen
Additional examples of multi disciplinary item sets
(i.e., that share a common clinical vignette)
are provided in your handout
NBME, p.46
An unresponsive 58-year-old woman is brought
to the emergency department after collapsing at
a shopping mall. Her family reports that she felt
well that morning but developed a headache that
progressively worsened while she was shopping.
She has had hypertension and atrial fibrillation and is taking
an antihypertensive medication and an oral anticoagulant.
Her blood pressure is 220/130 mm Hg and her respiratory
pattern is one of apnea alternating with hyperpnea. She
responds only to noxious stimuli with extensor posturing
involving the right arm and leg. Fundoscopic examination
reveals papilledema involving the left optic disc. Pupils are
3.0/7.0 (R/L) with no reaction to light on the left. There is a
left gaze preference. There is diffuse hyperreflexia (R>L)
and Babinski’s sign is present bilaterally.
NBME, p.46
The dilated, unreactive left pupil is most
consistent with injury to the left
A.
B.
C.
D.
E.
optic nerve
optic tract
(*)
oculomotor nerve
lateral geniculate nucleus
superior colliculus
NBME, p.46
The extensor posturing on the right is most
consistent with injury to which of the
following areas of the brain on the left?
A. Telencephalon
B. Diencephalon
C. Midbrain
D. Pons
E. Medulla
(*)
NBME, p.46
Her respiratory pattern is best described as
A.
B.
C.
D.
E.
normal
Cheyne-Stokes (*)
central neurogenic hyperventilation
apneustic
ataxic
NBME, p.46
Which of the following herniation syndromes
is most consistent with her clinical
presentation?
A. Cingulate gyrus beneath the falx.
B. Temporal lobe uncus across the tentorium. (*)
C. Diencephalon through the tentorial notch.
D. Brain stem through the tentorial notch.
E. Cerebellar tonsils through the foramen magnum.
Examples of how items can help with
integration of basic and clinical sciences
BioChem
A variety of genetic disorders can now be
diagnosed using molecular genetic techniques.
The disease alpha1-antitrypsin deficiency is
caused by a single base mutation (G -> A),
resulting in an amino acid change (Glu -> Lys)
at residue 342 of the alpha1-antitrypsin gene.
What would you expect the alpha-antitrypsin gene to
exhibit?
A.
B.
C.
D.
E.
Contain 1 intron
Contain 2 introns
Contain 3 introns
Be uninterrupted
Can’t tell as insufficient data are provided
(*)
Which sample of polynucleotide would be
most appropriate to use in prenatal
diagnosis of this disorder?
A.
B.
C.
D.
E.
Fetal blood cells.
Fetal epithelial cells. (*)
Fetal lung cells.
Parental epithelial cells.
Sibling epithelial cells.
Whereas the previous question would be Biochemistry’s,
this question now integrates Biochemistry with ObGyn
Which of the following steps, involved in replication of
DNA, would be most appropriate for diagnosis?
A.
B.
C.
D.
E.
Determination of restriction fragment length polymorphism.
Monitor level of expression of alpha1-antitrypsin gene
transcript.
Determination of melting temperature of short oligonucleotide
fragment. (*)
Monitor level of alpha1-antitrypsin protein expression.
None of the above.
. . . and similarly this third question integrates with ObGyn
The most important advantage in using
a clinical vignette:
If your objective is to measure the application
of knowledge (rather than testing for factual
knowledge) . . .
the use of a clinical vignette greatly helps.
Advantage in using clinical vignette
 Improves item’s
discrimination power
 The poorer and better students
perform differently
 Students who only memorize
often cannot answer an
application question as well
Med
A 54-year-old female has frank aortic regurgitation,
with a murmur graded six out of six, a diastolic thrill
along the left sternal border and egg shell
calcification of the aortic arch shown on x-ray. Her
main complaint is angina progressing over several
years. She is now in congestive failure.
The most informative laboratory examination would
be a/an:
A.
B.
C.
D.
E.
Positive LE cell preparation.
Elevated antistreptolysin O titer.
Elevated LDH and heart muscle isoenzymes.
Positive rheumatoid factor in the serum.
Positive VDRL. (*)
Medicine
Are there pedagogical reasons to structure
questions to test more than recall of isolated
facts . . .
particularly when it’s so important to
have a good factual base?
The answer (generally) is Yes :
 But not because:
• the application question is more difficult
 Not because facts are thought,
inappropriately, to be unimportant
The answer (generally) is Yes:
 Because:
• study preparation for answering application of facts is
different, helping student remember more of the facts being
used in the application
• for medical students facts are easier to remember if they
become organised in a clinical, logical & relevant fashion
Providing a CONTEXT is the secret
Some evidence to show that context is important
& needs to be provided
 Have you ever gone from one room to another, looking for
something, and then when you got there you couldn’t remember
what it was you were intending to get?
 Did you go back to your room & then recalled what it was that
you needed?
 This was not co-incidental.
• your own room provided the context
• your room cognitively trigged recall of the object
because it was there that you needed to use it
Experimental evidence to support thesis that context
is important & needs to be provided
 Classical experiment with scuba diver qualification tests
• results on written tests (re decompression) for students who
took the test underwater were better
• in comparison to those who were tested in a classroom
• given nitrogen toxicity build-up that occurs during a
dive, opposite result would be expected
Still need further experimental proof about
importance of providing context?
… maybe it will help if we do a small, but
soundly designed, scientific experiment right
now that hopefully will prove the point
Two members in this audience are needed
 We will randomly select two members
 Both will leave the room, after randomly assigning
which will come in back first and take a little test
 Second person will then come in and take the test
 Both are requested to try to do as well as possible
Randomly select
Randomly assign
Candidates please leave the room;
will call each of you when to return
Although we’re using a sample of two (i.e., my statistical power
for this experiment is minimized),
the effect size is so large that I’m still confident of the
experimental outcome --- so confident that I will make a
wager.
The Wager
 I bet I can predict which of the two will do better on the test
 To make it even more difficult for me to win
• I will give my choice significantly less time to prepare for the test
• Some of you will score each candidate on the test, not me
 If I am proved to be wrong, I will give this group $500 HK
(to buy a coffee or tea or a coke)
 If I am right, I get to keep my money, but hopefully you will
remember a most important principle
First Candidate
You have one minute to study the following material.
Please try to remember as many points as you can.
Standing sometimes helps. Rain adds danger. After awhile
you may get thirsty. Using cement is better than grass and
grass is better than rocks. Getting geared up makes things
harder. Using one hand is optional but only when seated.
Chains work but ropes do not. Leaning forward helps a lot,
but leaning a lot in other directions can be a hindrance.
Using sidewalks makes people unhappy. Using both legs is
easier than one. Wind can hinder but having good air
pressure is helpful. You can’t go backwards easily. It is quite
easy to learn and difficult to forget. Tipping is not a gratuity
and often hurts more. Nails are obstacles.
Second Candidate
I will give ¾ minute to study the following material on
How to Ride a Bike.
Please try to remember as many points as you can.
How to Ride a Bike
Standing sometimes helps. Rain adds danger. After awhile
you may get thirsty. Using cement is better than grass and
grass is better than rocks. Getting geared up makes things
harder. Using one hand is optional but only when seated.
Chains work but ropes do not. Leaning forward helps a lot,
but leaning a lot in other directions can be a hindrance.
Using sidewalks makes people unhappy. Using both legs is
easier than one. Wind can hinder but having good air
pressure is helpful. You can’t go backwards easily. It is quite
easy to learn and difficult to forget. Tipping is not a gratuity
and often hurts more. Nails are obstacles.
Let's see who remembered the most points
Remembering details is much more
onerous if . . .
a meaningful context is not available
Examine the nature of a meaningful context
i.e., it will elicit cognitive skill of application (not only recall of facts),
& at the same time actually helps one to remember facts
 For MB ChB students
• as their interests/orientation is clinical, patient-centered
• using a clinically oriented vignette is certainly helpful
• but only describing some patient (or a disease or trauma, or some
Hx or Px findings) doesn’t necessarily make the task meaningful
 To develop a meaningful context requires task to
mimic real life
Nature of Meaningful context
 A clinically oriented question that still only measures recall
• begins by citing a disease & then asks which patient findings are to be
expected
 Such a sequence is flawed: i.e., it is backwards to what
happens in real life . . .
• a patient rarely tells a doctor what disease s/he has, after which
the doctor then asks for her/his signs and symptoms
• the sequence is however (& unfortunately) similar to how many
textbooks do cover material
Here’s an illustration of an item that lacks
meaningful context
(note that the question is clinically oriented,
but it doesn’t provide a meaningful context)
Which of the following diseases is an
indication for doing HLA B-27 testing in
a 28-year-old male?
A.
B.
C.
D.
E.
Ankylosing Spondylitis
Reiter’s Disease
Rhuematoid Arthritis
Psoriasis
Scleroderma
What’s wrong ?
In clinical practice, doctors don’t go around
searching for the most appropriate patient in
order to do some HLA B-27 testing.
The logical clinical task (meaningful
context) would be something such as:
A 28-year-old clerk has been complaining of
pain and stiffness of the spine & hip joints
for a year. He has now developed a kyphosis
and requires a cane for support.
Radiographs show scaroilitis.
Which of the following investigations is the most appropriate
to undertake at this time?
A.
B.
C.
D.
E.
test for presence of Bence Jones Protein
do a VDRL test
do a HLA B-27 test
test for presence of C-Reactive Protein
test for presence of anti-nuclear antibody
NBME, p.54
Which of the following is an
indication for foetal karyotyping in a
28-year-old woman?
A.
B.
C.
D.
E.
Paternal age 55
Foetal cystic hygroma on ultrasound exam
Previous child with spina bifida
Previous miscarriage of a triploid foetus
Trisomy 21 in the woman’s brother
Again, what’s wrong ?
As before, in clinical practice doctors don’t
search for appropriate pregnant woman in
order to do some karyotyping.
NBME, p.54
A healthy 28-year-old teacher is at 11 weeks’
gestation. Family history is unremarkable
except that both of her brothers have severe
mental retardation, her mother died at 55 years
of age of breast cancer, and her father is estranged. No
family health records are available.
Which of the following investigations is appropriate?
A. Blood test for fragile X carrier status
B. Blood test for phenylketonuria carrier status
C. Chorionic villus sampling for Duchene’s muscular
dystrophy
D. Chorionic villus sampling for chromosome analysis
E. Amniocentesis for α-fetoprotein (*)
The logical clinical task (meaningful
context) would be something such as:
Thus, best guideline for composing good
questions is:
structure items in a clinically realistic manner
such as,
- give some findings, then
- ask student to indicate the underlying disease
Surgery
Mr. Kwok, a 70-year-old male, presents
with new onset atrial fibrillation.
Physical examination reveals findings
consistent with atrial fibrillation at a
rate of 150/min. and a short ejection murmur at the
base of the heart, S3 JVP 10 cm absolute, BP 120/84
and bibasilar inspiratory crackles.
The most likely etiology for his atrial fibrillation is:
A.
B.
C.
D.
E.
Thyrotoxicosis.
Hypertension.
Aortic stenosis.
Sick sinus syndrome.
Silent ischemic heart disease. (*)
Templates
You may find that a template helps you to keep
this sequence as you develop items
See NBME templates in Word Document (enclosed on disk) for
the clinical sciences; included in the latter pages are templates
for the basic sciences & for integrating clinical & basic sciences
Item templates can help you compose items
- easier, quicker
- more than one item can be developed from the same
template.
- a few shown now; when we break into groups to
develop items (using these templates) you will
find the examples useful
Acknowledgement
• These templates are drawn from:
Swanson DB & Case SM, Constructing
Written Test Questions for the Basic and
Clinical Sciences.
(3rd Edition, NBME: Philadelphia, 2001)
See web site: http://www.nbme.org/PDF/2001iwg.pdf
Clinical Sciences (Appendix see Word Document)
If writing items for Health and Health
Maintenance Tasks:
 Assess ability to
• evaluate risk factors
• understand epidemiological data
• apply preventive measures
NBME, p. 61
NBME, p. 61
Health and Maintenance items usually fall
into one of the following categories:
 screening tests
 emergency intervention
 immunisations / travel medicine
Basic Strategy:
NBME, p. 61
Begin with a vignette that describes the patient
and include a relevant clinical scenario:
 History and physical examination
 Information about immunisation history, risk factors,
and family history
 Relevant information about the community
 Question should focus on individual patient
NBME, p. 61
After the scenario, these lead-ins are useful for
assessing health and maintenance topics:
 Which of the following is the most appropriate
screening test?
 Which of the following tests would have predicted
these findings?
 Which of the following is the most appropriate
intervention?
 For which of the following conditions is the patient
at greatest risk?
NBME, p. 61
Lead-ins for health and maintenance (cont’d):
 Which of the following is most likely to have
prevented this condition?
 Which of the following is the most appropriate next
step in management to prevent
[morbidity/mortality/disability]?
 Which of the following should be recommended to
prevent disability from this injury/condition?
 Early treatment with which of the following is most
likely to have prevented this patient’s condition?
Radiology & Surgery
A 2-year-old child presents with a
midline cystic neck nodule which
moves on tongue protrusion and
swallowing. A clinical diagnosis of a
thyroglossal duct cyst was made.
A ultrasound scan is indicated to evaluate:
A.
B.
C.
D.
E.
The exact size of the cyst.
The exact delineation of the associated tract.
The presence and status of the thyroid gland.
The presence/absence of the hyoid bone.
The thickness of the cyst wall.
(*)
NBME, p.62
A 33-year-old woman, gravida 1, para 1,
spontaneously delivers a 2460g female at 38
weeks’ gestation. The newborn has
hepatosplenomegaly, patent ductus arteriosus, and
cataracts. At 8 weeks’ gestation, the mother
developed a maculopapular rash, enlarged cervical lymph
nodes, sore throat, and arthralgias that spontaneously resolved
in 1 week. The subsequent prenatal course was uncomplicated.
Which of the following tests during pregnancy is the most
likely to have predicted the findings of the foetus?
A.
B.
C.
D.
E.
Amniocentesis to determine karyotype
Culture for herpes simplex virus
Serial rubella titres (*)
Urinalysis for cytomegalovirus
VDRL test
NBME, p.63
Templates for testing Mechanisms of Diseases
 These test students’ knowledge of --• aetiology, pathogenesis, natural history, clinical
course, associated findings, complications, severity
of illness, and intended or unintended effects of
therapeutic intervention
Basic strategy:
NBME, p.63
Begin a mechanism item with a clinical vignette
of a patient and her/his symptoms, signs, history,
lab study findings, etc. . . .
then ask a question such as one of the following:
Questions for testing mechanisms:
NBME, p.63
 Which of the following is the most likely explanation
for these findings?
 Which of the following is the most likely location of
the patient’s lesion?
 A biopsy is most likely to show which of the
following?
 Which of the following is most likely to be
increased/decreased?
 Which of the following is the most likely pathogen?
Surgery
A 70-year-old man has just returned from the
operating theatre after a transurethral resection
of the prostate. The duty House Officer noticed
there was no urine flow from the urethral
catheter.
Which of the following is the most likely cause?
A.
B.
C.
D.
E.
Hypovolumic shock.
Blood clot retention. (*)
Inadequate resection of the prostate.
Hyponatraemia.
Urethral stricture.
NBME, p.63
Example:
A 10-year-old girl develops gross haematuria 14 days after a sore
throat. She has a blood pressure of 170/100 mm Hg and 2+ pedal
and pretibial oedema. Serum urea nitrogen (BUN) level is 3.2
mg/dL.
Which of the following is the most likely cause?
A.
B.
C.
D.
E.
Coarctation of the aorta
Decreased production of endothelia-derived relaxant factor
Increased production of aldosterone
Increased production of catecholamine
Intravascular volume of expansion (*)
Templates related to Diagnosis
NBME, p.64
 Require the students to interpret a history, physical
findings, results of laboratory, imaging, and other
studies in order to determine
• the most likely diagnosis (or differential diagnosis);
• the next step in diagnosis (or, in diagnostic testing).
NBME, p.64
Basic strategy for items designed to diagnostic
ability begins with a patient description:




Age
Symptoms
History
Lab studies




Gender
Signs (duration)
Physical findings
Diagnostic findings
 Ends with one of the following questions
Questions related to diagnosis
NBME, p.64
 Which of the following is the most likely
diagnosis?
 Which of the following is the most appropriate
next step in diagnosis?
 Which of the following is most likely to confirm
the diagnosis?
Surgery
A 29-year-old woman has had a tender mass
in the upper outer quadrant of the right
breast for six weeks. The mass became
somewhat smaller during her last menstrual
period but then increased in size again. The mass is now
3 cm in diameter, smooth in contour, and is slightly
tender.
Which of the following is the most appropriate next
step in diagnosis?
A.
B.
C.
D.
E.
A mammogram.
An excisional biopsy.
Ultrasonography of the mass.
A needle aspiration. (*)
Re-examination in four weeks.
Paediatrics:
A 10-month-old infant takes his evening feeding
normally and falls asleep. Two hours later he
awakens, cries for 10 minutes, and vomits a
small amount of thin greenish fluid before
falling asleep again,
Two hours later he
awakens again, cries harder, then has a bloody
bowel movement.
Which of the following is the most likely diagnosis?
A. Bleeding Meckel’s diverticulum.
B. Gastroenteritis.
C. Midgut volvulus.
D. Intestinal obstruction due to ingested foreign body.
E. Intussusception
Templates related to Management
 Assess principles of chronic and acute care in
inpatients and outpatient settings
 Focus on aspects of care relevant to the level of
practice expected of the student
• supervised (according to most accreditation criteria)
• limited supervision (maybe ??)
• independent practice (very unlikely)
• sub-specialist (never)
NBME, p.65
Management questions include:
NBME, p. 65
 Which of the following is the most appropriate initial
or next step in patient care?
 Which of the following is the most effective
management?
 Which of the following is the first priority in caring
for this patient
• e.g., in emergency department
 Which of the following is the most appropriate
pharmacotherapy?
Psychiatry
Management (Psychiatry):
A 22-year-old woman with a diagnosis of
chronic undifferentiated schizophrenia was
started on perphenazine three days prior to the
onset of acute dystonia.
An appropriate drug for immediate treatment would be:
A.
B.
C.
D.
E.
Diazepam.
Phentolamine.
Benztropine mesylate.
Chlorpromazine.
Phenobarbital.
(*)
Medicine
Management (Medicine):
A 25-year-old male homosexual presents to
emergency with a two-week history of
progressive shortness of breath and
nonproductive cough. Examination reveals
tachypnea and fever. Chest x-ray shows
bilateral interstitial pulmonary infiltrates.
The most appropriate initial therapy should be:
A.
B.
C.
D.
E.
Erythromycin.
Ampicillin and gentamicin.
Trimethoprim-sulfamethoxazole. (*)
Withhold antibiotics pending culture results.
Cefotaxime for broad spectrum coverage.
Surgery
Management (Surgery):
A 60-year-old man with known alcoholism presents to the
Emergency Department with haematemesis. Upper GI
endoscopy revealed actively bleeding oesophageal varices.
The next treatment should be:
A.
B.
C.
D.
E.
Transjugular intrahepatic portosytemic shunt.
Emergency portocaval shunt.
Splenectomy.
Gastroesophageal devascularisation.
Sclerotherapy. (*)
NBME, p.66
Writing Items on Difficult Topics:
Medical Ethics and Jurisprudence
 Common belief is that certain topics do not lend
themselves to a selected-response format.
Difficult Topics:
NBME, p.66
 Consent and informed consent to treatment
• full disclosure, alternate therapies, risks and benefits
 Physician-patient relationship
• truth-telling, confidentiality, privacy, autonomy, public
reporting
 Death and dying
• diagnosing death, life-support, autopsy, organ donation,
euthanasia, suicide
 Birth-related issues
• prenatal diagnosis, abortion, maternal-foetal conflict
 Research issues
• consent, placebos, conflict of interest, vulnerable populations
NBME, p.66
Poor Question
(that attempts to test difficult topics)
The legal basis for an easing of restrictions on abortions
in S. Africa can be traced most closely to
A.
B.
C.
D.
S. African legislation
S. African court ruling
Practice in other countries
Rulings made by the S. Africa Medical Association
Tests for recall of some historical fact

To avoid the foregoing type of mistake, focus
on what the surgeon or physician should do
for example . . .
- evaluate the appropriateness of a doctor’s
actions that is indicated in a clinical vignette.
NBME, p.67
A nurse is hospitalised, for repair of a torn knee
ligament, at the medical centre where she is employed.
One week after discharge, the assistant hospital
administrator asks her orthopaedic surgeon what the
final diagnosis was.
Which of the following is the most appropriate response on the part
of the surgeon?
A.
B.
C.
D.
E.
Provide the answer, as it will expedite handling of insurance issues at
the medical centre
Provide the answer, since the administrator has access to information
about the patients
Provide the answer, in order to avoid the possibility of spreading
misinformation about the patient
Decline to answer, as the administrator is not a medical doctor
Decline to answer, as the information is confidential (*)
Writing items for Basic Sciences
& an Integrated Curriculum
See latter pages in the enclosed Word file on CD for
Assessment templates to help develop vertically integration
of Basic and Clinical Sciences
These templates may prove helpful if you are trying to elevate
the clinical relevance of some basic science questions
Curriculum revisions that seek to achieve greater vertical
integration also require that:
Clinical instructors become more involved in and
with coverage of basic science material (& vice versa).
This logically leads to revised student assessments --those that reflect this desired vertical integration.
If students are assessed in this fashion, they must also
be taught in this fashion.
As students progress use a short vignette to ask about both
clinical and basic science aspects that underlie a particular
patient problem
 Describe a laboratory task or a patient condition in which a
problem or question is inherent
 Then ask a question related to this task, patient or situation
. . . see the following templates
[Short vignette]
A (insert here a description of patient) is unable to
(insert here the patient’s functional disability).
[Lead-in question]
Which of the following (bones, muscles, ligaments,
cartilage) is most likely to have been damaged or
injured?
[Provide a list of optional injury sites]
A.
B.
C.
D.
E.
Anatomy
A 10-yr-girl demonstrates an inability to hold
a piece of paper firmly between the extended
middle and extended ring fingers.
Her inability to do so probably indicates an
injury to the:
A. median nerve above the wrist.
B. deep branch of the ulnar nerve. (*)
C. superficial branch of the ulnar nerve.
D. dorsal branch of the ulnar nerve.
E. posterior interosseous nerve.
NBME p. 131
For example:
A 65-old-man has difficulty rising from a
seated position and straightening his trunk,
but he has no difficulty flexing his leg.
Which of the following muscles is most
likely to have been injured?
A.
B.
C.
D.
E.
Gluteus maximus (*)
Gluteus minimus
Hamstrings
Iliopsoas
Obturator internus
NBME, p. 39
 A (patient description) has a (type of injury and location).
• Which of the following structures is most likely to be
affected?
Provide a list of optional structures
Anatomy
A 18-year-old male patient has an awkward gait,
characterized mainly by dragging his toes on the
ground during the "swing phase" and slapping his
foot on the ground at "heel strike".
This young man likely has a lesion affecting the:
A. nerve to tibialis anterior.
B. superficial peroneal nerve.
C. tibial nerve.
D. common peroneal nerve.
E. medial plantar nerve.
(*)
Anatomy
A surgeon accidentally cuts a nerve while removing
the left lobe of the thyroid gland in a 55-year-old
male.
The most probable reduced or loss of function that
this patient will experience is:
A. a loss of speech.
B. his suprahyoid muscles are paralyzed.
C. the amount of gastric acid he can secrete is decreased.
D. his left vocal cord is paralyzed. (*)
E. he has partial paralysis of the epiglottis.
NBME, p. 39
 A (patient description) has (abnormal findings).
• Which (additional finding) would suggest a diagnosis
of (injury 1) rather than (injury 2)?
Provide list of optional findings
NBME, p. 39
 A (patient description) has (abnormal findings).
• Which of the following (positive laboratory results)
would be expected?
Provide a list of optional lab results
NBME, p. 39
 Following (a procedure), a (patient description)
develops (symptoms and signs). Laboratory findings
show (findings).
• Which of the following is the most likely cause?
Provide a list of optional causes
NBME, p. 39
 A patient has (abnormal findings) but (normal findings).
• Which of the following is the most likely diagnosis?
Provide list of optional diagnoses
NBME, p. 39
 A (patient description) dies of (trauma/injury).
• Which of the following is the most likely finding
on autopsy?
Provide list of possible findings
NBME, p. 39
 A patient has (symptoms and signs).
• Which of the following is the most likely
explanation for the (findings)?
Provide a list of possible causes
NBME, p. 132
For example:
A 62-year-old man develops left-sided limb ataxia,
Horner’s syndrome, nystagmus, and loss of
appreciation of facial pain and temperature sensations.
What artery is most likely to be occluded?
Student then has to pick from a list of optional arteries.
Anatomy
Following a boil on the upper lip, a farmer
from China became comatose.
It is likely that the infection spread to the
cavernous sinus from the facial vein by its
communication with the:
A. anterior jugular veins.
B. straight sinus.
C. superior sagittal sinus.
D. maxillary sinus.
E. ophthalmic veins.
(*)
Anatomy
A 35-year-old female is examined by an orthopaedic
surgeon in the OPD. He finds that her skin at the
lower lateral aspect of the leg, front of the ankle and
dorsum of the foot has sensory loss.
The nerve that is implicated is:
A. sural nerve.
B. saphenous nerve.
C. superficial peroneal (musculo-cutaneous)
nerve. (*)
D. deep peroneal nerve.
E. lateral plantar nerve.
NBME, p. 39
 A (patient description) has (symptoms and signs).
• These observations suggest that the (disease) is a
result of the (absence or presence) of which of the
following (enzymes, mechanisms)?
BioChem
A patient with diabetic ketoacidosis presents in
emergency with a pH of 6.91. The emergency room
physician includes 175 mEq of NaHCO3 in this initial
therapy. The patient is noted to be in severe distress
and dies of a cardiac arrest twenty minutes after the
NaHCO3 infusion. Blood gases taken during the cardiac arrest
indicate a pH of 7.45.
The most likely cause of death is:
A. Respiratory arrest associated with the removal of the acidotic stimulus to breath.
B. The effect of sudden pH change on the intracellular shift of Na+ into cardiac cells.
C. The effect of the NaHCO3 on stimulation of 2, 3 DPG (diphosphoglycerate)
synthesis in red blood cells.
D. The effect of increased pH on the oxygen dissociation curve of hemoglobin. (*)
E. Central nervous system alkalosis as a result of NaHCO3 infusion and decreased
respiration.
NBME, p. 39
 A (patient description) has (history findings) and is
taking (list her medications).
• Which of the following medications is the most likely
cause of her (one of: history, PE or lab findings)?
NBME, p. 39
 A (patient description) follows a (specific dietary regime).
• Which of the following conditions is most likely to occur?
NBME, p. 39
 A (patient description) has (symptoms, signs, or
specific disease) and is being treated with (drug
or drug class).
• The drug acts by inhibiting which of the following
(functions, processes)?
NBME, p. 39
 (Time period) after a (event such as trip or meal with
certain foods), a (patient or group description) became
ill with (symptoms and signs).
• Which of the following (organisms, agents) is most
likely to be found on analysis of (food).
NBME, p. 39
 A (patient description) has (symptoms and signs).
• Exposure to which of the (toxic agents) is the most
likely cause?
NBME, p. 39
 Which of the following is the most likely mechanism of
the therapeutic effect of this (drug class) in patients
with (disease)?
Options are sometimes difficult to develop
 Alternatives to the correct  Options to use for various
answer
lead-in questions
• should be seductive
• diagnostic of students typical
“mis-understanding” or “mislearning”
• enable you (the teacher) to
provide helpful feedback to
the student
• see the following
After your vignette, if your lead-in question is . . .
Which of the following ________ is
(present / normal / abnormal)?
NBME, p. 40
(or is most likely to be)
Options sets could include:
-sites of lesions
-list of nerves
-list of muscles
-list of enzymes
-list of hormones
-types of cells
-list of neurotransmitters
-list of toxins,
-list of molecules
. . . or
Anatomy
A patient on routine physical examination is found to
have an area of anesthesia on the face extending from
the midline to a line drawn approximately from the
right nostril upwards past the lateral canthus of the
right eye to the top of the skull.
Which injury has most probably occurred?
A. Injury to the lacrimal branch of the ophthalmic division of the
trigeminal nerve.
B. Injury to the frontal branch of the ophthalmic division of the trigeminal
nerve.
C. Injury to the ophthalmic division of the trigeminal nerve in the cranial
cavity. (*)
D. Injury to the fifth nerve in the trigeminal cave.
E. Injury to the fifth nerve as it leaves the brain.
If your lead-in question is . . .
NBME, p. 40
Which of the following findings is most likely?
Options sets could include:
- list of laboratory results
- list of additional physical signs
- DNA analysis results
- serum levels
- results of microscopic examination of fluids, muscle or joint
tissue
. . . or
NBME, p. 40
Which of the following _______is (defective or deficient
or nonfunctioning)?
Options sets could include
-list of enzymes;
-list of feedback mechanisms;
-list of endocrine structures, dietary elements, vitamins.
. . .or
NBME, p. 40
Which of the following (mechanisms, medications,
drugs, agents) is the most likely cause?
Options sets could include
-list of underlying mechanisms of the disease;
-medications that might cause side effects;
-drugs or drug classes;
-toxic agents;
-hemodynamic mechanisms, viruses, metabolic defects.
. . . or
NBME, p. 40
Which of the following _______ should be administered?
Options sets could include:
-drugs;
-vitamins;
-amino acids;
-enzymes;
-hormones.
Let’s recap the pertinent issues
if using any of these
clinical or integrative templates
Clinical vignettes help to establish a realistic context within
which the students’ knowledge base becomes stored, organised
& more easily recalled & used:
 Helps one to focus on important information, not trivia
• i.e., for the student (while studying) & the teacher (as an item
writer)
 Helps identifying students who have only memorised factual
information
• i.e., those who are unable to use the information effectively
 Helps to improve the relevance of material
• i.e., examinees are required to ‘solve’ clinically-oriented,
patient- centred problems
Remember to
NBME, p.37
 Focus items on key concepts and principles that are
essential information for all examinees to understand
 Test material that is relevant to learning in clinical
clerkships
 Focus on common or potentially catastrophic
problems
• avoid esoteric or interesting topics that are not essential
 Mimic clinical practice by describing a patient then
asking the student to take some course of action
 Pose the type of decision-making tasks that are
expected of a successful examinee
• those that must be able to be undertaken at the next
stage of training in a MB ChB programme
NBME, p.38
The patient description within the vignette should include
some or all of the following:
Age, Gender (e.g. A 45-year-old man)
Site of Care (e.g. comes to the emergency department)
Presenting Complaint (e.g. because of headache)
Duration (e.g. that has continued for 2 days)
Patient History (with/without Family History)
Physical Findings
+/- results of diagnostic workup
+/- initial treatment, subsequent findings, etc.
A lead-in question follows the vignette, such as:
 Determine the most likely hypothesis or diagnosis
 Indicate what additional laboratory studies should be
done
 Formulate the next step in managing this patient or in
conducting this experiment
 Predict the most likely additional finding
NBME, p.41
When writing your options, make sure that they are:
 Homogeneous in content (e.g. all are diagnoses; all are
next steps in patient care)
 Incorrect or inferior to the correct answer
 Plausible and attractive to the uninformed
 Similar to the correct answer in construction and length
 Grammatically consistent and logically compatible with
the stem
What questions do you have at this
point --or concerns, needed clarifications,
issues to debate?
Practice in writing & critiquing MCQs
Using clinical vignettes
Break into groups of five
 Each group is expected to include both clinical and basic science
teachers and the group is to write at least two items
• may involve using only one or two clinical vignettes
• develop one of the items so it integrates clinical & basic science material
• you will have 25 minutes
 Copy one of the items onto an overhead
 Group rep will present the item to the total group
 Group will critique
 Revise the item based on group’s feedback
• review the other item that you did not present to the group, to see if
revisions are also needed
 Submit final versions of both items to your Block Assessment
Groups for use in this next year’s student assessments
Additional relevant information not as yet
covered . . .
for writing items to assess students’
Clinical Problem Solving Ability
Some Don’ts in using clinical vignettes:
NBME, p.55
 Avoid clinical situations handled by post graduates
and specialists
Some Don’ts (cont’d):
NBME, p.57
 Avoid or least minimise use of negatively phrased
questions:
• e.g., avoid using “except” or “not” in the lead-in
•
why test for something is not applicable?
–
who cares?
–
certainly not the students, nor the patients
General
The following conditions could have caused
carbon dioxide retention, EXCEPT:
A.
B.
C.
D.
E.
Myasthenia gravis.
Pentobarbital overdosage.
Severe chronic airflow obstruction.
Severe metabolic alkalosis.
(*)
Renal tubular acidosis.
General
Which of the following is NOT a proper
treatment for akathisia ?
A.
B.
C.
D.
E.
Beta blockers.
Benzodiazepines.
Lithium carbonate.
Antihistamines.
Anticholinergics.
(*)
Avoid using a vignette from a textbook or one already
discussed in class
• if the vignette describes a patient identical to one the
student has seen before, you’ll measure only recall
Issues re additional information in the vignette
NBME, p.58
 Avoid verbosity
• keep it succinct; extra wording takes more time, adds no validity
 Can include some “window dressing” to enhance realism
• “incidental findings”
• but tends to make vignette long, extra reading & time
 Avoid including “red herrings”
• i.e., information that is intended to lead examinees away from
correct answer
• this is usually only suitable for postgraduates’ or specialists’
examinations
NBME’s research on use of vignettes:
NBME, p.58
 None, short and long vignettes
• Items became more difficult as patient findings were
presented in a less interpreted form
• Discrimination increased (not statistically significant)
• Regarded generally more appropriate because vignettes tested
application of knowledge to patient situations, and posed
appropriate clinical challenges (face validity)
For the lower-performing students,
the following 3 items were not identical
NBME, p.59
Non Vignette:
The most likely renal abnormality in
children with nephrotic syndrome and
normal renal function is:
A.
B.
C.
D.
acute poststreptococcal glomerulonephritis
hemolytic-uremic syndrome
minimal change nephrotic syndrome (*)
nephrotic syndrome due to focal and segmented
glomerulosclerosis
E. Schönlein-Henoch purpura with nephritis
A B C D E
hi 1 0 99 0 0
lo 8 1 90 1 0
NBME, p.59
Short Vignette:
A 2-year-old boy has a 1-week history of oedema. His
blood pressure is 100/60 mm Hg, and there is a
generalised oedema and ascites. Serum concentrations
are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569
mg/dL. Urinalysis shows 4+ protein and no blood.
The most likely diagnosis is
A.
B.
C.
D.
E.
acute poststreptococcal glomerulonephritis
hemolytic-uremic syndrome
minimal change nephrotic syndrome (*)
nephrotic syndrome due to focal and segmented glomerulosclerosis
Schönlein-Henoch purpura with nephritis
A B C D E
hi 0 0 98 2 0
lo 5 2 82 8 1
NBME, p.60
Long Vignette:
A 2-year-old black child developed swelling of his eyes
and ankles over the past week. Blood pressure is 100/60
mm Hg, pulse 110/min, and respiration 28/min. In addition
to swelling of his eyes and 2+ pitting oedema of his ankles, he has
abdominal distention with a positive fluid wave. Serum concentrations are:
creatinine 0.4mg/dL, albumin 1.4g/dL, and cholesterol 569 mg/dL.
Urinalysis shows 4+ protein and no blood.
The most likely diagnosis is
A.
B.
C.
D.
E.
acute poststreptococcal glomerulonephritis
hemolytic-uremic syndrome
minimal change nephrotic syndrome (*)
nephrotic syndrome due to focal and segmented glomerulosclerosis
Schönlein-Henoch purpura with nephritis
A B C D E
hi 0 1 98 1 0
lo 10 9 66 10 5
Helpful Statistics
What are the typical statistics?
 If a reasonably well written clinical vignette
is used?
General
Example
A 63-year-old man complains of burning
numbness of his feet and weakness of his legs.
Examination shows reduction of the tendon
reflexes and superficial sensation in the legs.
The muscles are weak and tender.
The diagnosis is likely to be:
A.
B.
C.
D.
E.
Polymyositis.
Sympathetic reflex dystrophy.
Amyotrophic lateral sclerosis.
Peripheral neuropathy. (*)
Multiple sclerosis.
General
However, use of a clinical vignette doesn’t solve all
problems. Notice the following question and the pattern
of the students’ responses at the University of Alberta
???
A 29-year-old woman has had a tender mass
in the upper outer quadrant of the right breast
for six weeks. The mass became somewhat
smaller during her last menstrual period but
then increased in size again. The mass is now
3 cm in diameter, smooth in contour, and is
slightly tender.
Which of the following is the most appropriate next
step in diagnosis?
A.
B.
C.
D.
E.
A mammogram.
An excisional biopsy.
Ultrasonography of the mass.
(*)
A needle aspiration.
Re-examination in four weeks.
???
Remember importance of context
To mimic clinical reality you still have to provide
the correct orientation, even if you are asking basic
science or integrated questions
One of the numerous ways that the templates will
help, is they guide developing a correct orientation
when creating a suitable context for a question
The correct orientation is to ask something about a
described patient, such as
• Predict the physical findings
• Predict lab findings if further investigation is undertaken
• Give a diagnosis
• Estimate patient's type/level of injury
Generally the incorrect orientation is:
- ask student to identify a patient that requires a
particular medical or surgical investigation or
procedure
NBME, p. 47
That is, the correct orientation is illustrated as in . . .
A 30-year-old man has loss of pain and
temperature sensation from the neck down on
the right side of the body and on the left side
of the face; partial paralysis of the soft palate,
larynx, and pharynx on the left; and ataxia on the left.
This syndrome is most likely to result from thrombosis
of which of the following arteries?
A.
B.
C.
D.
E.
Basilar
Right posterior inferior cerebellar
Left posterior inferior cerebellar
Right superior cerebellar
Left superior cerebellar
(*)
BioChem
 Predict his/her physical findings
 Predict lab findings if further investigation is
undertaken
 Predict the sequelae
NBME, p. 48
A patient with posthepatitic cirrhosis develops
rapid enlargement of the liver associated with
deterioration of hepatic function.
Serum concentration of which of the
following is most likely to be abnormal?
A.
B.
C.
D.
E.
α1-Antitrypsin
Carcinoembryonic antigen
Chorionic gonadotropin
α-Fetoprotein (*)
Gastrin
 Identify underlying cause/diagnosis
 Identify cause of drug responses
 Identify drug/dosage to administer
NBME, p. 49
Drug Y has a volume of distribution (Vd) of 75 L
in both younger and older adult men. In younger
adults, it has a clearance rate of 15 L/h, 50% of
which is via the liver and 50% via the kidneys. For younger
men, the maintenance regimen is 100 mg every 6 hours.
Which of the following regimens will produce essentially the
same steady-state concentration in an older man, whose
creatinine clearance is reduced to half that of younger men,
but whose hepatic function is unimpaired?
A.
B.
C.
D.
E.
F.
75 mg every 3 hours
(*)
75 mg every 6 hours
75 mg every 9 hours
100 mg every 3 hours
100 mg every 6 hours
100 mg every 12 hours
To minimize odds of guessing correctly
when only five options are given
 Extended Matching format has come into
more use by medical test committees
 Referred to as R-type questions
Extended Matching (R-type)
Theme: Nerves
A.
B.
C.
D.
E.
F.
Median Nerve
Ulnar Nerve
Posterior Interosseous Nerve
Anterior Interosseous Nerve
Axillary Nerve
Radial Nerve
G.
H.
I.
J.
K.
L.
Accessory nerve
Musculoscutaneous Nerve
Long Thoracic Nerve
Deep Branch of Median Nerve
Deep Branch of Ulnar Nerve
Sensory Branch of Radial Nerve
For each of the following patients with motor/sensory impairment,
select the nerve that is most likely to be involved.
A construction site worker fell and landed on his arm. He was initially
treated by a bone setter but is now complaining of an inability to abduct
E
the arm.
Extended Matching (R-type)
Theme: Nerves
A.
B.
C.
D.
E.
F.
Median Nerve
Ulnar Nerve
Posterior Interosseous Nerve
Anterior Interosseous Nerve
Axillary Nerve
Radial Nerve
G.
H.
I.
J.
K.
L.
Accessory nerve
Musculoscutaneous Nerve
Long Thoracic Nerve
Deep Branch of Median Nerve
Deep Branch of Ulnar Nerve
Sensory Branch of Radial Nerve
For each of the following patients with motor/sensory impairment,
select the nerve that is most likely to be involved.
A kitchen worker accidentally injured himself at work, sustaining a
penetrating injury over the forearm. Repair was done in the A&E
department. He now complains of inability to actively flex the IP joint of
the thumb and the DIP joint of the index finger.
D
Extended Matching (R-type)
Theme: Nerves
A.
B.
C.
D.
E.
F.
Median Nerve
Ulnar Nerve
Posterior Interosseous Nerve
Anterior Interosseous Nerve
Axillary Nerve
Radial Nerve
G.
H.
I.
J.
K.
L.
Accessory nerve
Musculoscutaneous Nerve
Long Thoracic Nerve
Deep Branch of Median Nerve
Deep Branch of Ulnar Nerve
Sensory Branch of Radial Nerve
For each of the following patients with motor/sensory impairment,
select the nerve that is most likely to be involved.
An 18 year old martial-arts student blocked an opponent's kick with his
forearm, during a kung-fu tournament. He now complains of an inability to
extend his fingers & thumb.
C
A cyclist was knocked down while riding. He sustained a mid-shaft fracture
of the humerus. Physical examination reveals he is unable to extend his wrist, F
thumb & fingers.
NBME, p. 74
If you want to start using R-type
 Start by selecting your theme
• select from the list of topics provided on the
last page of the attached MS Word document
or pick another
NBME, p. 74
Step Two in writing R-type questions
 Compose a lead-in
• "For each patient described below, select the
most likely (diagnosis / abnormality / injury site /
treatment / etc.
NBME, p. 74
Step Three
 Generate a list of Options fitting the theme
• options should be single words or very short
phrases
• the list is specified in alphabetical order
• unless there is another inherently logical order
Criteria for a good set of options
NBME, p. 75
 Use single words or very short phrases
 Make all homogeneous (e.g., all are diagnoses, or management
steps, or anatomical sites, or vitamins, or enzymes)
 If desired, can be labeled areas in a graph or a picture, or
expressed in tabular form
 Include all relevant options that are appropriate for the examinees'
stage of learning
 Avoid subtle distinctions and uncommon diagnoses if assessing
undergraduates (although might be acceptable for postgraduates)
 Use as few as three options if it is only appropriate to do so for
some questions, while for others, use a list of up to 26 if needed
See Appendix IV
NBME, p. 75
Some Topics Used as a Basis for Option Lists
Arteries
Nerves
Muscles
Amino Acids
Peptides
Hormones
Enzymes
Cell Components
Cell Types
Blood Components
Molecules
Karyotypes
Proteins
Lipids
Pathogens/Bacteria/Fungi
Viruses
Cytokines
Toxins
Vitamins/Minerals
Diagnoses
Connective Tissue Types
Anatomic Structures
Endocrine Structures
Neurotransmitters
Metabolic Defects
Immune Disorders
Motor System Components
Cardiac Structures
Organelles
Congenital Anomalies
Segments of the Spinal Cord
Central Nervous System Components
Secretory Products
Extracellular Matrix Components
Management Alternatives
Drugs/Drug Classes
Pathologic Processes
Pathophysiologic States
Electrolyte Abnormalities
Diagnostic Tests
NBME, p. 74
Step Four
 Write the items (vignettes)
• All items should be similar in structure
• Short patient vignettes are very appropriate
NBME, p. 74
Step Five
 Review the items
• is there a single "best" answer in the option list?
• are there at least four others in the option list
that are reasonably seductive?
• does a colleague agree with your conclusions?
Wrap Up to First Section
If your learning objectives are clinical
reasoning & decision making skills . . .
you might wish to use some of these
ideas.
What & How you assess will influence
 how students study
 what students learn
 how well students develop clinical reasoning
& problem solving skills
Recall the workshop’s objectives
 “At the end of this workshop you will be able to:
• determine which tests & techniques best help students
develop clinical and scientific reasoning skills
• and which simply assess & reinforce their ability to
memorise factual information
• compose MCQs that measure clinical & scientific
reasoning abilities”
It is Apparently Clear
 You accomplished these objectives.
 Congratulations!
 Thank you for attending
Second Section
More on R-type items
(Extended Matching)
Relationship of A-type to R-type Items
• Can change an A-type into a R-type
• See the following illustration
NBME, p. 97
A 5-option A-type item
A patient with the classic phenotypic features of trisomy 21
(Down syndrome) has 46 chromosomes on each of 100
metaphase karyotypes.
Which of the following is the most likely explanation for this
finding?
A.
B.
C.
D.
E.
Deletion
Mosaicism (*)
Somatic mutation
Translocation
Undetected trisomy
NBME, p. 97
To transform this A-type into a R-type (extended-matching) item:
A.
B.
C.
D.
E.
Deletion
Mosaicism
Somatic mutation
Translocation
Undetected trisomy
For each patient with genetic abnormalities, select the genetic
pattern that is most likely to be involved.
A patient with the classical phenotypic features of trisomy 21
(Down syndrome) has 46 chromosomes on each of 100
metaphase karyotypes.
NBME, p. 97
Additional options make it more challenging
A.
B.
C.
D.
E.
Deletion
Genomic imprinting
Mosaicism
Pleiotropy
Reduced penetrance
F. Somatic mutation
G. Translocation
H. Undetected trisomy
I. Variable expressivity
For each patient with genetic abnormalities, select the genetic
pattern that is most likely to be involved.
Four Components for R-type items
• Theme
• Option List
• Lead-in
• 2 or more items (preferably in vignette form)
NBME, p. 71
Theme:
Fatigue
Options:
A.
B.
C.
D.
E.
Acute leukemia
Anemia of chronic disease
Congestive heart fatigue
Depression
Epstein-Barr virus infection
F. Folate deficiency
G. Glucose 6-phosphate
dehydrogenase deficiency
H.
I.
J.
K.
L.
Hereditary spherocytosis
Hypothyroidism
Iron deficiency
Lyme disease
Microangiopathic hemolytic
anemia
M. Miliary tuberculosis
N. Vitamin B12 (cyanocobalamin) deficiency
Lead-in:
For each patient with fatigue, select the most likely diagnosis.
NBME, p. 71
Lead-in:
For each patient with fatigue, select the most likely diagnosis.
Stems:
A 19-year-old woman has had fatigue, fever, and a sore throat for
the past week. She has a temperature of 38.3 oC (101 F), cervical
lymphadenopathy, and splenomegaly. Initial laboratory studies
show a leukocyte count of 5000/mm3 (80% lympho-cytes, with
many lymphocytes exhibiting atypical features). Serum aspartate
aminotransferase (AST, GOT) activity is 200 U/L. Serum bilirubin
concentration and serum alkaline phosphatase activity are within
normal limits.
Answer: E
NBME, p. 71
2. A 15-year-old girl has a two-week history of fatigue and back
pain. She has widespread bruising, pallor, and tenderness
over the vertebrae and both femurs. Complete blood count
shows hemoglobin concentration of 7.0 g/dL, leukocyte count
of 2000/mm3, and platelet count of 15,000/mm3.
Answer: A
Sample Lead-ins & Option Lists
NBME, p. 40
For each of the following patients, select the ________
that is (or is most likely to be) (present / normal / abnormal)?
Options sets could include:
-sites of lesions
-list of nerves
-list of muscles
-list of enzymes
-list of hormones
-types of cells
-list of neurotransmitters
-list of toxins,
-list of molecules
. . . or
NBME, p. 74
For each of the following patients, select the ______ that
would be expected to be found?
Options sets could include:
-list of laboratory results
-list of additional physical signs
-DNA analysis results
-serum levels
-results of microscopic examination of fluids, muscle or joint
tissue
. . . or
NBME, p. 74
For each of the following patients, which _______ is
most likely (defective or deficient or nonfunctioning)?
Options sets could include
-list of enzymes;
-list of feedback mechanisms;
-list of endocrine structures, dietary elements, vitamins.
. . .or
NBME, p. 74
For each of the followin patients, select the (mechanisms,
medications, drugs, agents) that is the most likely cause?
Options sets could include
-list of underlying mechanisms of the disease;
-medications that might cause side effects;
-drugs or drug classes;
-toxic agents;
-hemodynamic mechanisms, viruses, metabolic defects.
. . . or
NBME, p. 40
For each of the following patients, select the _____ that
should be administered
Options sets could include:
-drugs;
-vitamins;
-amino acids;
-enzymes;
-hormones.
NBME, p. 40
For each of the following patients, select the most
appropriate next step in management
Options sets could include list of:
-pharmacologic therapies;
-lab studies;
-or a mix of treatment sets;
-or a mix of laboratories studies;
-or a combo of the above two.
NBME, p. 40
For each of the following patients with (chief complaint),
select the most likely diagnosis
Options sets could include a list of diagnoses:
Microbiology NBME, p. 77
Example good & poor item stems using the same option list.
Theme: Pathogens (viruses and bacteria)
A.
B.
C.
D.
E.
F.
G.
H.
Adenovirus
Aspergillus fumigatus
Bacillus anthracis
Candida albicans
Chlamydia psittaci
Coccidioides immitis
Coronavirus
Corynebacterium
diphteriae
I. Coxiella burnetii
J. Coxsackievirus
K. Epstein-Barr virus
L. Haemophilus influenzae
M. Histoplasma capsulatum
N. Mycobacterium tuberculosis
O. Mycoplasma pneumoniae
P. Neisseria gonorrhoeae
Q. Neisseria meningitidis
R. Pneumocystis carinii
S. Rhinovirus
T. Streptoccus pneumoniae
U. Streptococcus pyogenes
(group A)
NBME, p. 77
For each patient with fever, select the pathogen most likely to have
caused his/her illness.
1. A 7-year-old girl has a high fever and a sore throat. There is
pharyngeal redness, a swollen right tonsil with creamy exudate,
and painful right submandibular lymphadenopathy. Throat culture
on blood agar yields numerous small
β-hemolytic colonies that are inhibited by bacitracin.
Answer: U (Question requires application of knowledge)
2. An encapsulated gram-positive organism that usually grows in
pairs or short chains
Answer: T (Question assesses recall of a questionably useful
fact for medical students)
NBME, p. 78
Theme: vitamins and minerals
Options.
A.
B.
C.
D.
E.
F.
G.
H.
Vitamin A
Vitamin B1
Vitamin B2
Vitamin B6
Vitamin C
Vitamin D
Vitamin E
Vitamin K
I.
J.
K.
L.
M.
N.
O.
P.
Biotin
Copper
Folate
Iodine
Iron
Magnesium
Niacin
Zinc
Lead-in:
For each patient with clinical features caused by metabolic
abnormalities, select the vitamin or mineral that is most likely to be
involved.
NBME, p. 78
For each patient with clinical features caused by metabolic
abnormalities, select the vitamin or mineral that is most likely
to be involved
1. A 70 yr old widower has ecchymoses, perifollicular
petechiae, and swelling of the gingiva. His diet consists
mostly of cola and hot dogs.
Answer: E
(requires a diagnosis, a high order ability)
2. Involved in clotting factor synthesis.
Answer: H
(requires recall of a fact)
Microbiology / NBME, p. 72
Example of Flawed Extended-Matching
A. is motion sickness
B. have no effects on people
C. indirectly increase CO2
D. cause death
E. increased odor sensitivity
F. is a reduction in visibility
G.esthetics, economics, health
H. products of fossil fuel
combustion
I. are completely controlled
J. cause plant and eye damage
K. are negligible
L. increase risk of skin cancer
M. cannot be controlled
N. excess acute respiratory
illness among children
O. contrary to public opinion
1. Factors that people consider when evaluating air quality
2. A principal effect of particulate matter in air
3. The products of photochemical smog
Problems:
What is the common theme? What common lead-in could be used?
NBME, p. 80
Birth Gender
A. 1980
B. 1975
C. 1970
D. 1965
E. 1960
F. 1955
G. 1950
H. 1945
I. 1940
J. 1935
Male
Female
Female
Female
Male
Female
Female
Male
Male
Female
Occupation
Pregnant Childhood
Diseases
History
Medical
Allergy
Student
Painter
Teacher
Lawyer
Painter
Clerk
Nurse
Executive
Driver
Homemaker
—
Yes
Yes
No
—
No
No
—
—
No
None
None
None
None
None
Diabetes
None
Hypertension
Splenectomy
Gastritis
Egg products
Penicillin
None
Gelatin products
Tetanus toxoid
None
None
None
None
Sulfonamides
None
None
None
None
Measles
Mumps
Varicella
Measles
Rubella
Unknown
For each vaccination, select the patient profile that represents its most appropriate use.
1. Measles vaccine
2. Meningococcal vaccine
Problem:
This is clinically backward; given a vaccine, pick the patient
NBME, p. 82
Example R-type for Anatomy
Theme: Arteries
A. Left anterior cerebral artery
B. Right anterior cerebral artery
C. Left middle cerebral artery
D. Right middle cerebral artery
E. Left posterior cerebral artery
F. Right posterior cerebral
artery
G. Left lenticulostriate arteries
H. Right lenticulostriate arteries
Lead-in:
For each patient with neurologic abnormalities, select the artery
that is most likely to be involved.
NBME, p. 82
1. A 72-year-old right-handed man has weakness and hyperreflexia of the right lower limb, an extensor plantar response
on the right, normal strength of the right arm, and normal
facial movements.
Answer: A
2. A 68-year-old right-handed man has right spastic hemiparesis, an extensor plantar response on the right, and
paralysis of the lower two-thirds of his face on the right.
His speech is fluent, and he has normal comprehension
of verbal and written commands.
Answer: G
NBME, p. 83
Example: R-type for Pharmacology
Theme: Drugs
A. Acetaminophen
B. Amiodarone
C. ACE inhibitors
D. Aspirin
E. Atenolol
F. Bleomycin
G.Cytosine arabinoside
H. Furosemide
I. Metronidazole
J. Nalidixic acid
K. Nitrofurantoin
L. Penicillin
M. Prednisone
N. Procainamide
O. Propranolol
P. Sulfasalazine
Q. Tetracycline
R. Verapamil
Lead-in:
For each patient, select the drug most likely to have caused
the adverse effect.
NBME, p. 83
1. A 56-year-old man with recurrent ventricular arrhythmias
began taking an antiarrhythmic drug 5 months ago. He now
has progressive dyspnea, cough, and low-grade fever.
Erythrocyte sedimentation rate is increased. X-ray film of the
chest shows a diffuse interstitial pneumonia. Pulmonary
function tests show that diffusing capacity for carbon
monoxide is decreased.
Answer: B
2. A 62-year-old man with chronic obstructive pulmonary
disease begins therapy with an antihypertensive drug. Two
weeks later, he has marked worsening of dyspnea and
clearly audible wheezing.
Answer: O
NBME, p. 84
Example: R-type for Physiology
Theme: Arterial Blood Gas Findings
A.
B.
C.
D.
E.
F.
G.
pH
7.15
7.15
7.30
7.40
7.50
7.50
7.50
PO2 mm Hg
98
98
56
100
100
100
56
PCO2 mm Hg
33
24
80
40
33
24
33
HCO3 – mEq/L
11
8
38
25
25
18
25
Lead-in:
For each patient described below, select the most likely arterial
blood gas finding.
NBME, p. 84
For each patient described below, select the most likely arterial
blood gas finding.
1. A 22-year-old man with a 3-week history of polyuria and
polydipsia has had nausea, vomiting, and decreased
responsiveness for the past 12 hours. Urinalysis (dipstick)
shows 4+ glucose and 4+ ketones.
Answer: B
2. A 25-year-old woman is brought to the emergency
department 12 hours after a suicide attempt. She took
approximately 100 500-mg aspirin tablets.
Answer: F
NBME, p. 85
Example: R-type items for Diagnosis
Theme: Back Pain
A. Ankylosing spondylitis
B. Intervertebral disc infection
C. Multiple myeloma
D. Myofacial pain
E. Osteoporosis
F. Spinal stenosis
G. Spondylolysis
H. Tuberculosis of the spine
Lead-in:
For each patient with back pain, select the most likely diagnosis.
NBME, p. 85
1. A 26-year-old man has insidious onset of low back pain and
early morning stiffness. The pain alternates from side to side
and occasionally radiates into the buttocks and back of the
thighs, but not below the knees. The patient has acute anterior
uveitis, diffuse low back and sacroiliac tenderness, and
restricted range of motion at the hips. His erythrocyte
sedimentation rate is 40 mm/h; latex fixation test is negative;
and mild hypoproliferative anemia is present.
Answer: A
2. Twelve hours after being struck from the rear while driving her
car, a 28-year-old woman has vague cervical and lumber pain
associated with headache and restricted cervical mobility. She
is now very anxious. Rope-like bands of muscle are present in
the lumbar area and over the left buttock; the bands are painful.
Answer: D
NBME, p. 88
Example: R-type items for Management:
Theme: diagnostic testing
A. Test of the stool for occult blood
B. Fasting serum glucose level
C. Hemoglobin level
D. Prostate-specific antigen level
E. Serum cholesterol level
F. Serum iron level
G. Thyroid function test
H. Exercise tolerance test
I. Digital prostate examination
J. ECG
K. Spirometry
L. X-ray film of the chest
M. Sigmoidoscopy
Lead-in:
For each patient who comes to the physician for a health
maintenance examination, select the most appropriate
diagnostic study.
NBME, p. 88
1. A 22-year-old man who weighs 89 kg (196 lb) and is 175 cm
(69 in) tall has smoked one pack of cigarettes daily for 8 years;
he does not exercise. His last examination was 5 years ago.
His father had a myocardial infarction at the age of 48 years.
Physical examination shows no abnormalities.
Answer: E
2. A 28-year-old woman who weighs 70 kg (154 lb) and is
173 cm (68in) tall has smoked one pack of cigarettes daily for
12 years; she does not exercise. Her last examination was 5
years ago, though she had a Pap smear 9 months ago that
showed normal results. Her father had a myocardial infarction
at the age of 48 years. Her grandmother was diagnosed with
colon cancer at the age of 62 years. Physical examination
showed no abnormalities.
Answer: E
NBME, p. 81
Review: the steps for writing extended-matching items
1. Identify the common theme for the set.
2. Write the common lead-in for the set.
3. Prepare a list of options that fit the theme.
4. Write the items, preferably in vignette format.
5. Review the items
a. place emphasis on application not recall
b. remember to avoid the same writing mistakes that
apply to A-type questions
Multiple Response Items
Includes
Pick n best choices from N options (Pick n)
&
Key Features
If the next best step or best option is not a
singular case (or is not known to singular)
• To simulate clinical situations one must allow for
instances
– when multiple next steps are realistic and optimal
– where is no unanimity among experts as to what is the best
or most likely
• To accommodate for these instances use another
format
– i.e., Pick n choices from N options
Example of Pick n
A. Diabetic polyneuropathy
B. Huntington’s disease
C. Lateral medullary syndrome
D. Lead encephalopathy
E. Medulloblastoma
NBME, p. 100
F. Multiple sclerosis
G. Parkinson’s disease
H. Pontine glioma
I. Tabes dorsalis
J. Wernicke’s encephalopathy
A 50-year-old man has the gradual onset of mental confusion,
disorientation, and loss of short-term memory. He has a left
footdrop. A blood smear shows microcytosis and basophilic
stippling of erythrocytes.
Select the two most likely diagnoses
NBME, p. 100
With the previous item, there would be disagreement about
which diagnoses are likely, but the task becomes clearer if the
examinee is told to select the two most likely diagnoses.
The options can be laid out along a diagnostic continuum
something like the following:
F
C
E D
A G
H
B
__________________________________________________
Least likely diagnosis
Most likely diagnosis
Even if disagreement exists about H & B, there is agreement that both
are more likely than all the others.
Additional examples for Pick n
A. Calcium
B. Fluoride
C. Folic acid
D. Iron
E. Vitamin A
NBME, p. 100
G. Vitamin B6
H. Vitamin B12 (cyanocobalamin)
I. Vitamin C
J. Vitamin D
K. Vitamin E
For each child select the appropriate vitamin or mineral
supplements.
1. A 1-month-old infant is brought to the physician for a wellchild examination. He has been exclusively breast-fed, and
examination shows normal findings. (SELECT 2 SUPPLEMENTS)
Answer: B, J
NBME, p. 100
2. A 6-year-old girl has cystic fibrosis. She has been taking no
Medications. (SELECT 3 SUPPLEMENTS)
Answer: E, J, K
Additional examples of Pick n
A.
B.
C.
D.
E.
F.
G.
H.
I.
NBME, p. 101
Analysis and culture of cerebrospinal fluid
Blood culture
Complete blood count
Examination of the stool for leukocytes
Measurement of serum electrolyte levels
Urinalysis
Urine culture
X-ray film of the abdomen
X-ray film of the chest
For each child with fever, select the appropriate initial diagnostic
studies.
NBME, p. 101
1. A previously healthy 1-year-old girl is brought to the
emergency department because of fever for 1 day. Her
temperature is 41oC (105.8 F). She is otherwise
asymptomatic. Physical examination shows no
abnormalities. (SELECT 4 STUDIES).
Answer: B, C, G, I
2. A previously healthy 10-day-old newborn is brought to the
emergency department because of fever for two hours. He
was born at term after an uncomplicated pregnancy. His
temperature is 39oC (102.2 F). Physical examination shows
no abnormalities. (SELECT 6 STUDIES)
Answer: A, B, C, E, G, I
NBME, p. 101
3. A 7-year-old boy with sickle cell disease is brought to the
emergency department because of fever for 1 day and chest
pain for 1 hour. His temperature is 39.5oC (103.1 F). Breath
sounds are slightly decreased in the right lower lung; he is not
in respiratory distress. (SELECT 3 STUDIES)
Answer: B, C, I
PATIENT CHART
Patient History
Sex:
Current age:
male
28 years
Chief complaint:
health maintenance examination
Social history:
Marital status:
Occupation:
Alcohol:
Smoking:
Exercise prgm:
single
computer programmer
2-4 beer/weekend
5-10 cigarettes daily from age 16-24
sedentary
Medical history:
Childhood:
Immunizations:
Screening:
NBME, p. 102
obese since grade school
all childhood immunizations; last tetanus toxoid age 15;
no immunizations since childhood.
no physician visits since college
Family history:
Parents:
Siblings:
Children:
father age 57; hypertensive mother age 55; obese & hypertensive
none
none
Current meds:
none
Allergies:
none
NBME, p. 102
Physical Examination
Height:
Weight:
178 cm (70 in)
134 kg (295 lb)
Vital signs:
Blood pressure:
Pulse:
Respirations:
148/86 mm Hg
90/min
16/min
Skin:
erythematous rash in groin
Abdominal:
obese
Laboratory studies:
none ordered
NBME, p. 103
1. For the patient whose chart is shown, select the conditions
for which he is at increased risk.
SELECT 4 CONDITIONS
A. Alcoholism
B. Colon cancer
C. Coronary artery disease
D. Diabetes mellitus
E. Hemochromatosis
F. HIV
G. Hypertension
H. Hypothyroidism
I. Osteoarthritis
J. Skin cancer
K. Thyroid cancer
L. Urinary tract infection
NBME, p. 103
2. For the patient whose chart is shown, select the most
appropriate health maintenance interventions.
SELECT 4 INTERVENTIONS
A. Complete blood count
B. Dietetic counseling
C. Exercise counseling
D. Fasting serum lipid profile
E. Hepatitis B immunization
F. HIV testing
G. Influenza immunization
H. Serum urea nitrogen (BUN)
& creatinine level measmnts
I. Tetanus toxoid vaccine
J. Thyroid function tests
K. Urinalysis
L. X-ray film of the chest
Summary
These are the common selected-response
formats that medical test committees use
Following the guidelines for their items’ development (provided
in the foregoing), one can better meet the principle criteria for
all assessments:
1. Objective
(correct or keyed response is indeed correct)
2. Reliable
(consistently reproducible)
3. Valid
(adequately measures intended trait/ability)
4. Defensible
(when challenged by examinees/society)
5. Efficient
(time and cost is justified)
6. Practical
(usable within educational/assessment context)
7. Educational
(aids in driving learning in desired direction)
There are additional assessment formats
• Including other selected-response formats & a selection of
constructed-response formats
• The Hong Kong Consortium for Sharing Medical Student
Assessment Banks & its IDEAL system platform
– supports use of selected-response items discussed in this presentation
– but also:
•
•
•
•
•
Key Features (special case of Pick n)
OSCE (Objective Structure Clinical Examination)
OSPE (Objective Structure Physical Examination)
MEQ (modified essay question)
SAQ (short answer quesiton)
• Other presentations have been prepared on these additional
supported formats and are included on the CD
The Hong Kong International Consortium hopes these guidelines
& presentation prove to be useful to each of you.
The Consortium gratefully acknowledges the use of Swanson, Case
& NBME’s materials in the preparation of this training package