Transcript Learning Objective

Pitfalls of Writing Learning Objectives and Writing
Multiple Choice Exam Questions
Beth DeJongh, Pharm.D., BCPS, BCPP
Assistant Professor of Pharmacy Practice
Susan Gallanis, MS, CI and CT
Instructional Development Specialist, CELT
1
Learning Objectives
 Evaluate the components of learning objectives using the ABCD model.
 Revise learning objectives based on the common pitfalls described during this
session.
 Revise multiple choice exam questions based on the common pitfalls described
during this session.
 Correctly link learning objectives and multiple choice exam questions to more
effectively assess student learning.
2
Resources
The CELT Blog: http://bit.ly/WriteLearnObj (bit.ly/WriteLearnObj)
 Blooms Taxonomy
 Sample Verbs
 Writing Learning Objectives
 How to Write Better Tests
 Writing Multiple Choice Questions
3
Learning Objectives
4
Points to Ponder
 Is this learning objective measurable, observable, and specific?
 Will the learner know what is expected of them after reading the objective?
 Does this learning objective focus on important information the student should
know?
 Guides writing of exam questions
 Is this learning objective written at an appropriate level of learning?
 Highlights importance of Bloom’s Taxonomy
5
Writing Learning Objectives: A Brief Review
 A = Audience
 Almost always the student
 B = Behavior
 What do you want students to be able to do?
 Measurable, observable
 C = Condition
 What the learner will use, have access to, real world condition
 “Given the…”
 D = Degree
 How well, measurement of criteria
 Might be quality, accuracy, based on time/speed, or simply “without error”
6
Pitfalls of Writing Learning Objectives
 Including more than one verb per learning objective
 Lacks focus
 Using verbs that are not measurable/observable or specific
 Makes assessment difficult
 Missing “condition”
 Missing “degree” when not assumed
 Do not cover a variety of levels of Bloom’s Taxonomy
7
Pitfalls with LOs
 Do not cover a variety of levels of Bloom’s Taxonomy
 Easy exam versus a very difficult exam
 May not discriminate between those who know the material well and those who do not
 May not allow lower level learners to demonstrate their knowledge
8
Active Learning Time
9
Pitfalls with LOs: Examples 1&2
 Including more than one verb per learning objective
 Lacks focus
10
Multiple verbs
One verb
Classify a patient with an anxiety disorder (GAD, SAD,
PD, PTSD, OCD) and recommend a first-line
treatment for the disorder.
Classify a patient with an anxiety disorder (GAD, SAD,
PD, PTSD, OCD) based on the patient’s presenting
symptoms and DSM-5 diagnostic criteria.
Evaluate a patient’s characteristics (age, comorbid
conditions, concomitant medications, renal/hepatic
function) to select an antidepressant when given a
patient case.
Design a pharmacologic treatment plan for a patient
with depression using patient specific characteristics
(age, comorbid conditions, concomitant medications,
renal/hepatic function).
Pitfalls with LOs: Examples 3&4
 Using verbs that are not measurable/observable or specific
 Makes assessment difficult
Vague/Ambiguous Verbs
Measurable/observable and specific
Know the potential drug-drug and drug-food
interactions associated with antidepressants (SSRIs,
SNRIs, MAOIs, TCAs).
Describe the potential drug-drug and drug-food
interactions associated with a specific antidepressant
(SSRI, SNRI, MAOI, TCA) when given a medication
profile.
Understand strategies for managing the
extrapyramidal symptoms associated with
antipsychotics.
Recommend strategies for managing the
extrapyramidal symptoms associated with
antipsychotics based on a patient’s presenting
symptoms.
Avoid words like: know, understand, comprehend, appreciate, believe, recognize, and memorize
11
Pitfalls with LOs: Examples 5-8
 Missing “condition”
Missing Condition
Condition Included
Develop a monitoring plan for metabolic side effects associated
with second-generation antipsychotics.
Describe strategies for managing the adverse effects associated with
stimulants.
Apply the monitoring protocol recommended by the 2004
ADA/APA consensus on antipsychotic drugs and obesity and
diabetes to a patient taking a second generation antipsychotic for
schizophrenia.
Explain relevant monitoring parameters for the tricyclic
antidepressants.
Describe strategies for managing common adverse effects
associated with stimulants for patients with ADHD.
Design treatment regimens for supratherapeutic INRs or active
bleeds.
Decide which patients taking a tricyclic antidepressant for
depression should receive an EKG based on their age and
comorbidities.
Design a treatment regimen for a supratherapeutic INR or active
bleed when given a patient case on warfarin.
12
Pitfalls with LOs: Examples 9-11
 Missing “degree” when not assumed
Missing Degree
Degree Included
Calculate a patient’s creatinine clearance using a calculator.
Calculate a patient’s creatinine clearance within 5 mL/min of the
correct value when using a calculator.
List monitoring parameters required with lithium according to the
APA guidelines.
Educate a patient on the adverse effects associated with the SSRIs.
13
List three monitoring parameters required with lithium according
to the APA guidelines.
Educate a patient on the common adverse effects associated with
SSRIs within 10 minutes.
Multiple Choice Exam Questions
14
Points to Ponder
 Does this exam question test important information the examinee should know or
does it test a trivial detail or obscure fact?
 Would a typical pharmacist in your field be able to answer this question or would they
have to look it up?
 Highlights importance of linking exam questions to learning objectives
 Does this exam question reflect the appropriate level of training?
 Is this something a student should know or would this be more appropriate for a
resident to understand?
 Highlights importance of carefully selecting Bloom’s verbs in learning objectives
 Is this exam question written in a manner that is defensible, focused, clear, and
unambiguous?
15
Boland R, Lester N, Williams E. Writing Multiple-Choice Questions. Academic Psych 2010; 34(4): 310-316.
Pitfalls with MC Exam Questions
 Writing unfocused stems
 Reward recognition over recall
 Confusing
 Contain irrelevant details
 Test more than one concept
 Superfluous information in the stem
 Detracts from the primary focus
 Increases reading time
 Increases complex sentences and awkward wording
 Negatively phrased questions
 Makes question unnecessarily difficult
 May not discriminate between those who know the material well and those who do not
 Distractors that differ in content, length, or grammar
 May not discriminate between those who know the material well and those who do not
16
Pitfalls with MC Exam Questions
Unfocused Stem
Focused Stem
Which of the following is true of fluoxetine?
a. It is structurally a heterocyclic compound.
b. A common side effect is weight loss.
c. It is fully metabolized in 48 hours.
d. It requires a 4-week wash-out period before starting an MAOI.
Switching antidepressant agents from fluoxetine to phenelzine
requires a 4-week wash-out period due to increased risk of
developing which of the following?
a. Stevens-Johnson Syndrome
b. Serotonin Syndrome
c. Neuroleptic malignant syndrome
d. Agranulocytosis
17
Boland R, Lester N, Williams E. Writing Multiple-Choice Questions. Academic Psych 2010; 34(4): 310-316.
More Appropriate Use of Cases
A 27-year-old patient with a history of bipolar disorder presents to the hospital with an 8-day history of insomnia, pressured speech, flight of ideas,
hyperactivity and delusions. The first manic episode began at the age of 22. Since then, the patient has had numerous manic episodes (~2 episodes per
year) with occasional depressive episodes (~1 episode per year). The past medical history is significant for lower back pain that he adequately treats with
as needed naproxen (Naprosyn®) The laboratory work-up was unremarkable other than a sodium level of 130 mEq/L (normal 135-145 mEq/L).
Which one of the following medications would be MOST appropriate for this patient based on his presenting symptoms, past medical history, and lab
results?
a. Carbamazepine (Tegretol®)
b. Divalproex sodium (Depakote®)
c. Lamotrigine (Lamictal®)
d. Lithium (Lithobid®)
A 17-year-old comes to your clinic for follow-up monitoring of attention deficit hyperactivity disorder (ADHD) medications. The past medical history is
significant for partial complex seizures. The patient’s mother is present at the appointment and reports he/she was caught selling Adderall® (mixed
amphetamine salts) at school and is no longer going to receive prescriptions for controlled substances. She asks for your recommendation for an
alternative treatment option.
Physical Exam:
Weight: 81kg
Height: 70 inches
Blood Pressure: 100/68 mmHg
Heart Rate: 52 bpm
Respiratory Rate: 16 rpm
Which of the following medications is the BEST alternative medication option for this patient?
18
a. Atomoxetine (Strattera®) b. Bupropion (Wellbutrin®) c. Guanfacine ER (Intuniv®) d. Methylphenidate (Concerta®)
Pitfalls with MC Exam Questions
Examples
Venlafaxine works by reuptake of which of the following neurotransmitters?
a. Serotonin and histamine
b. Dopamine and norepinephrine
c. Serotonin and norepinephrine
d. Norepinephrine and acetylcholine
A patient with Parkinson’s disease presents to your pharmacy to pick up a refill of their carbidopa/levodopa 10/100 that they take three times daily.
You note they should have run out 2 weeks ago and politely question the patient about their adherence. They tell you that they often skip doses
because it makes them nauseous. What can be done to help this patient?
a. Adjunctive therapy should be considered.
b. The dose of carbidopa should be increased to prevent the peripheral conversion of levodopa to dopamine.
c. The patient can take this 30 minutes before meals.
d. Evaluate for an infectious cause of the nausea.
A 23-year-old patient presents to his psychiatrist with complaints of an increased number of panic attacks over the past month and worries about
having more. The patient has been drinking 1/5th of vodka on a daily basis to “take the edge off ”. Which of the following is the BEST treatment
option?
a. Clonazepam as needed and escitalopram
b. Propranolol and escitalopram
c. Clonazepam bridge and escitalopram
d. Escitalopram
19
Active Learning Time
20
Pitfalls with MC Exam Questions: Example 1
Unfocused Stem
Focused Stem
Which of the following pharmacokinetic statements about mood
stabilizers is TRUE?
a. Carbamazepine (Tegretol®) and oxcarbazepine (Trileptal®)
induce their own metabolism which affects the time to steady
state concentration.
b. Carbamazepine (Tegretol®) decreases divalproex sodium
(Depakote®) concentrations through competition at protein
binding sites.
c. Unbound divalproex sodium (Depakote®) may produce signs
and symptoms of toxicity without changes in total drug
concentration.
d. Divalproex sodium (Depakote®) significantly induces the
metabolism of lamotrigine (Lamictal®) thus requiring
careful dose titration to prevent rash.
Which of the following mood stabilizers increases a patient’s risk
for developing a serious rash by inhibiting the metabolism of
lamotrigine (Lamictal®)?
a. Carbamazepine (Tegretol®)
b. Divalproex sodium (Depakote®)
c. Lithium (Lithobid®)
d. Oxcarbazepine (Tegretol®)
21
Pitfalls with MC Exam Questions: Ex 2
Superfluous
Non-superfluous
A 30yo physically healthy patient with a diagnosis of major depressive disorder is currently experiencing
another episode of depression. The patient admits to missing doses of his medications 2-3 times per week
due a hectic home and work schedule. A request is made for a long-acting shot, “like with Haldol.” On
assessment, the patient appears generally healthy with stable vital signs and an unremarkable physical
examination.You explain that, although there is not currently an antidepressant injection, you can offer
the medication with the longest half-life. Which of the following would be prescribed to this patient?
a. Citalopram
b. Fluvoxamine
c. Paroxetine
d. Fluoxetine
Which of the following
SSRIs has the longest
elimination half-life?
a. Citalopram
b. Fluvoxamine
c. Paroxetine
d. Fluoxetine
22
Boland R, Lester N, Williams E. Writing Multiple-Choice Questions. Academic Psych 2010; 34(4): 310-316.
Question 101 Writing Video
Pitfalls with MC Exam Questions: Ex 3
Superfluous
You are working in CUW’s research laboratory and identify
a group of people who appear to have a genetic
hypercoagulability disorder, in which they produce excess
Factor II. You decide you would like to help these people by
inventing an oral medication that directly inhibits thrombin
(Factor IIa). You begin the drug development process by
basing the mechanism off an injectable anticoagulant that is
able to directly inhibit thrombin. Which of the following
injectable anticoagulants is a direct inhibitor of thrombin?
a. Bivalirudin (Angiomax®)
b. Enoxaparin (Lovenox®)
c. Fondaparinux (Arixtra®)
d. Unfractionated heparin
23
Non-superfluous
Which of the following injectable anticoagulants is a
direct inhibitor of thrombin?
a. Bivalirudin (Angiomax®)
b. Enoxaparin (Lovenox®)
c. Fondaparinux (Arixtra®)
d. Unfractionated heparin
Pitfalls with MC Exam Questions: Ex 4
Negatively Phrased
Positively Phrased
A 40-year-old woman with a history of breast cancer is
taking tamoxifen. She presents for treatment of
depression. Which of the following antidepressants
should not be prescribed?
A 40-year-old woman with a history of breast
cancer is taking tamoxifen. She presents for
treatment of depression. Which of the following
antidepressants is MOST appropriate for this
patient?
a.
b.
c.
d.
e.
24
Desvenlafaxine
Amitriptyline
Mirtazapine
Paroxetine
Venlafaxine
Boland R, Lester N, Williams E. Writing Multiple-Choice Questions. Academic Psych 2010; 34(4): 310-316.
a.
b.
c.
d.
e.
Sertraline
Fluoxetine
Paroxetine
Venlafaxine
Escitalopram
Additional Pitfalls with MC Exam Questions
 Using “all of the above” or “none of the above” as distractors
 Low ability to discriminate
 Can correctly answer without fully understanding the material
 All of the above
 Can rule it out if you can rule out one distractor
 None of the above
 Can select it if you know two alternatives are correct
25
Boland R, Lester N, Williams E. Writing Multiple-Choice Questions. Academic Psych 2010; 34(4): 310-316.
Question 101 Writing Video
Additional Pitfalls with MC Exam Questions
 Implausible distractors
 Avoid silly distractors
 Avoid distractors that do not or cannot exist
 No consensus on 4 versus 5 answer choices
 Use only as many distractors as are plausible
 Utilize psychometrics in Exam Soft to help make determination
 Research suggests 4 answer choices is optimal
 Additional distractors increase difficulty but not discrimination
26
Tresivan M. Estimating the optimum number of options per item using an incremental option paradigm. Educ Psychol Meas 1994; 54: 86-91.
Taylor A. Violating conventional wisdom in multiple choice test construction. College Student J 2005; 39: 141.
Swanson D, Holtzman K, Clauser B, et al. Psychometric characteristics and response times for one-best-answer questions in relation to number and source of options. Acad Med 2005; 90: S93-96.
Boland R, Lester N, Williams E. Writing Multiple-Choice Questions. Academic Psych 2010; 34(4): 310-316.
Question 101 Writing Video
Linking Learning Objectives and
Exam Questions
27
Points to Ponder
 Does the exam question reflect the behavior expected from the learning objective?
 Identify the content the students are interacting with
 Identify what students are doing with the content
 Reevaluate the Bloom’s level and the appropriateness of the specific verb
28
Compare the following:
 Learning Objective
 Determine State the appropriate washout period required when switching to or from an MAOI to help
prevent serotonin syndrome.
 The Exam
A 36yo female presents to your clinic with worsening depression. She has been taking the maximum
recommended dose of citalopram (Celexa®) for the past three months and has not noticed improvement in
her symptoms. She has already tried and failed multiple other first-line antidepressants. After speaking to
the team psychiatrist, you decide to switch the patient to phenelzine (Nardil®).
What is the minimum washout period required between discontinuing citalopram and initiating phenelzine? (1 point)
(Objective 5) (Evaluation – Blooms level 6)
2 weeks
What potentially life-threatening condition is the patient at risk for if the washout period between citalopram and
phenelzine is too short? (1 point) (Objective 3) (Comprehension – Blooms level 2)
Serotonin syndrome
29
Pitfalls with Linking LOs and Exam Questions
 Exam question and LO do not match
Learning Objective
Exam Question
Revised Exam Question
Design a pharmacologic treatment
plan for an agitated patient based
on patient specific characteristics
(e.g. age, comorbid conditions,
concomitant medications,
behavior).
Which of the following agents is
available as a short acting injectable?
a. Aripiprazole
b. Lurasidone
c. Quetiapine
d. Risperidone
An 80yo patient is brought to the ER by family for increased
paranoia. The patient is confused and screaming at family
members. Which of the following medications would be the BEST
option to start with?
Describe potential drug-drug and
drug-food interactions when
presented with a specific
antidepressant and medication
profile.
Which of the following
antidepressants is MOST likely to
be involved in CYP-mediated drugdrug interactions?
a. Bupropion
b. Citalopram
c. Fluoxetine
d. Venlafaxine
A patient presents to your clinic with worsening depression
despite taking the maximum dose of citalopram for the past 6
months. The patient has already tried and failed multiple other
first-line antidepressants. A decision is made to switch the patient
to phenelzine.
What type of foods should the patient avoid while taking
phenelzine?
What potentially life-threatening condition is the patient at risk
for if he/she does not follow the dietary advice you provide?
30
Pitfalls with Linking LOs and Exam Questions
 Intended level of Blooms does not match the difficulty level of the exam question
Learning Objective
Intended Bloom’s
Level
Compare and contrast the significant
Analysis
adverse effects associated with SSRIs,
SNRIs, bupropion, mirtazapine, MAOIs,
TCAs, and the serotonin
antagonist/reuptake inhibitors for patients
with depression.
31
Exam Question
Actual Bloom’s
Level
Which of the following SSRIs is MOST likely to
cause or worsen constipation in an elderly
patient?
a. Escitalopram
b. Fluoxetine
c. Paroxetine
d. Sertraline
Comprehension
Learning Objectives
 Evaluate the components of learning objectives using the ABCD model.
 Revise learning objectives based on the common pitfalls described during this
session.
 Revise multiple choice exam questions based on the common pitfalls described
during this session.
 Correctly link learning objectives and multiple choice exam questions to more
effectively assess student learning.
32
Questions
33