Transcript Training in Higher Education: Assessment and Evaluation

Ethics Education: Considerations for
Development and Evaluation
Presented at:
University of Illinois - Champaign
May 2012
Presented by:
Dr. Mike Mumford
Ethics Education: Needs Assessment
• New federal mandates highlight need for
effective, validated ethics education programs
• Yet, validated instruments or methods for
evaluating ethics programs are not applied
• Moreover, traditional approaches to ethics lack
complexity and empirical evidence
Why Evaluate Ethics Education?
• Specific federal mandates to do so
• Quantify the real value of training program
• Ethics education is a loaded gun
– It may encourage ethical behavior or it may actually
discourage it, and the consequences for either are great
– Some (widely-used) programs haven’t worked……
Traditional Ethics Programs
• Kohlberg-Based Training
– Help individuals move from one level of moral
development to another
– Emphasize others’ rights or philosophical rights of
humanity in making ethical decisions
– Deemphasize self-interest in ethical dilemmas
• CITI Training
– Provide instruction on the major ethical guidelines
and rules – all declarative knowledge
Meta-analyses of Prior Ethics Course
Results
Overall Effectiveness
Criterion Type
Moral Development
Ethical Analysis
Program Quality
Below Average
Average
Above Average
Study Quality
Below Average
Average
Above Average
Investigator’s Field Health/Medicine
Philosophy
Psychology
Mandatory Course No
Yes
Program Setting
Integrated into Curriculum
Stand-alone Seminar
Md
.42
.36
.61
.18
.39
.72
.16
.48
.65
.38
.54
.80
.53
.32
.37
.51
SD
.27
.26
.16
.25
.02
.15
.23
.20
.20
.17
.21
.00
.25
.23
.22
.30
k
26
17
9
5
8
6
10
12
4
9
4
6
10
13
11
15
N
3041
2229
812
674
1301
445
817
1688
536
1313
648
441
909
1818
1832
1209
Md
Instructional Approach Decision-Making
.52
Ethical Sensitivity
.42
Moral Development
.17
Type of Skills
Real-World, Global
.64
Limited, Domain-Specific .35
Reasoning Errors
No
.33
Yes
.57
Strategies
No
.22
Yes
.52
Ethical Domains
No
−.11
Yes
.48
Delivery Method
Classroom-Based
.36
Case-Based
.53
Learning Activities 3 or more
.12
4 or more
.48
Participant Interaction Low
.05
Moderate
.37
High
.63
SD
.15
.11
.28
.11
.27
.17
.30
.25
.20
.14
.14
.25
.14
.34
.14
.10
.16
.09
k
9
7
4
9
15
10
9
7
13
4
16
8
9
8
13
4
6
7
N
1234
780
619
744
2257
1845
575
1229
1245
460
2014
1091
1214
692
1995
411
1198
722
Ethics Education Evaluation:
Federal RCR Mandates
• RCR training required in exchange for federal
funds
– Faculty, graduate students, some undergraduates
– No online substitute
• Evidence of training effect required
– Validated measurement
• Failure to comply will have implications for grant
and contract awards
– Institutions are requiring training for IRB
Ethics Education Evaluation:
Federal Standards
• United States Code and the Code of Federal
Regulations requires that all training programs be
evaluated
– Annual evaluation to demonstrate program
effectiveness towards organizational goals
• Executive Order 11348: Training for all
government sponsored employees
– Agency heads must plan, implement, evaluate, and
maintain records of training programs
– Agency heads must monitor training effectiveness and
actively conduct research to improve program
effectiveness
Ethics Education Evaluation:
Quality Control
• Systematic evaluation provides insight into
program success
– What worked and why did it work
• Systematic evaluation provides insight into
program failure
– Is failing a structural problem or a conceptual one
• Increases public perception that program has real
value
– Increases stakeholder “buy-in”
Evaluation Methods
• Kirkpatrick’s four level model of training
evaluation
– Trainee reactions: how do trainees like elements of
the training
• Affect-based; immediate and short-term
– Learning: Tests to determine acquired declarative
knowledge of training material
Evaluation Methods
• Kirkpatrick’s four level model of training
evaluation
– Behavior: Transfer or application of trained knowledge
and skills (i.e. trainees’ post-training application of
ethics education content)
– Results: Impact of training on organizational outcomes
(e.g. actual increases or decreases in ethical
misconduct, financial gains or losses as a result of
training outcomes)
Sensemaking Training
• Ethics education programs should be grounded in
realistic theory of ethical behavior
• Traditional models to ethics emphasized:
– Moral development
– Rational, black-and-white models
• Sensemaking training is grounded in EDM
Sensemaking Training Overview
• Two-day seminar (16 hours of instruction)
• 10 modules
– 2 out-of-class modules
– 8 in-class modules
• Key Instructional Objectives
– Develop understanding of the ambiguous, complex
nature of problems that may be encountered in
professional work
– Learn strategies that facilitate identification, analysis,
and acting within ethical situations
Sensemaking Training Content
•
•
•
•
•
•
Knowledge of guidelines
Awareness of own biases and common errors
Model of ethical decision-making (EDM)
Situational analysis and interpretation
Strategies, or tools, for decision-making
Field differences
Sensemaking Training Validation
• Social & biological sciences graduate student
sample
• Engineering graduate student and professional
sample
• Undergraduate REU sample
• Pre-post ethical decision-making test
– Outcomes examined for pre-post changes
• Ethicality of decisions
• Strategy use
Sensemaking Training Validation
Effect Size (Cohen’s d)
Scientist Engineer
Undergrad
Sample
Sample
Sample
Decision-Making Ethicality
Data Management
Study Conduct
Professional Practices
Business Practices
Strategies
Recognizing one’s circumstances
Seeking help
Questioning one’s judgment
Dealing with emotions
Anticipating consequences
Analyzing personal motivations
Considering others’ perspectives
* Effects held over 6-month follow-up period
.66**
1.46**
.61**
.49*
.53*
.66*
1.06**
1.82**
1.79**
.34
1.68**
1.09**
1.24**
.84**
1.27**
.90**
.77**
1.36**
.45**
.16
−0.65*
1.92**
1.51**
.33
.60*
.08
.35
−0.01
2.45**
2.56**
.79**
1.05**
.27
EDM Measure Development
• SMEs generate field-specific ethical examples
– Real-world, ambiguous examples
• Equal representation of EDM items across ethical
behavior taxonomy
–
–
–
–
Study Conduct
Data Management
Professional Practices
Business Practices
EDM Measure Development
• Parallel forms of measure for pre-post format
– Test forms counterbalanced
• “Pick two” scoring scheme
– Pick the “best” two out of possible eight
• Code response options for:
– Ethicality of Response
– Strategies underlying response
– Social motives underlying response
Example EDM Measure Item
(Social Science)
Dr. Cedar, a young developmental psychologist, obtained an Early Career Research Grant from the
National Institute of Child Health and Human Development to study aggression in elementary
school children. Cedar suspects that some children with a certain genetic makeup will be especially
susceptible to the effects of television violence. Part of the project requires obtaining a cheek swab
for DNA analysis, but interviewing and observing children in the classroom constitutes the major
effort. Cedar is collaborating with a well-known senior social psychologist, Dr. Dawson. Given the
geographic distance between their universities, the labs communicate primarily via email.
1.
Considering the sensitive nature of the project, Cedar is concerned that parents will be reluctant to
allow their children to participate. He is writing the informed consent form and worries that too
much detail might discourage participation. How should Cedar handle this issue? Choose two from
the following:
a)
b)
c)
d)
e)
f)
g)
Use wording like that of a past study involving similar procedures.
Offer parents the option to contact participants from past data collection efforts to alleviate their concerns.
Contact parents via phone to explain the importance of granting permission for their child’s participation.
Send a follow up letter to non-consenting parents explaining that the risks are minimal.
Mention the minor potential risks in the informed consent along with the benefits of the research.
Describe all possible risks in the informed consent form, no matter how trivial.
Send a handout along with the informed consent form addressing parents’ common questions and
concerns.
h) Since the risks of the study are trivial, deemphasize them in the informed consent form.
EDM Measure Validation
• Divergent validity evidence:
– Narcissism & cynicism (r = -.01 to -.29*)
– Exposure to unethical events (r = .02 to -.51*)
• Substantive validity evidence:
– Meta-cognitive reasoning strategies
(r = .12 to .52)
– Anti-social behaviors (r = -.07 to -.52)
EDM Measure Validation
• Criterion-related validity evidence:
–
–
–
–
Severity of violation (β = -.02 to .35*)
Perceived frequency of violation (β = -. 04 to .04)
Perceived importance of punishment (β = -.01 to .30*)
Severity of punishment recommended (β = -.01 to
.26*)
EDM Process Strategy Studies
• Purpose
1. Provide evidence for the validity of key EDM processes
2. Identify strategies underlying those processes
3. Identify interventions to promote strategy application
• 5 key process strategies
1.
2.
3.
4.
5.
Forecasting
Emotion regulation
Framing
Self-reflection
Ethical sensemaking
EDM Process Strategy Model
(Sensemaking)
RCR Principles &
Guidelines
Perceived
Causes of the
Situation
Professional
and Personal
Goals
Framing
Perceived
Requirements for
Attaining Goals
Emotions
Forecasting
Prior
Professional
Experience
Self-reflection
Sensemaking
Decision
Prior
Personal
Experience
EDM Process Strategy Studies
• Sample protocol (Self-Reflection)
1. Recall past or predict future experiences that are
either positive or negative
• Consider outcomes or the process of making ethical
decision
2. Complete EDM measure
• Sample protocol (emotion regulation)
1. Train on the use of ER strategies
2. Induce anger via feedback (anger vs. fear)
3. Complete EDM measure
EDM Process Strategy Studies
• Key Findings: Best Strategies
1. Forecasting
• Analysis of critical causes improved forecasting & EDM
• Forecasting critical consequences improved EDM
• Time pressure didn’t matter
2. Self Reflection
• Reflection on positive events and process increased EDM
• Using reflections to predict future ethical events
improved EDM
3. Emotion Regulation
• Anger inhibited EDM, moderate fear facilitated EDM
• Reappraisal buffered anger – was an effective technique
Case-Based Learning Studies
• Purpose:
– Cases are frequently used in ethics education, yet
little is known about development of cases
– Little is known about methods for case delivery
– Three rounds of studies:
1. Case content - finished
2. Process methods (strategies for delivering cases) – in
progress
3. Content & process interactions - 2013
Case-Based Learning Studies
• Case Content:
–
–
–
–
Character affect & power dynamics
Codes of conduct & forecasting
Critical causes & outcomes
Social context & goals
• Process methods:
–
–
–
–
–
Elaboration & self-generation of cases
Sequential case presentation vs. case comparison
Alternative scenarios & outcome evaluation
Incremental case building & forecasting
Note-taking & review
Case-Based Learning Studies Findings
• Content Studies:
– Character emotions enrich cases  increase in
knowledge acquisition & transfer of ethical principles
– Highlight codes of conduct  “
“
– Include causal information and potential outcomes
• Process Studies:
– Ask individuals to elaborate on cases
– Have individuals compare multiple cases
– Providing alternative case endings confuses readers
Alternative Methods of Evaluation
• Knowledge measures
– Declarative Ethical Knowledge Measure (DuBois and
Colleagues (2012)
– Knowledge of codes of conduct and regulations
– Average Cronbach’s alpha = .71
– Correlates highly with EDM measure (r = .40, p < .01)
Alternative Methods of Evaluation
• Trainee reactions
– Trainees in sensemaking training have reported high
levels of satisfaction with training activities and
content (M = 5 - 6 on a 7-point scale)
• Change in ethical conduct
– Distal evaluation
– Track ethical violations over time
A Cautionary Ending
• It is dangerous not to evaluate ethics training
programs
– We expect that ethics programs only help, but in
reality they can do more harm than good
• Success or failure (and the consequences)
grounded in:
– Effective models for understanding ethical behavior
– Multi-faceted, comprehensive program evaluation