Transcript Introduction to Assessment - University of Nevada, Reno

Introduction to
Assessment
Jennifer Lowman
Coordinator, Student Persistence Research
Credit for the content of this presentation should go to Marilee Bresciani and her many co-authors;
& Gavin Henning & others with Academic Impressions
Outline
•
•
•
•
Define Assessment
Identify different Types of Assessment
Review Cycle of Assessment
Define Outcomes
– Writing “Learning Outcomes”
• Practical Assessment Techniques
• Ethics
• Concluding Remarks
What is Assessment?
• Assessment, Evaluation, & Research
• Assessment
– An effort to gather, analyze, and interpret evidence to
understand effectiveness and improve programming
• Evaluation
– Using assessment information to make an informed
judgment of program quality or worth
• Research
– Collection and analysis of data to build or confirm
theory or conceptual foundations
Institutional Assessment
•
•
•
•
•
•
Mission
Goals or Objectives
Program, Product, Service Inputs
Outcomes
Improvement or Accountability
Assessment Planning part of Programming
– Embed Assessment
• Design, Measurement, & Analysis
• Reporting & Learning
Embedded Assessment
• Make assessment part of your program, event,
service, or activity
• Decades of research shows that students
respond to intentional activities that are
linked to positive outcomes (Bresciani, Zelna,
& Anderson, 2004).
– If you tell students what you expect them to learn
and how it will benefit them, they are more likely
to learn
Embedded Assessment is…
• Continual assessment for continual
improvement
• Track changes in program impact over time, as
well as…
– Improve measurement over time
– Test new measures, while still using old
• Make it a habit – a state of mind!
Culture of Assessment
• Build shared trust: Begin by lowering social and
interpersonal barriers to change.
• Build shared motivation: Collectively identify
goals worth working toward and problems worth
solving.
• Build a shared language: Develop a collective
understanding of new concepts needed for
transformation.
• Build shared guidelines: Develop a short list of
research-based guidelines for using assessment
to promote learning
Types of Assessment
Types of Assessment
Tracking Usage
Needs Assessment
Satisfaction Study
Culture/Climate Assessment
Outcomes Assessment
Resource Effectiveness Study
Benchmarking
Program Review
Strategic Planning
Forms of Assessment
Operational or Descriptive Assessment
• Tracking Usage
• Needs Assessment
• Satisfaction Study
• Culture/Climate Assessment
Formative Assessment
• Outcomes Assessment
• Resource Effectiveness Study
• Benchmarking
Political Assessment
• Program Review
• Strategic Planning
Types of Assessment?
(activity)
Cycle of Assessment
Cycle of Assessment: External Perspective
1.
2.
3.
4.
5.
6.
7.
8.
9.
Define issue/problem
Identify Goals
Align mission & goals
Identify stakeholders
Identify
theoretical/conceptual
framework
Develop measurable
outcomes from goals
Identify and measure
inputs
Classify strategies from
framework to reach
goals
Isolate action steps to
implement strategies
10. Develop formative
assessment plan to
understand process
11. Measure Outcomes
12. Make sense of results
13. Report / Present
findings
14. Review, Reallocate,
Advocate for Resources
15. Retool Program
16. Plan for Iteration #2
Cycle of Assessment: Internal Perspective
Identify
Outcomes
Improve
Program
Mission
Purpose
Gather,
Interpret
Data
Create
Opportunity
Outcomes-Based Assessment
Assessment & Planning
• Assessment and Planning are linked,
intertwined, and interrelated
• “Backwards Design” or “Start with the end”
• Programs are designed this way too!
Program
Inputs
Desired
Outcome
From Goals to Outcomes
• Goals/Objectives – end result in broad terms
(optimal outcomes)
• Outcomes – more specific statements derived
from goals that are measurable
– Operational Outcomes
– Learning Outcomes
– Program Outcomes
Learning v. Program Outcomes
• Learning Outcomes (aka Individual Outcomes) –
a learning outcome is participant-centered and
describes the desired effect of a program on a
student’s cognitive or intellectual state.
• Statements of what students will value, do, or
know as a result of participation.
– ASK: attitude, skills, & knowledge
• Program Outcomes (aka Aggregate Outcomes) –
the desired aggregate effect of a program.
Program or Learning Outcome?
(worksheet)
Learning Outcomes
• Participant - centered
– Focus on what students can do
– Not on what you did!
• Purpose to reveal the impact of ‘Action Steps’
– Action Steps are the point of contact
– May or may not be perceived by students
– Outcomes may need to take variation in contact into
account
– Variation is key for identifying effectiveness!
Learning Outcomes Assessment
• What are students learning and why?
• How do we know?
• Are students learning what we intend for
them to learn?
• How do we use assessment information to
improve learning?
• Do the improvements we make work?
SMART Outcomes
• SMART  Specific
Measurable
Aggressive/Attainable
Results-oriented
Time-bound
Writing Learning Outcomes
• ABCD formula
– Audience (Who)
• Who is the target?
– Behavior (What)
• What should the target be able to know, do, value (ASK)?
– Condition (How)
• What Action Steps will facilitate the learning?
– Degree (How much)
• How much learning will be demonstrated?
Writing Learning Outcomes
Condition + Audience + Behavior + Degree
As a result of participating in the event +
students +
will demonstrate +
3 of 5 skills.
ABC v. ABCD…
or why Degree is important!
• The degree of impact you expect to have on
your students is important to…
– Set criterion for success AHEAD of program
implementation
– Demonstrate Minimal Benefit
• Transparency & Accountability
– Signal intention to students and stakeholders
– Establish standard for data-driven decision-making
Degree of Impact
• Independent Indicators (Index)
– Inter-changeable, equally important information
– Differences between students at start/finish are not import.
• Ex. 3 of 5 study habits
• Ex. 5 of 7 coping strategies
• Continuous Scale
– Cumulative or increasing attitude, skill, or knowledge
– Expect differences between students at start/finish
• Increased leadership skills from baseline eval to next level (basic to
intermed.; intermed. to adv.)
• Process
– Student must master step 1 to move onto step 2…etc.
– Describe steps in process within LO
– Break process in to multiple LOs, but avoid yes/no outcomes
Degree of Impact
≠
Depth of Learning
Degree of Impact & Depth of Learning
• Influenced by the opportunity provided
– Exposure
– Relevance
– Immediacy
– Salience
– Feedback
–…
• L.O. reflects depth of learning in the “Action
Verb”
Depth of Learning
Bloom’s Taxonomy (revised)
• Remembering: recognizing, recalling, labeling
• Understanding: interpreting, summarizing,
contrasting
• Applying: executing, implementing, classifying
• Analyzing: differentiating, organizing, attributing
• Evaluating: critiquing, integrating, generalizing
• Creating: generating, planning, producing
(teaching)
List five ways identified in the readings that can help you reduce stress?
(Remembering)
Based on the floor meeting, discuss at least three reasons why we don’t allow
alcohol in the residence halls?
(Understanding)
How can you use what you learned in these activities in your student
organization?
(Application)
After participating in the retreat, compare the effectiveness of leadership
tactics?
(Analyzing)
As you reflect on this past year as I have served as your organization’s advisor,
in what areas have I been most effective and in what areas can I improve
(Evaluating)
Based on our conversation regarding career action steps, recommend four
action steps to each of your group members?
(Creating)
The Depth of Learning expected is evident in the Action Verb!
Action Verbs (Observable Behaviors)
Lower-Level
List, Label, Name
Define, Describe
Identify, Interpret
Contrast, Discuss
Classify, Complete
Relate, Modify
Demonstrate
• Don’t Use
Solve, Apply
– Appreciate
Analyze, Explain
– Become
Infer, Integrate
aware
Plan, Create
– Be familiar
Design, Generalize
with
Assess, Rank
– Know
Recommend
– Learn
Compare, Teach
– Understand
Higher-Level
Practice Writing Learning
Outcomes
(Activity)