Transcript Cost Effective use of Simulations in Online

Cost effective use of simulations in
online assessment
Ruth Thomas, Colin Milligan, Bill Austin
JeLSIM Partnership
Helen Ashton, Cliff Beevers, Dave Edwards
Heriot-Watt University
Simulations and assessment
Greater range of questions
Greater interactivity
Can test higher order skills
More authenticity
Feedback
Intrinsic
Cost effectiveness
Keep costs low.
Re-use pre-existing simulations
Multi-purpose new simulations
Avoid use of programmers
Keep effectiveness high.
Understanding of range of uses
Staff confidence and empowerment
Simulation developers should consider
assessment.
The aim of the presentation
Look at the technological perspective
Develop a framework for assessing the
potential of simulations in assessment
Practicalities of simulations and
assessment
Integration of simulation and AE
1. No integration: The simulation can be included in
a question as a media object, but there is no
communication between simulation and assessment
engine.
2. Low level of integration: The simulation can
pass a single variable, ‘answer’ to the assessment
engine. There is no facility to set up questions.
3. High levels of integration: There is two-way
communication between the AE and simulation to
set up and mark questions
PASS-IT AE-JeLSIM system
Educational use of simulations
Understanding a system
Developing thinking and problem
solving skills
Learning about processes and
procedures
Modelling
Assessments based on “understanding the system”
Activity
Description
Example
Possible Mechanisms
Relationships
between
variables
If variable X is
increased what
happens to variable Y?
If inflation increases, what
happens to unemployment
Multiple choice, Free text entry,
Cause and
effect
If condition A is
changed what will the
outcome be?
What will happen to
concentration over time if
the initial concentration is
changed? *
Multiple choice, Free text entry,
Sketch on a graph
Discover relationships
Establish the genetic laws of
inheritance for a virtual
animal
Free text entry
Predict result
Simulation variables  AE
What is the value of
the missing variable?
Examine the behaviour of
the sun to determine your
latitude. *
Multiple choice, Numeric input
AE, Numeric input sim.
Appropriate
ranges
Give typical values for
variable Z?
What is a typical level for
background radiation?
Multiple choice, Numeric input
AE, Numeric input sim.
Conditions and
phenomena
What combination of
variables leads to
phenomenon H?
Under what conditions will a
vehicle skid?
Free text
Score  AE
Simulation variables  to AE
Set the simulation to
condition J.
Set a system of vibrating
masses to resonance
Score  AE
Simulation variables  to AE
At what value of
variable X does
condition Y occur?
At what pH will a precipitate
begin to appear?
Multiple choice, Numeric input
AE, Numeric input sim
Assessment based on development of thinking skills
Activity
Examples
Specific
Example
Possible Answer
Mechanisms
Trouble
shooting/
problem
solving
Put the
simulation in a
state where it
functions in a
specified way.
Keep an
endangered
population from
extinction
Scored in simulation – score
passed to AE for marking.
Case-based
scenarios
Evaluate the
viability of a
simulated
scenario
Given a cash flow
and trading
forecast, comment
on the viability of a
business. *
Free text entry.
Locate the
broken
components in
a system.
Which resistors
are malfunctioning
in a circuit? *
Scored in simulation – score
passed to AE for marking
Fault
finding
Combination of simulation
variables passed to AE
Combination of simulation
variables passed to AE
Combination of simulation
variables passed to AE
Assessment based on use of procedures and
development of skills
Activity
Description
Examples
Possible
mechanisms
Following
procedure
Follow correct
protocols to
complete a task
Follow emergency
resuscitation procedure.
Record variables as
task proceeds.
Shutting down a nuclear
power station
Single variable out
(score)
Multiple variables out
(simulation state).
Testing a
skill
Demonstrate a skill
Landing a plane
Single variable out
(score)
Multiple variables out
(simulation state).
DC Circuit
Question type
Examples
Relationships
between
variables
If voltage increases what happens to current. What
effect does changing a resistor have?
Cause and effect
If the resistor is changes what will be the effect on a
graph of current vs. voltage for a specific resistor
Given a circuit with a number of resistors work out
the value of some of those resistors.
Conditions and
phenomena
What value of resistor X gives a voltage of Y across
it?
Trouble shooting/ If a component requires a certain voltage to operate,
given a set of resistors and voltage sources, which
problem solving
combinations allow the component to operate?
Fault finding
Determine which resistors are short or open circuit.
Following
procedure
Follow a procedure to determine the location of a
fault.
Bizstartup
Question type
Examples
Relationships between
variables
As overheads increase what happens to the break-even
point?
Cause and effect
If sales are 5% below target what will happen to the bank
balance
Some figures are missing from the trading forecast – work out
what they should be .
Appropriate ranges of
variables
What are typical costs for various overheads in the simulated
business?
Conditions and
phenomena
Adjust the selling price to achieve a specified breakeven
point.
Trouble shooting/ problem A business is projected to make a loss. What can be done to
bring it into profit?
solving
Case based scenario
Given a cash flow and trading forecast consider the viability
of the business and the validity of the assumption made
Fault finding
There is an error in the trading forecast – locate it
Following procedure
Given data for a start up company, generate a cash flow and
trading forecast
Practicalities
 Not all simulation-assessment systems can
cope with all question types
 Most can use many of less complex types
 Reuse old simulations if possible
 Take care in design of new simulations
 Consider other educational uses
 Maximise re-use from by:
 Pedagogic neutrality
 More complex model
Aiding question setters
New technique
Question setters need confidence
Examples and framework
Templates – mini question types
a JeLSIM model;
a template JeLSIM interface
a question set in PAE to demonstrate
communications
Contact and further info.
Ruth Thomas – [email protected]
Helen Ashton - [email protected]
JeLSIM website – www.jelsim.org
Exemplars - http://www.calm.ac.uk/simasses.html
Original applet is at: http://www.ida.liu.se/~her/npp/demo.html
Original applet is at: http://www.ida.liu.se/~her/npp/demo.html
Back