Transcript SheffieldTalk.pps - School of Computing Science

Probabilistic User interfaces
Roderick Murray-Smith
Department of Computing Science,
University of Glasgow &
Hamilton Institute, NUI Maynooth
[email protected]
http://www.dcs.gla.ac.uk/~rod
With John Williamson, Parisa Eslambolchilar, Andy Crossan, Steve Strachan,
Vuokko Lantz & Stephen Brewster
Overview
1.
2.
3.
Dynamics & Statistics in
Interaction
Uncertain, Dynamic
Feedback mechanisms
Demos
1.
2.
4.
Hex Entry: Intelligent
adaptation of handling
qualities during interaction
Pointing without a pointer:
Control models in interaction
design
Conclusions
Dynamics & Statistics in HCI?
• Why introduce dynamics – is that not harder?
– We can only control what we can perceive.
– Dependent on feedback, so upper limits on the speed of
change of display.
– Dynamics allows us to slip in ‘intelligence’ which couldn’t be
done with a static interaction technique
• Why uncertain interaction?
– Uncertainty in user’s mind about what to do next, and
system uncertain about user’s intentions.
– With mobile devices, interaction with the user is now
continuous instead of discrete, and input devices are noisier.
Feedback & Inference
• A model of user interaction in
a closed-loop system involving
uncertainty.
• Feedback of the results of the
inference process is provided
to the users which they can
then compare with their goals.
• Inference may often be about
user’s beliefs, desires or
intentions…
Display and control
The display is to provide the user with information
needed to exercise control. i.e. predict consequences of
control alternatives, evaluate status and plan control
actions.
• Display augmentation
– Improve input to human to simplify control task
• Control augmentation
– Change the effective dynamics between control
input and system output
Ambiguous displays
•
•
•
•
Used in psychophysics experiments (e.g Körding & Wolpert 2004)
Transfer idea to user interface design. If the system is uncertain about
inputs or user intentions, present data in an appropriately ambiguous
fashion.
Does it regularise user behaviour & improve usability appropriately?
Pattern recognition and displays are interdependent and should be
developed together
Mobile & Acceleration Sensing
• HP IPAQ
• Xsens accelerometer
– 3 DOF linear accelerometer
– Samples up to ~100 Hz
– Weight ~10.35g
• Potential for one handed /
screen free interaction
• Mobile devices used in many
contexts, subject to varying
levels of disturbance
– Ideal testbed for
probabilistic interaction
– Small screen
– Vibrotactile/audio feedback
Audio displays - Granular Synthesis
• A structured approach
to probabilistic audio
• Quantum theory of
sound
• Accumulate short grains
from waveforms
sources
• Select grains according
to some probability
distribution
Sonification of Probability
Distributions
• Straightforward sonification of probabilistic models
– Associate distributions with collections of source waveforms
– Continuous distributions can be sonified via sampling from a
parametric synthesis algorithm
• Produces a smooth texture representing the
changing probabilities
Feedback for gesture recognition
•
•
•
•
Mapping from an input trajectory to an audio display via a number of gesture recognition
models.
Each gesture is associated with a model and the output probabilities are fed to the synthesis
algorithm.
Can be combined with direct sonification of gesture movements.
Users can explore functionality
–
–
Feedback from the goals, depends on accuracy of gesture & estimated skill of user
User behaviour can be ‘shaped’, starting with simple, blurred gestures and progressing to sharper,
more complex expression.
Outline of the gesture recognition and sonification system
Acceleration
measurements
of the phone’s
movements
Parametric model of
gestures (Dynamical
motor primitives,
Locally Weighted
Learning)
Model parameters fitted to
the current observations
Sonification of the
performed gesture
(Granular
Synthesis)
Audio feedback
generation
Audio source 1
Audio source 2
Audio source 3
Sonification of the recognition
result and its confidence
(Granular Synthesis)
Audio feedback
generation
A posteriori
probabilities for
different
gestures
…
Audio source N
Gesture
recognition engine
Feedback for gesture recognition
• Benefits of feedback
– Feedback on how the gesture recognition engine is performing, i.e.
recognition result and confidence
– Gives the user insight into the pattern recognition mechanics.
– Feedback on how the user is performing, i.e. sonification of the
users actions
• Coupling of gesture recognition and feedback generation
– Simple, parametric representation for the observed gesture data
– Gesture model parameters can act as pattern features of the
recognition engine
– Recognition engine produces a posteriori probabilities for each
gesture class
– Parametrized feedback generation on the basis of gesture feature
vectors or classification results, e.g. Granular Synthesis of
audio/vibro sources
Haptic Targeting
• Spatially & Time
Varying Vector Field
• Directional Grains
using Vector
Summation
• Highlights Areas of
High Uncertainty
Quickening/Predictive displays
•
Augmentation of a display with predictive information
–
“Experience indicates that, by using a properly designed predictor instrument, a novice can in 10 minutes
or less learn to operate a complex and difficult control system as well as or better than even the most
highly skilled operator using standard indicators”,
from Kelley, C.R. “Manual and Automatic Control” 1968
•
Standard technique in manual control systems
–
•
e.g quickening of helicopter displays, Showing derivatives of current state
Quicken the probabilistic audio display
–
Add predictions of change of probability to the display, e.g. if derivative of probability is increasing,
decrease if derivative is decreasing…
n
v  p   ki
i 1
–
•
dp i
d it
Allows users to determine when they are moving towards regions of high probability; aids in targeting of
modes
Models such as Gaussian processes allow derivative uncertainty to be included.
Demo: Nonlinear dynamics &
Monte-Carlo simulations
Feedback conclusions
• Provided examples of granular synthesis for
sonifying probabilistic interfaces
– with quickening, & Monte Carlo predictions can help
improve interfaces to a continuously controlled
environment which involves uncertainty
– can be extended to force- and vibrotactile feedback
– Helps users learn gestures for mobile devices
– Allows flexibility to give feedback about different
orders of derivatives, applications in rehabilitation
engineering.
Hex Entry: Intelligent adaptation of
handling qualities during interaction
• Flexibility brought by dynamic models allows intelligent
interaction,
– handling qualities of the dynamics of the interface are adapted
depending on current inferred user goals.
– actions require less effort, equivalent to a lower bit rate in
communication terms, the more likely the system’s
interpretations of user intentions.
• Serves as example of continuous interaction system,
– with gestures, augmented control and potential for audio
feedback
• Predictions of future trajectories
– Could be linked to sound as MC samples
• Quickening via velocity & acceleration info
Hex: The Aims
• A continuous input system –
all entry is one single smooth
sequence
• Incorporating a probabilistic
model to represent uncertainty
and increase performance
• But with a structure that can be
learned – gestures must be
repeatable
– Support transition from novice
(tightly closed control loop) to
expert (open-loop, learned
behaviour)
Current
most
probable word
Future path
Current
entry
Cursor
Example Words
“Hello”
“GIST”
“Hexago
ns”
Augmented control
• System provides augmented, nonlinear control
– Don’t perform actions for user - help user reproduce ideal
behaviour themselves
– Adapts to context, changing properties of the control system
– Minimises effect of disturbances and errors
• Goal is that although initial use is very dependent on
feedback, user learns open-loop gesture-like
behaviour.
Nonlinear dynamics
Current most
probable word
Future path
Current
entry
Cursor
• Vector field adapts to
current context
– In this case ‘Q’ has been
chosen
• Handling qualities
improved appropriately
– Makes it easy to get to ‘U’
Semantic Pointing (Blanch, Guiard, Beaudouin-Lafon 2004.)
• Motor space and
Display space have
different properties
• Control-Display ratio
adapted depending on
proximity of target
Predictive control & Word
Autocomplete
• Also show top k most probable paths
– fitting a cubic spline through hexagon centres
Hex Conclusions: From control to
gestures
• Progress from feedback control to open-loop
gesturing
– With audio/tactile feedback, can be less reliant on screen
– Progression to higher-order control
– Provides new users with a way to gradually learn system.
• Gestures can be used for simple common tasks
(Autocomplete, delete etc)
• Dynamic representation allows ‘intelligent systems’
make life easier for the user.
• Current system would need a lot of development
before being a natural text-entry system (only ca. 17
words per minute)
The Selection Problem
• How can we determine user intention?
• Evaluate probability distribution over potential goals
• Closed-loop interaction
– Goals are negotiated with the system in continuous time
– Continuous feedback on user state with respect to goals
• Selection for devices for which pointing is non-intuitive
Perceptual Control Theory
System
Goal
Control loop
Feedback
• How can we determine intention?
• Perceptual Control Theory – Powers, et al.
– Fundamental hypothesis: Humans act to control their perceptions
– Test for this control behaviour
• Hypothesis: Identify intentions via correlations between
input and known disturbance patterns
An Agent Perspective
• Reformulate selection in terms of agents
– Each goal or item is considered an independent agent
• Agents probe user
– “experiment”, look for response
– Evaluate probabilities p(selectedi)
– Akin to MCMC Sampling from users mind!
Demos
Example: Movement
Test: Compare distribution of histories over some time window
Agent Disturbance
User Control
 e 
Hence, we have: pi  f  
a 
Result
If controlled,
If no control,
e
1
a
e
1
a
Interpretations
• Can be seen as control, damping, imitation,
gesture recognition or excitation
– Control/damping
– Imitation of the motion of the disturbance
(pursuit task)
– Gesture recognition with dynamically
created gestures
– User excites “modes” of the object,
inducing a meaningful disturbance in the
object
Feedback
• Real-time feedback on potential goals
– Visual example in demo
– Mapping entropy to audio dissonance
Log P(i)
Entropy
Sampling from user’s mind
• E.g. Bubbles (Schyns et al
2003)
• Adapt idea for use with
fisheye interfaces and
continuous interaction
instead of discrete
accept/reject.
PCT Conclusions
• Probabilistic selection method suitable for nonconventional sensing and feedback systems,
supporting real-time feedback on progress towards
potential goals, incorporating models from manual
control theory to optimize performance
• Much scope for extending and applying these general
ideas to practical interfaces
– More sophisticated user models
– Disturbance/experiment design
– Feedback design…
Outlook
• Dynamics allow intelligence to be sandwiched into an
interface
– ‘look-and-feel’ of an interface, ‘noisy channel’, or in control
terms, the ‘adaptive handling qualities’?
• Augment the display or the control?
• Adapt ambiguity in display to context
– E.g. walking, in bus, at desk
• Exciting overlap between:
–
–
–
–
human motor control,
Statistics
manual control/dynamic systems
human-computer interaction
Other fun things…