Transcript pku-2010-mcguffin-02..

Interaction: gestural input, popup
widgets, interaction techniques, and
hardware input devices
Prof. Michael McGuffin
Gestural Input
• Gestures can be executed rapidly, and can be used
as symbols to activate commands or select objects
• Sketches
– can be used to quickly enter text or diagrams, without
requiring the user to switch from the pointing device to
something else
– Have an informal, loose, implicit structure
– Can carry much more (implicit) information than typed text
• Gestures as characters to recognize
– Graffiti, Unistroke
Gestural Input
Kurtenbach and Buxton (video)
• Example gestures: circle
objects, drag to move, or
drag + draw a "C" to copy
• Such a gesture selects the
noun (objects) and verb
(operation) in a single
action, which may be more
natural, fluid, or faster than
doing selection + Ctrl-C/CtrlX + Ctrl-V
Gestural Input
"Teddy" (Igarashi et al. 1999).
See Java applet at Takeo
Igarashi’s website.
Phun (≈2008)
• A free game/physical simulator that uses gestural input
• http://www.acc.umu.se/~emilk/
Phun
• Alvarado and Davis 2001
• http://www.youtube.com/watch?v=NZNTgglPbUA
• LaViola 2007 http://doi.acm.org/10.1145/1281500.1281558
Observation
• Most user interfaces that use gestural input
require the user to first learn the gestures
before they can be used. This can be a
significant obstacle for novice users.
• In the following slides, we will see a technique
called Marking Menus that eliminates this
problem.
Modes, and some examples of
interaction techniques using gestural
input or other input
Interaction Techniques
• “are ways to use input devices to enter
information”
• "are made up of single input-device actions.”
(Foley, et al. “Computer Graphics: Principles and
Practice”)
Verbs
(actions,
commands,
tools,
operations)
Nouns
(objects,
locations)
Verbs in a
pulldown
menu
Nouns
(objects,
locations)
Nouns
(objects,
locations)
Verbs in
a popup
menu
Summary
• Modes can lead to mode errors, where the user thinks
they are in one mode but are really in another
• Visual feedback (example: changing cursor shape) to
indicate the current mode is good, but often not enough
to avoid mode errors
• Popup menus help…
– Avoid mode errors
(via transient modes, and kinaesthetic feedback where the user
feels pressure through their finger from holding down a button)
– Increase screen space devoted to content, because the menu is
only displayed when desired
– Decrease the distance travelled by the cursor, because there is
no back-and-forth motion between the workspace and
peripheral interface elements
– Create a tighter integration, or fusion, of the selection of noun
and verb, leading to better mental chunking (Buxton 1986)
Given that popup menus have all
these advantages, can we design
other widgets that are even better?
Radial Menu (or “Pie Menu”)
Demo: Radial Menu in SimplePaint
An interface that uses, effectively,
radial menus
Yatani et al., CHI 2008
Radial menus
versus linear menus
Radial menus
versus linear menus
Radial menus
versus linear menus
Radial menus
versus linear menus
Radial menus
versus linear menus
Radial menus
versus linear menus
• Directions are easier to remember
and reproduce than distances
Hierarchical Radial Menu
"Mouse Gestures" for Firefox
Marking Menu
• “Scale invariant recognition”: The recognition of the gestures
(marks) does not depend on the length of the segments; only
the angles matter. Hence, the marks can be drawn small and
quickly, or ballistically.
• A user who knows what mark to draw doesn’t even need to
see the menu popup.
The set of gestures is discoverable and “selfrevealing”, contrary to other gestural interfaces
Improved graphical presentation
Marking Menus
• Video/Demonstration
Marking Menus
• Video/Demonstration
Marking Menus
• Video/Demonstration
Transition from novice to expert
Traditional menus:
Pointing versus shortcuts
Marking Menus:
Transition is gradual
and natural !
Summary
Marking Menus
• Can be operated faster than linear menus
(ballistic, directional, scale-invariant marks)
• Have a self-revealing gesture set
• Enable a smooth transition
from novice use to expert use
• Can be used to select both a noun and verb
• Are limited to 8 items per submenu,
and limited to a depth of about 3 levels
Marking Menus
• Allow for faster selection than with linear
menus
(directional, ballistic movements)
• Can be used without looking at the screen
(“eyes-free operation”)
• Have a discoverable set of gestures
• Allow for a gradual and natural transition
from novice to expert
• Can be used to select a noun and verb
• Are limited to approximately 8 items per
submenu, and a depth of about 3 levels
Summary
•
•
Temporary modes, held by holding down a button or key with kinesthetic
feedback …
– Help avoid mode errors
– Example: a key that must be held down to maintain a temporary mode
change, such as the Shift key on keyboards
– Example: popup widgets
• Help combine noun+verb selection
• Only require screen space when in use
One popup widget with many advantages: Marking Menus
– Allow for ballistic gestures
– Allows users to gradually transition from novice to expert
Other examples
of modes, tools, etc.
• video: "Selection and Positioning tasks",
Bill Buxton 1983
Some observations
•
Techniques shown in the video:
–
–
–
–
•
1. Drag-and-drop
2. Modal toolbar
3. « Moving menu » (a kind of popup menu)
4. « Moving menu » with memory
Questions: of the 4 techniques, which ones …
– Are not modal (or only have a temporary mode with kinesthetic feedback), and
so help to avoid mode errors ? (Answer: 1 and 3)
– Avoid back-and-forth movements between the workspace and the toolbar, and
do not have screenspace consumed by a toolbar ? (Answer: 3 and 4)
– Have visible affordances (toolbar), therefore showing the possible operations to a
novice user ? (Answer: 1 and 2)
– Allow the same shape to be created many times repeatedly, very quickly ?
(Answer: 2 and 4)
– Ensure that each shape is always selected with the same dragging gesture,
allowing the user to learn the gestures by heart and then execute them rapidly ?
(Answer: 1 and especially 3.)
Can we extend popup menus/widgets
for other uses?
Other popup menus and widgets
• Hotbox (Kurtenbach et al., 1999),
Control Menus (Pook et al., 2000),
Flow Menus (Guimbretière et Winograd, 2000),
FaST Sliders (McGuffin et al., 2002),
Tracking Menus (Fitzmaurice et al., 2003),
Trailing Widget (Forlines et al., 2006)
Hover Widgets (Grossman et al., 2006),
PieCursor (Fitzmaurice et al., 2008)
• These widgets and interaction techniques are designed for:
–
–
–
–
A large number of commands
Controlling continuous variables
Input of text and numbers with gestures
Use with a stylus (pen), for example, on a “tablet PC”
FlowMenus (Guimbretière et al., 2000)
2D manipulation with FlowMenus
(Guimbretière et al., 2000)
The Hotbox in Maya: a 2D menu
[Kurtenbach et al., 1999]
The Hotbox in Maya: a 2D menu
[Kurtenbach et al., 1999]
The Hotbox in Maya: a 2D menu
[Kurtenbach et al., 1999]
Demo of a more recent kind of Hotbox
(McGuffin and Jurisica 2009) used for
manipulating a network visualization
Tracking Menu (copy of video at
http://www.dgp.toronto.edu/~gf/videos/tm.avi
at George Fitzmaurice’s website)
Hover Widgets (copy of video at
http://www.dgp.toronto.edu/~ravin/videos/chi2006_hoverwidgets.mov
at Ravin Balakrishnan’s website
)
Toolglass: bimanual input
Click-through: Simultaneous
selection of verb and noun!
Bimanual 3D Navigation
Erick Velazquez-Godinez and Michael McGuffin (unpublished, 2010)
http://www.youtube.com/watch?v=ggoLLnxHwhA
Having the 3D position of each hand is enough to control the 6
degrees-of-freedom of a 3D camera: translate hands together to
translate the camera, rotate hands around each other to rotate the
camera, move hands toward or away from each other to zoom.
Tracking 2 or more fingers with a
Nintendo Wiimote
• See online videos by Johnny Chung Lee
http://johnnylee.net/projects/wii/
Other uses of two hands?
Two-Handed (Bimanual) Input
Potential uses:
• Dominant hand (DH) on mouse,
non-dominant hand (NDH) on keyboard
• Two mice, two cursors, symmetrical:
– Rapid clicking by alternating between hands?
– Simultaneous rotation+scaling+positioning in 2D
or rotation+zooming+panning in 2D
• Two mice, asymmetrical:
– NDH for camera, DH for selection/manipulation
– NDH for tool palette, DH for clicking-through
(Toolglass)
Kinematic Chain Model
(Yves Guiard 1987)
• NDH (non-dominant hand) performs
slower and less precise movements than
the DH (dominant hand)
• The NDH moves before the DH
• The NDH establishes a frame of reference
within which the DH performs work
Input Devices
Keyboards
http://world.std.com/~jdostale/kbd/SpaceCadet1.jpeg
Foldable keyboard for Palm Pilot
Flexible keyboard; 24 $ at
http://www.thinkgeek.com/computing/input/5a7f/?cpg=ab
(price from 2008)
Projected Keyboard
http://en.wikipedia.org/wiki/Projection_keyboard
160$ at http://www.thinkgeek.com/computing/input/8193/
(price of 2008)
Optimus
http://computer.howstuffworks.com/keyboard.htm/printable
Optimus Maximus
Each of the 113 keys contain a 48x48 pixel screen.
1864 $ US ! (price of 2008)
http://www.artlebedev.com/everything/optimus/
Optimus Maximus
Configured for English
Optimus Maximus
Configured for Russian
Optimus Maximus
Configured for the game "Half-life"
Optimus Maximus
Configured for "Photoshop"
Bottons on a tape recorder
The status quo:
Uniform buttons
Buttons with different forms: allow use without looking
(“eyes-free operation”)
Piano Keyboard
modern design
naïve design
Apple iPhone Keyboard
Scholes Keyboard versus Dvorak Keyboard
Designed in 1873 !
Still the standard :-(
Still not in widespread use
Notice that all vowels are under one hand in
the midlde row.
Comparison of QWERTY and Dvorak
http://infohost.nmt.edu/~shipman/ergo/parkinson.html
“Half Keyboard” from Matias Corp.
http://www.halfkeyboard.com/
« Septambic keyer »
or « chorded keyboard »
4 x 24 – 1 = 63 possible "chords"
Pointing Devices
The first mouse
•
•
•
•
1968
Douglas Engelbart
Stanford Research Institute
Two rollers for x and y
40+ years later
Rotation sensing
Rockin’ Mouse
Haptic feedback
(or “force feedback”)
Touchpad
Other kinds of 2D pointing devices
• Digitizing tablet / graphics tablet with a stylus (pen)
and/or puck (mouse)
• Touchscreen
• Light pen
• Touchpad
• Joystick
– isometric (rigid, senses pressure, for example: Trackpoint)
– elastic (like isotonic, but returns to center when released)
– isotonic (can be moved freely)
• Trackball
• Eye tracker
TrackPoint (isometric joystick)
Touchpad
Logitech Trackball
Graphics tablets
Wacom Bamboo
5.8x3.7 inches; 80$
Wacom Intuos3
12x19 inches; 750$
(prices of 2008)
Wacom Cintiq 21UX
with integrated screen
21.3 inches (17x12.75); 2500$
Input devices for tablets
Styluses can have
• a pressure sensor on their tip
• a button on the tip to click
• a button on the side ("barrel button")
• a scroll wheel that can be rotated
IntuPaint (Vandoren et al. 2008)
and FluidPaint (Vandoren et al. 2009)
Properties of Pointing Devices
• Absolute vs relative sensing
– Which is more general ?
• Direct vs indirect
–
–
–
–
–
Which is more “intuitive” ?
Which is more predictable ?
Which is more relaxing ?
Which leaves visual feedback more visible ?
Which do artists prefer ?
• Discrete vs continuous
– Which is more general ?
• Position control vs velocity control
Velocity control:
– Which is more general ?
– Which typically requires less movement/smaller footprint ?
Velocity control pointing devices, for 3D
Spaceball
Spacemouse
Magellan
SpaceNavigator (60$)
3dconnexion.com
(in 2008)
Spaceball 5000
Other products from
3dconnexion.com
(in 2008)
Taxonomy of input devices (Buxton)
M: Mechanical intermediary
T: Touch
3-State Model (Buxton, 1990)
State 0: no (x,y) coordinates
States 1 and 2: (x,y) position is known
Examples:
• Graphics tablet: states 0, 1, 2
• Mouse: states 1, 2
• Touchscreen: states 0, 1
TouchMouse
(Hinckley and Sinclair 1999)
States 0, 1, 2
TouchMouse
(Hinckley and Sinclair 1999)
“Pop Through” Buttons
(Zeleznik et al. 2001, http://doi.acm.org/10.1145/502348.502384 )
PreSence (Rekimoto et al. 2003)
Haptic Pen
Lee et al., "Haptic pen: a tactile
feedback stylus for touch screens",
UIST 2004
Handheld Projector and Pen
Cao et Balakrishnan,
"Interacting with Dynamically
Defined Information Spaces
using a Handheld Projector
and a Pen", UIST 2006
PenLight
Song et al., "PenLight: combining a mobile projector
and a digital pen for dynamic visual overlay", CHI 2009