Transcript satin-guir-jul1999

SATIN:
S ketch And T ransformational
IN frastructure
Project Status Report - 07/22/1999
Jason Hong
UC Berkeley
Group for User Interface Research
Motivation
• More and more apps are using pens and
some form of sketching
•
•
•
•
•
•
Pilot
• XLibris
Elec. Cocktail Napkin • NotePals
Dynomite
• Flatland
Pegasus
• SILK
FiloChat
• Tivoli
Zenpad
• Kramer's Patches
• These apps are usually
– built from scratch
– not extensible or reusable
Motivation
• Architecture of windowing systems
optimized for keyboard and mouse
– Need to explore principled architectures
for alternative input
– Current state is analogous to windowing
systems in the early 1980s
High-Level Research Goals
• Create extensible toolkit supporting
creation of pen and sketch-based apps
• Explore architectures for pen-based
applications
• Explore new interaction techniques for
using pens and for sketching
• Explore new event models for other
input devices
Overview
•
•
•
•
•
•
Motivation
Envisioned Applications
Features and Current Status
High-Level Architecture
Future Development
Feedback
Envisioned Applications
• SILK
• DENIM
Envisioned Applications
• PosterBoard
– Create posters using
DigitalDesk
Envisioned Applications
• PatchBoard
– Multi-device whiteboard
How SATIN Fits Into the Big Picture
The Required Berkeley CS Diagram
Application
Users may already have
these parts or may have to
download dynamically
SATIN
Swing
Core Classes
All versions of Java
have these parts
Java Virtual Machine
What SATIN Will Provide
• Scenegraph and rendering support
– GraphicalObjects
 Objects with state, behavior, and view
 State and behavior defined thru interpreters
 Multiple views
Find
– Transformations
 Per GraphicalObject, Rotate / Scale / Translate
– Styles
 Translucency, fill colors, line width, etc
 Just set it - automatically used!
 Can define styles in text files
 Cascading Styles
• e.g. selected objects or greyed-out objects
What SATIN Will Provide
• Scenegraph and rendering support
Add, remove, group, move, etc
Pick GraphicalObject at location
Layering of objects
Automatic animation for smooth transitions
Semantic zoom
What SATIN Will Provide
• Patches
Interpreted regions of space to help
organize, process, and manipulate sketches
•Step A
•Step B
•Step C
What SATIN Will Provide
• Built-in support for Undo/Redo/Macros
Command pattern scheme
Automatic time-indexing of operations
Multiple streams and histories of events
• Integration with Java Swing
Cut / copy / paste
Standardized Drag and Drop
Export as GIF / XML
Allow use of Java widgets
Be used as a Java widget
Current Status
• Recognizers
Rubine's Recognizer
• Interpreter
Graffiti Interpreter
• Pie Menus
• Performance
Minimal damage-redraw
• Needs some more optimization though
Current Status
• Basic Debugging Support
Debugging Window
Graphical Bounds
• Properties file
System initializations
• Sheet colors, location of data files, etc
• Documentation
What you are reading
LOTS of Javadoc
Overview
•
•
•
•
•
•
Motivation
Envisioned Applications
Features and Current Status
Future Development
Feedback
High-Level Architecture
Short-term Future Plans
• More Patch Operations
– pull up, push down, dissolve, auto-arrange,
• Multiple Views
• Advanced Dispatching
– "tolerant" dispatching
– top-to-bottom dispatching
• Streams and histories
– Object histories or system histories?
• Activators
– If "is matrix" then load matrix interpreter
Feedback?
• What do you need to make this work
for you?
• What do you like? What do you not like?
Overview
•
•
•
•
•
•
Motivation
Envisioned Applications
Features and Current Status
Future Development
Feedback
High-Level Architecture
GUIR Object Notation
Vehicle
Each instance of
Car has four
instances of Wheel
The up arrow points
to the parent class
Semi
This color means
classes already
provided by Java
JComponent
Car
1
4
Wheel
Pinto
1
n
Gadget
Each instance of Pinto has n instances
of Gadget. Each Gadget has a reference
back to the Pinto it belongs to.
Simplified Overview of Class Architecture
Interpreter
Interpreter
Interpreters
Timed
Stroke
Watchable
Watcher
GraphicalObject
(GOb)
GraphicalObject
Collection
GraphicalObject
Group
JComponent
Patch
Sheet
Runtime Architecture
Dispatching
Java Event
Subsystem
Stroke
Assembler
Sheet
Patch
GraphicalObject
Group
GraphicalObject
(GOb)
Interpreter
Interpreter
Interpreter
Command
Objects
Commands
Command
Queue
(Undo, Redo, Macros)
Java2D Drawing
Rendering
Subsystem
Clipboard
Selected
GObs
GraphicalObjects in Depth
Cloneable
Transform
Serializable
Style
Shape
GraphicalObject
(GOb)
Bounding Points
(Polygon)
Watchable
SingleStroke
Listener
Progressive
Stroke
Listener
Interpreter
Mediator
GraphicalObject
Group
1
n
Interpreter
Interpreter
Interpreters
The Lifecycle of a Stroke (Current)
Mouse Events get forwarded here
until the full stroke is completed. The
Stroke Assembler then dispatches to
the Sheet.
Sheet
1 onSingleStroke()
Stroke
Assembler
2 onSingleStroke() GraphicalObject
Group
Sheet determines if
the stroke should be
redispatched to
another
GraphicalObject
Patch
3 onSingleStroke()
4
GraphicalObject
(GOb)
handleSingleStroke()
The deepest
object tries to
handle the stroke
first.
Interpreter
Mediator
Interpreter
Interpreter
Interpreter
The Lifecycle of a Stroke (Current)
Stroke
Assembler
Sheet
GraphicalObjectGroup GraphicalObjectGroup Graphical
/ Patch
/ Patch
Object
mouseDragged()
onSingleStroke()
A
onSingleStroke()
onSingleStroke()
A
A
handleSingleStroke()
isHandled?
B
B
onSingleStroke()
isHandled?
isHandled?
A
dispatchee = getDispatchee()
if (dispatchee != null) {
isHandled = dispatchee.handleSingleStroke();
}
if (!isHandled) {
isHandled = handleSingleStroke();
}
return (isHandled)
B
if (!isHandled) {
isHandled = handleSingleStroke();
}
return (isHandled)
The Lifecycle of a Stroke (Future)
Sheet
Stroke
Assembler
GraphicalObjectGroup GraphicalObjectGroup
/ Patch
/ Patch
Graphical
Object
mouseDragged()
dispatch()
process ()
dispatch ()
process ()
dispatch()
process()
dispatch()
process()
handle()
handle()
handle()
handle()
isHandled?
isHandled?
isHandled?
Every GraphicalObject has nine methods:
dispatchNewStroke()
processNewStroke()
handleNewStroke()
dispatchUpdateStroke()
processUpdateStroke()
handleUpdateStroke()
dispatchSingleStroke()
processSingleStroke()
handleSingleStroke()
GraphicalObject Coordinate Spaces
• Every GraphicalObject has three
coordinate spaces
The top-left corner of
this stroke is at (246,335) in
absolute or screen coordinates.
The top-left corner of
this stroke is at (91,59) in
relative or parent's coordinates
The top-left corner of
this stroke is at (0,0) in
local or self coordinates
In local coordinates, every
GraphicalObject will have it's
top-left corner at (0,0)
GraphicalObject Coordinate Spaces
Here, we have two screen shots.
The one on the left has no transform.
The one on the right has a rotating
transform on the Sheet.
In absolute coordinates,
the top-left corner of the
Sheet is always (0,0).
This is (0,0) for the Patch
This is NOT (0,0) for
the Patch. The
transform was applied
to the Sheet, changing
absolute coordinates,
but not anyone's local
coordinates.
If you turn Graphical Debugging on,
the coordinates displayed are absolute
GraphicalObject Coordinate Spaces
• How it works internally…
Sheet
Absolute Transform =
[txA][txB][txC]
Transform
(txA)
Patch
Relative Transform =
[txC]
Local Transform =
[1]
Transform
(txB)
GraphicalObject
Transform
(txC)
Absolute coordinate methods:
Relative coordinate methods:
Local coordinate methods:
getAbsoluteTransform()
getAbsolutePosition()
getAbsoluteBounds()
getAbsoluteBoundingPoints()
getAbsoluteWidth()
getAbsoluteHeight()
setAbsoluteBoundingPoints()
getTransform()
getPosition()
getBounds()
getBoundingPoints()
getWidth()
getHeight()
setBoundingPoints()
getLocalTransform()
getLocalPosition()
getLocalBounds()
getLocalBoundingPoints()
getLocalWidth()
getLocalHeight()
setLocalBoundingPoints()
Recognizers and Interpreters in Depth
Stroke
Recognizer
Recognizers
Classification
Progressive
Recognizer
Single-Stroke
Recognizer
Rubine
Recognizer
Neural Net
Recognizer
Interpreter
Recognizer
GraphicalObject
(GOb)
Stroke
Classification
Handled?
Interpreters
Progressive
Interpreter
Single-Stroke
Interpreter
App-Specific
Code
Internal Data Structures
Interactor
Tree
Probabilistic
Tree
Internal Data Structure Helpers
Graffiti
Interpreter
Rectilinear
Interpreter
Ink Clustering
Algorithms
(Spatial, Chronological)
Recognition
Data Struct(s)