Transcript eu_20070922_wmtr

Scenario-Based Requirements
for Web Macro Tools
Christopher Scaffidi,
Allen Cypher, Sebastian Elbaum,
Andhy Koesnandar, Brad Myers
PBE and browser automation
• Programming-by-example (PBE) is a success story
– Research systems: Eager, Marquise
– Systems in widespread use: Microsoft Office macros
• Researchers have applied PBE to browser automation
– e.g.: Internet Scrapbook, Koala, Miro, Robofox, Turquoise
– These tools…
1. Watch the user perform a task
2. Infer the user’s intent
3. Record a web macro representing the intent
4. Allow the user to replay the macro on new data
2
introduction  scenarios  requirements
 Robofox  conclusion
Step 1:
Open http://www.oanda.com/converter/classic
Step 2:
Submit the amount, date, foreign currency type, and target currency.
Step 3:
Retrieve the result in the target currency.
PBE web macro tools are
not widely adopted
• The web context introduces new challenges to PBE:
– Web site changes can break macros without warning
 Need features for detecting and responding to site change
– Office tasks involve not just browsers
but also spreadsheets and other applications
 Need integration with other applications
• What other unmet requirements are there?
4
introduction  scenarios  requirements
 Robofox  conclusion
Missing: a methodical characterization of
requirements for web macro tools
• Many papers use scenarios to explain and motivate a
PBE tool’s features.
• Such scenarios…
– Are retrospective: The scenarios show the features that
actually work, rather than opportunities for future work.
– Lack pedigree: Are the scenarios hypothetical or based on
a user study? Will the tool features be useful in practice?
5
introduction  scenarios  requirements
 Robofox  conclusion
Outline
• We have…
– Analyzed 10 scenarios where web macros would be desirable
– Characterized 25 requirements that tools must support
to be useful for many real-world tasks
– Used these requirements as a benchmark to identify
opportunities for enhancing a particular tool (Robofox)
6
introduction  scenarios  requirements
 Robofox  conclusion
Catalog of scenarios
• Using a web site to look up information – 6 scenarios
– Currency converter, Package tracker, Per diem lookup,
Staff lookup, Stock analysis, Watcher for eBay
– Typical users: office workers
• Using a web form to order products – 1 scenario
– Path to procurement
– Typical users: online shoppers
• “Scraping” a large part of a site – 3 scenarios
– Peoplesoft scraper, Person locator scraper,
Scraper for CMS
– Typical users: System administrators
7
introduction  scenarios  requirements
 Robofox  conclusion
Scenarios came from a variety of sources
• Contextual inquiry – 3 scenarios
– Currency converter, Per diem lookup, Staff lookup
• Co-workers – 2 scenarios
– Path to procurement, Peoplesoft scraper
• Our own experiences – 2 scenarios
– Package tracker, Scraper for CMS
• Online sources – 3 scenarios
– Person locator scraper, Stock analysis, Watcher for eBay
8
introduction  scenarios  requirements
 Robofox  conclusion
What is in a scenario?
*
Title
Pithy name to make it easy to refer to the scenario
*
Typical User
Tells what type of person is likely to perform the task
*
Overview
Context and forces that would prompt a user to do the task
*
Starting Conditions Pre-conditions that hold true before the scenario
*
Result
Post-conditions that should hold true after the scenario
*
Actions
Steps that the user performs (maybe with a tool)
Action Details
Screenshots or other clarifying information
Variations
Discusses likely tweaks of the main scenario
Macro Maintenance Examines how scenario evolution prompting macro changes
*
Scenario Source
9
Summarizes empirical data that generated this scenario
* = required section
introduction  scenarios  requirements
 Robofox  conclusion
Requirements: Triggering macros
• On-demand execution
– Required to support users’ work tasks
• Scheduled execution
– Especially for scrapers
• Event-based triggers
– Watcher for eBay executes when user visits Google
homepage
• Subroutines
– Especially for lookup macros
10
introduction  scenarios  requirements
 Robofox  conclusion
Requirements: Using objects on pages
• Text snippets
– Almost all scenarios require reading text
• Tabular information
-
– Almost all scenarios read tabular data
• Web form widgets
– Read/write textbox, dropdown, checkbox widgets
• Other HTML structures
– Watcher for eBay
11
introduction  scenarios  requirements
 Robofox  conclusion
Requirements: Adapting to site changes
• Adaptation to changing page layout
– Changes to text location, text color, page structure
• Adaptation to changing form fields
-
– Changes to field names and field values
• Adaptation to changing URLs
– “This page has moved to X. Please update your links.”
12
introduction  scenarios  requirements
 Robofox  conclusion
Requirements: Reading and writing data
outside pages
• Browser APIs
– Read/write highlighted text, show popup window
• Spreadsheets and other files
-
– Often one record per spreadsheet row or XML node
• Parameters containing user input
– Prompting for an input (e.g. password) during playback
13
introduction  scenarios  requirements
 Robofox  conclusion
Requirements: Transforming data
• Reformat to equivalent value
-
– For comparing values and for making outputs consistent
• Extracting values’ parts
– For comparing values
• Combining values
– Such as arithmetic and string concatenation
14
introduction  scenarios  requirements
 Robofox  conclusion
Requirements: Executing control structures
• Looping operations
-
– Especially in scrapers
• Conditional operations
– Macro behavior depends on data provided by user as well
as data returned by web sites
15
introduction  scenarios  requirements
 Robofox  conclusion
Requirements: Recovering from failure
• Partial restarts
– Especially for scrapers
• Exception handlers
– Integration with assertions
16
introduction  scenarios  requirements
 Robofox  conclusion
Requirements: Supporting macro
maintenance
• User-understandable representation
– For maintenance and for sharing macros with other users
• Editable macros
– Analogous to features supported by textual editors
• Features for debugging
– Traces, breakpoints, step-by-step execution, etc
• Maintenance at runtime
– On error, enter maintenance mode
17
introduction  scenarios  requirements
 Robofox  conclusion
Example benchmark use: Robofox
Triggering macros
On-demand execution
Yes
Scheduled execution
Yes
Event-based triggers
Yes
Subroutines
No
Using objects on web pages
Text snippets
Yes
Tabular information
No
Web form widgets
Yes
Other HTML structures
No
• Inserts “sanity checks” to detect
changing page structure.
• Includes heuristics for “refinding”
page elements based on color,
label, DOM location.
• Some changes (e.g.: form field
renaming) are not handled
automatically.
Adapting to site changes
Adaptation to changing page layout
Limited
Adaptation to changing form fields
Yes
Adaptation to changing URLs
No
18
introduction  scenarios  requirements
 Robofox  conclusion
Example benchmark use: Robofox
Reading and writing data outside pages
Browser APIs
Limited
Spreadsheets and other files
Limited
Parameters containing user input
Limited
Transforming data
Reformat to equivalent value
No
Extracting values’ parts
Yes
Combining values
No
• Can read/write spreadsheets, but
only if they have a specific layout
• Can do a “foreach” loop but not a
“while(condition)” loop
Executing control structures
Looping operations
Conditional operations
19
Limited
Yes
introduction  scenarios  requirements
 Robofox  conclusion
Example benchmark use: Robofox
Recovering from failure
Partial restarts
No
Exception handlers
Yes
Supporting macro maintenance
User-understandable representation
Yes
Editable macros
Yes
Features for debugging
Yes
Maintenance at runtime
Yes
20
• User-defined exception handlers
can fire when assertions fail
introduction  scenarios  requirements
 Robofox  conclusion
Summary of Robofox limitations
• Of 25 requirements…
– Supported: 13
– Limited support: 5
– Unsupported: 7
• Of 10 scenarios…
– Supported: 3
– Limited support: 2 (unsupported scenario variations)
– Unsupported: 5
21
introduction  scenarios  requirements
 Robofox  conclusion
Opportunities to enhance Robofox
• Adding support for 2 requirements…
– Tabular information
– Reformat to equivalent value
• … would leave only…
– 2 unsupported scenarios
– 2 unsupported scenario variations
22
introduction  scenarios  requirements
 Robofox  conclusion
Extending the benchmark
• These 25 requirements are “Version 1” of a benchmark
for evaluating web macro tools.
• As researchers meet existing requirements, new
scenarios and new requirements will appear.
• We created a wiki where YOU can upload scenarios.
– http://softwaresurvey.cs.cmu.edu/wmcorpus.html
23
introduction  scenarios  requirements
 Robofox  conclusion
Thank You…
• To VL/HCC for the opportunity to present.
• To EUSES Consortium members for suggestions.
• To NSF and EUSES for funding
(CCF-0325273, CCF-0438929, CCF-0324861).
24
introduction  scenarios  requirements
 Robofox  conclusion