Transcript click( “ feeling lucky ” )

Automation and Customization
of Rendered Web Pages
Rob Miller
User Interface Design Group
MIT Computer Science & Artificial Intelligence Lab
[email protected]
http://uid.csail.mit.edu/
Joint work with Michael Bolin, Greg Little, Eric Lieberman,
Marcos Ojeda, Matt Webber, Philip Rha, Tom Wilson, Darris
Hupp, Ricarose Roque
Web Applications
• The Web is now an application platform
– Many web sites are now rich applications
– As we spend more time using them, we want to customize them for
our own needs
MIT Computer Science and Artificial Intelligence Laboratory
2
NPUC
July 24, 2006
Web Automation & Customization
• Automate repetitive operations
– Bookmark my latest bank statement
– Click on the nearest download mirror site
– Print or save a list of web pages
– Fill in defaults on a web form
MIT Computer Science and Artificial Intelligence Laboratory
3
NPUC
July 24, 2006
Web Automation & Customization
• Automate repetitive operations
• Transform appearance
– Change color schemes for better contrast
– Concatenate a sequence of pages into one page
– Simplify site to remove unnecessary or distracting features
MIT Computer Science and Artificial Intelligence Laboratory
4
NPUC
July 24, 2006
Web Automation & Customization
• Automate repetitive operations
• Transform appearance
• Integrate multiple web sites (“mashups”)
– Bookstore has links for New Books, Used Books, Auctions… but it
doesn’t know about books in my local library
– Real estate site has lots of data about houses for sale… but it can’t
compute the length of my trip to work
MIT Computer Science and Artificial Intelligence Laboratory
5
NPUC
July 24, 2006
Web Apps Are Inherently Open
• Web applications have built-in hooks for scripting and
modification
can be automated
HTTP requests
HTML/XML replies
can be parsed for data
HTML/CSS display
can be modified to change appearance
• Desktop applications don’t have these hooks unless the
developer intentionally provides an API
• So it’s possible for an end user to change a web application
– Greasemonkey for Firefox, User Javascript for Opera
– But it’s not easy
MIT Computer Science and Artificial Intelligence Laboratory
6
NPUC
July 24, 2006
Problem: It’s Hard to Customize
• Web applications are complex under the hood
• The user has to know how to program
var xpath = "//text()[(parent::" + allowedParents.join(" or parent::") + ") and " +
"contains(translate(., 'HTTP', 'http'), 'http')]";
var candidates = document.evaluate(xpath, document, null,
XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE, null);
for (var cand = null, i = 0; (cand = candidates.snapshotItem(i)); i++) {
if (urlRegex.test(cand.nodeValue)) {
http://www.mit.edu/...
var span = document.createElement("span");
var source = cand.nodeValue;
MIT Computer Science and Artificial Intelligence Laboratory
7
NPUC
July 24, 2006
Our Approach
• Avoid complexity by looking at the user interface, not the
HTML code
• Avoid learning a programming language by using keywords
rather than language syntax
click feeling lucky button
push “I’m Feeling Lucky”
feeling lucky
MIT Computer Science and Artificial Intelligence Laboratory
8
NPUC
July 24, 2006
Outline
Introduction
• Web automation
– Demo
– How it works
• Keyword commands
– Motivation
– Demo
– How it works
• Discussion
MIT Computer Science and Artificial Intelligence Laboratory
9
NPUC
July 24, 2006
Web Automation
• Chickenfoot is our new web automation system
– User shouldn’t have to look at HTML source to customize the Web
– Runs in Mozilla Firefox web browser
– Available for download from http://uid.csail.mit.edu/chickenfoot
MIT Computer Science and Artificial Intelligence Laboratory
10
NPUC
July 24, 2006
Web Automation Demo
DEMO
MIT Computer Science and Artificial Intelligence Laboratory
11
NPUC
July 24, 2006
Web Automation Commands
• Page navigation
go(url)
openTab(url)
fetch(url)
• Clicking and form manipulation
click(button-or-link)
enter([textbox], value)
check(checkbox)
pick([listbox], choice)
• Pattern matching
find(pattern)
• Page modification
insert(pattern, html)
remove(pattern)
replace(pattern, html)
• Widgets & input handling
Button(label, action)
Link(label, action)
onClick(pattern, action)
MIT Computer Science and Artificial Intelligence Laboratory
12
NPUC
July 24, 2006
Keyword Patterns
• Keywords + component type
feeling lucky button
depart textbox
search web form
• Component type is optional if it can be inferred
– For example, click() command assumes a button or a link
MIT Computer Science and Artificial Intelligence Laboratory
13
NPUC
July 24, 2006
Pattern Matching Algorithm
Keyword pattern
google search button
Web page
Weighted list of
components
1.0
Matcher
0.5
MIT Computer Science and Artificial Intelligence Laboratory
14
NPUC
July 24, 2006
Pattern Matching Algorithm
Keyword pattern
google search button
Web page
Weighted list of
components
1.0
Matcher
0.5
Steps
1. Find labels matching the keywords
– Text in the page, button labels, menu items, image ALT attributes
– Matching ignores word ordering, punctuation, capitalization
MIT Computer Science and Artificial Intelligence Laboratory
15
NPUC
July 24, 2006
Pattern Matching Algorithm
Keyword pattern
google search button
Web page
Weighted list of
components
1.0
Matcher
0.5
Steps
1. Find labels matching the keywords
2. Find components matching label
MIT Computer Science and Artificial Intelligence Laboratory
16
NPUC
July 24, 2006
Pattern Matching Algorithm
Keyword pattern
google search button
Web page
Weighted list of
components
1.0
Matcher
0.5
Steps
1. Find labels matching the keywords
2. Find components matching label
3. Rank the components & pick the best match
– How well do the keywords match the label? (# of matching keywords)
– How well does the label match the component? (pixel distance)
MIT Computer Science and Artificial Intelligence Laboratory
17
NPUC
July 24, 2006
Outline
Introduction
Web automation
• Keyword commands
– Motivation
– Demo
– How it works
• Discussion
MIT Computer Science and Artificial Intelligence Laboratory
18
NPUC
July 24, 2006
Problem: How Do I Learn the Language?
tbox = find('textbox');
text = tbox.element.value;
click('advanced image search');
enter('all of the words', text);
pick('small');
pick('GIF');
click('search button');
button = new Button('Icon Search', doIconSearch);
insert(after('search images button'), button);
MIT Computer Science and Artificial Intelligence Laboratory
19
NPUC
July 24, 2006
Keyword Commands
• Save the user from learning details of syntax
– Don’t worry about precise command names, punctuation, parameter ordering
Chickenfoot
go(“google.com”);
enter(“npuc 2006”);
click(“feeling lucky button”);
• Divide command into keywords
• Search for keywords in context
Keyword Commands
go to google.com
type “npuc 2006”
click feeling lucky button
google.com
npuc 2006
feeling lucky
– Some keywords are found in user interface; others are command names
• Generate a ranked list of possible interpretations
– Punctuation and keyword ordering affect ranking, but are not required
• Use the highest-ranked interpretation
– If ambiguous, ask the user to choose an interpretation
MIT Computer Science and Artificial Intelligence Laboratory
20
NPUC
July 24, 2006
Keyword Commands Demo
DEMO
MIT Computer Science and Artificial Intelligence Laboratory
21
NPUC
July 24, 2006
Advantages of Keyword Commands
• Is this natural?
– Yes: this is how my 2-year-old son talks, and I can understand him
– Most users are at the same level when it comes to talking to a computer in a
formal language
– Our computers should become smart enough to understand us when we talk
this way
• Similar to keyword matching
– e.g., web search engines
• Benefits novice users and expert users differently
– Novices may use verbose, almost natural language commands
– But experts can be very concise
• Keyword commands vs. point & click interfaces
– More appropriate in some contexts (especially if combined with speech)
– May even be faster (no hunting through menus and dialog boxes)
• Keyword commands vs. macro recording
– Macro recordings don’t capture the user’s intention, but keyword programs do
– Keyword programs are text, which can be easily modified and shared with
others
MIT Computer Science and Artificial Intelligence Laboratory
22
NPUC
July 24, 2006
How Keyword Commands Work
Keyword command
press feeling lucky button
Function
Database
button
click
Program code
Translator
keywords
textbox string
enter
click,
press,
push,
…
MIT Computer Science and Artificial Intelligence Laboratory
url
keywords
findButton
findTextbox
go
button
textbox
23
NPUC
July 24, 2006
How Keyword Commands Work
Keyword command
press feeling lucky button
Translator
keywords
Function
Database
Program code
click(“feeling lucky”)
findButton
button
click
MIT Computer Science and Artificial Intelligence Laboratory
24
NPUC
July 24, 2006
Outline
 Introduction
 Web automation
 Keyword commands
• Discussion
– Robustness
– Security
– Resistance from web application developers
MIT Computer Science and Artificial Intelligence Laboratory
25
NPUC
July 24, 2006
Robustness
• When web applications change, user customizations and
mashups break
– Customizations need to be robust to change
• Open question: How does the abstraction level of a
customization affect its robustness?
Rendered web page
DOM or raw HTML
Web service
<SOAP-ENV:Body>
<ns1:doGoogleSearch xmlns:ns1="urn:GoogleSearch"
SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<key xsi:type="xsd:string">00000000000000000000000000000000</key>
<q xsi:type="xsd:string">shrdlu winograd maclisp teletype</q>
MIT Computer Science and Artificial Intelligence Laboratory
26
NPUC
July 24, 2006
Security
• Can user customizations open security holes for malicious
web sites to exploit?
– Greasemonkey’s GM_xmlhttprequest was an unexpected hole
– Chickenfoot has to open carefully-controlled holes in the browser’s
security model to make certain things possible
https:// page
file:// images
localUrl(“file://...”)
“chicken-bypass:4hXFDh:file://…”
• Can users be fooled into installing malicious
customizations?
– Yes: spyware is a case in point
– Making customizations transparent (easy to read and understand)
might help
MIT Computer Science and Artificial Intelligence Laboratory
27
NPUC
July 24, 2006
Resistance from Web App Developers
• “Don’t muck around with my DOM!”
–
–
–
–
Design: user customizations may make the site ugly or less usable
Financial: customizations that hide or remove ads
Server load
Copyright
• We’ve been down this road before
– Ad blockers
– Putting frames around other sites’ content
– Deep linking
• Web sites that fight their users will lose their users
MIT Computer Science and Artificial Intelligence Laboratory
28
NPUC
July 24, 2006
Conclusion
• Two major barriers to automation and customization
– Complexity of programming system
– Difficulty of language syntax
• We can tackle these barriers with two new techniques
– Programming by looking at the user interface
go(“google.com”);
enter(“npuc 2006”);
click(“feeling lucky button”);
– Eliminating syntax from commands
click feeling lucky button
push “I’m Feeling Lucky”
feeling lucky
• Chickenfoot is available for download from
http://uid.csail.mit.edu/chickenfoot
MIT Computer Science and Artificial Intelligence Laboratory
29
NPUC
July 24, 2006