Transcript Invariant-Based Automatic Testing of Ajax User Interfaces

Dynamic Analysis and Testing of
Ajax User Interfaces
Ali Mesbah, Arie van Deursen
1-4-2016 @ IPA Herfstdagen
Software Engineering Research Group
Delft University of Technology, The Netherlands
Delft
University of
Technology
Software Engineering Research Group
Classical Web Applications
HTTP
Client
Browser
GET
HTML
Server
Shortcomings of the Multi-page style:
• Low lever of user interactivity
• Redundant data transfer
• High user-perceived latency
• Client is passive
Dynamic Analysis and Testing of Ajax User Interfaces
2 | 26
Ajax
Asynchronous JavaScript and XML
• Asynchronous server communication
through XMLHttpRequest
• Web standards-based (HTML, CSS,
XML)
• Dynamic display through DOM
• JavaScript binds everything together
Features:
• No full page refresh!
• Single-page web interface
• Google Suggest, Gmail, Yahoo! Mail
Dynamic Analysis and Testing of Ajax User Interfaces
3 | 26
Ajax-based Web Applications
•
•
•
•
•
Client-side execution
State changes & navigation
Dynamic DOM
Delta-communication
Clickables: Elements having
event-listeners attached to
them
<a href="javascript:OpenNewsPage();">
<a href="#" onClick="OpenNewsPage();">
<div onClick="OpenNewsPage();">
<a href="news.html" class="news">
<input type="submit" class="news"/>
<div class="news">
$(".news").click(function() {
$("#content").load("news.php");
});
Dynamic Analysis and Testing of Ajax User Interfaces
4 | 26
Research Question
How can we automatically analyze (test) Ajax
web applications?
Dynamic Analysis and Testing of Ajax User Interfaces
5 | 26
Current Approaches
• General Approach: extract link, send a request to the server and
analyze the response. i.e., HTML (based on the Multi-page style)
• Capture/Replay: demands a substantial amount of manual effort
• Static Analysis: misses the complex run-time behavior
• Dynamic Analysis: promising but detecting the various doorways
to different dynamic states and providing proper interface
mechanisms for input values is challenging.
Dynamic Analysis and Testing of Ajax User Interfaces
6 | 26
A Method for Deriving Ajax States
• Conduct dynamic analysis of the user interface to find Clickable
elements and state changes
• Reverse-engineer a State-flow Graph to abstract the navigation
paths and state changes
Dynamic Analysis and Testing of Ajax User Interfaces
7 | 26
User Interface State Changes
• In classic web applications, each UI state is represented by a URL and
the corresponding web page
• In Ajax, it is the internal change of the DOM tree on the (single-page)
user interface that represents a UI state change
• To adopt a generic solution, we define a UI state change as:
a change on the DOM tree caused either by server-side state
changes propagated to the client, or client-side events
handled by the Ajax engine
Dynamic Analysis and Testing of Ajax User Interfaces
8 | 26
State-flow Graph
Dynamic Analysis and Testing of Ajax User Interfaces
9 | 26
Inferring the State Machine
(Crawljax)
Dynamic Analysis and Testing of Ajax User Interfaces
10 | 26
Detecting Clickables
•
•
•
Find a set of Candidate Elements (CC)
Expose each to en event (onclick, onmouseover, …)
Analyze the effect on the DOM
We distinguish between 3 ways of obtaining CCs:
1.
2.
3.
Full Auto Scan: HTML tagname (div, span, a, …) with
attribute constraints, e.g., div:{class=“menuitem”}
Annotation: element attribute crawljax=“true”, or “false”
A DSL: Crawling Ajax Specification Language (CASL)
Dynamic Analysis and Testing of Ajax User Interfaces
11 | 26
Detecting States
Input: DOM Trees before and after the event:
•
Compare DOM Trees:
•
•
•
•
•
As is
Textual: extract textual content and compare
Structural: extract structural content and compare
Meta-model: reverse-engineer the Schema for each DOM, then
compare the schemas
Comparison method:
Edit Distance, Levenshtein method with a similarity threshold
Dynamic Analysis and Testing of Ajax User Interfaces
12 | 26
Processing Document Tree Deltas
Upon every new state entry:
• Compute the differences between the previous document tree and the
current one
• We use an enhanced Diff algorithm to find the “delta updates”
• Delta updates may be due, for example, to a server request call that
injects new elements into the DOM
• Find new candidate elements in the delta updates, and examine them
in a recursive depth-first manner
Dynamic Analysis and Testing of Ajax User Interfaces
13 | 26
Navigating the States
•
Problem: going ‘Back’ in the browser
•
Solutions:
1.
2.
Register the state explicitly and use Back
Click through from the initial state
•
Use element IDs: not always present, not always persistent
•
Use XPath Expressions: much more reliable, not 100% water-proof
Dynamic Analysis and Testing of Ajax User Interfaces
14 | 26
Data Entry Points
Detecting DOM forms:
• For each state change, extract all the forms
• For each form, calculate a hashcode based on the form properties
• Check if form is already present in the database
• If not, store the form completely in the database with the
hashcode as its ID
• If present, use the corresponding custom values in the database
(if present) to fill the form in the browser and submit
• Upon submission, the resulting state change is recursively
analyzed.
Dynamic Analysis and Testing of Ajax User Interfaces
15 | 26
Full Auto Scan
For
1.
2.
3.
4.
5.
6.
each state:
Retrieve all candidate clickables
Fire event on each candidate element
Compare DOM trees: Edit Distance. Is DOM changed?
Make a new State and add to the State-flow graph
Find the XPath Expression of the Clickable Element
Add the clickable as an edge to the SFG between state before and
after the click
7. Analyze forms
8. Do all this recursively
Dynamic Analysis and Testing of Ajax User Interfaces
16 | 26
Testing Ajax States
• With access to different dynamic DOM states we can check the
user interface against different constraints
• We use invariants as oracle
• Invariants on the DOM tree
• Invariants between DOM states, and
• Application-specific invariants
Dynamic Analysis and Testing of Ajax User Interfaces
17 | 26
Generic DOM Invariants
Generic DOM Invariants:
• Validated DOM
• No error messages in DOM
• Accessible State
• Secure State
• Discoverable links
• No Dead Clickables
• Consistent Back-Button
Dynamic Analysis and Testing of Ajax User Interfaces
18 | 26
Application-specific Invariants
Examples:
• //case one: warn about collapsable divs within expandable items
String xpathCase1 = "//LI[contains(@class
,’expandable’)]/DIV[contains(@class ,’collapsable’)]";
• //case two: warn about collapsable items within expandable items
String xpathCase2 = "//LI[contains(@class
,’expandable’)]/UL/LI[contains(@class ,’collapsable’)]";
Dynamic Analysis and Testing of Ajax User Interfaces
19 | 26
Testing Ajax Paths
• While running the crawler to derive the state machine can be
considered as a first full test pass, the state machine itself can be
further used for testing purposes.
We generate test cases from the state machine:
• K shortest paths from Index to sinks (nodes with no outgoing
edges)
• Cycles included only once, All-transitions coverage
• Transform each path found into a JUnit test case
Dynamic Analysis and Testing of Ajax User Interfaces
20 | 26
Testing Ajax Paths
@Test
public void testcase1 () {
browser.goToUrl (url);
/*Element-info: SPAN class=expandable-hitarea */
browser .fireEvent(new Eventable (new Identification("xpath",
"//DIV[1]/SPAN[4]"), "onclick"));
Comp.assertEquals(oracle.getState ("S_1"). getDom(),
browser.getDom ());
/*Element-info: DIV class=hitarea id=menuitem2 */
browser.fireEvent(new Eventable(new Identification("xpath",
"//P[2]/ DIV[2]"), "onmouseover"));
…
}
Dynamic Analysis and Testing of Ajax User Interfaces
21 | 26
Testing Ajax Paths
Generated test suite can be used in several ways:
• It can be run as is on the current version of the Ajax application,
but for instance with a different browser to find browser
incompatibilities
• It can be applied to altered versions of the Ajax application to
support regression testing (against a Gold-standard)
• The typical use of the derived test suite will be to take apart
specific generated test cases, and augment them with applicationspecific assertions, capturing specific fault-sensitive click trails.
Dynamic Analysis and Testing of Ajax User Interfaces
22 | 26
Tool Implementation
• Crawljax: crawling through dynamic states
• ATUSA: plugin-based framework on top of Crawljax:
• preCrawling
• inCrawling
• postCrawling
•
•
•
•
•
Java 1.6
Watij (IE), XULRunner (Mozilla)
Numerous libraries
Java Plugin Framework
Open source
Dynamic Analysis and Testing of Ajax User Interfaces
23 | 26
Tool Implementation
Dynamic Analysis and Testing of Ajax User Interfaces
24 | 26
Empirical Evaluation
• 4 case studies (3 open source), 1 industrial
• TUDU, FreakTask, PetStore, CoachJeZelf
• Instrumented the server side code with Clover
• Instrumented the client-side JavaScript code with JSCoverage
• Fault seeding and code coverage
• Code Coverage: 73% server-side, 70% client-side
• More than 180 Violations per case
• 80% fault detection
Dynamic Analysis and Testing of Ajax User Interfaces
25 | 26
Concluding Remarks
We have proposed a method for testing Ajax applications
automatically through
1. A crawler that can detect clickables and forms and analyze state
changes
2. A testing framework that can check the application behavior
through generic invariants as well as application-specific
invariants
3. A test suite generation from the inferred state machine.
•
Tools are available for download:
http://spci.st.ewi.tudelft.nl/content/software
Dynamic Analysis and Testing of Ajax User Interfaces
26 | 26
Publications
• Ali Mesbah, Engin Bozdag, and Arie van Deursen (2008).
Crawling Ajax by Inferring User Interface State Changes.
In Proceedings of the 8th International Conference on Web
Engineering (ICWE'08), IEEE Computer Society.
• Ali Mesbah and Arie van Deursen (2009). Invariant-Based
Automatic Testing of Ajax User Interfaces. In Proceedings
of the 31st International Conference on Software Engineering
(ICSE'09), IEEE Computer Society.
Dynamic Analysis and Testing of Ajax User Interfaces
27 | 26