Transcript ONS2016 SystemTest v1.3

System Testing SDN
- A Case Study on ONOS
Suibin Zhang, ON.Lab
Jon Hall, Ciena
Quality Assurance on Software System
Known
UNKNOWNs
Unknown
UNKNOWNs
✔
Known
KNOWNs
#ONOSProject
Objective Reality
Subjective Perception
2
System Test - Where Used
• Black/Grey Box Testing
– Often mimic usage scenarios
– Validity not influenced by internal design
• Product Verification/Qualification
• Pre-Deployment Verification
#ONOSProject
3
Testing SDN OS – Nontrivial
Test
Exec.
Legacy Network Domain
APP
APP
SDN
CTRL
1
SDN
CTRL
2
APP
Distributed
Controllers
SDN
CTRL
n
SDN Network Domain
Dependencies
#ONOSProject
4
Challenges in Building System Tests
• Technical
– Accessing heterogeneous, evolving interfaces
– Scaling test environment
– Facilitate Debugging
• Arise from Community/Open Source
– Run “anywhere”
– Handling test dependencies
– Contribution and merging tests
#ONOSProject
5
A Case Study on ONOS
#ONOSProject
6
ONOS System Test Suites
• Functionality Suite
– NB: Intents and Flow operations
– SB: OF 1.0 &1.3, NETCONF, OVSDB, etc.
• High Availability Suite
– Various scenarios of cluster failures
• Performance and Scale-out Suites
– Intent, flow throughputs, latency as scaling-out
– Topology discovery latency as scaling-out
• Longevity
– Continuous operation under typical deployment scenarios
• Selected Use Case Suites
– SDNIP App, Segment Routing App
#ONOSProject
7
Onos System Test CI
 Jenkins Plugins
 Update Codes
 Publish Results
 Reset/Clean Test Env.
 Inject Run Params to
TestStation
 Start Test Cases
 Establish Test Handles
 Run-time Configuration
 Test Logic
#ONOSProject
8
Onos System Test CI
A Deeper Dive into TestON
#ONOSProject
9
TestON – Key Benefits
• TestON
o SDN-focused Testing Framework
o Test case authoring, execution
o Python-based
• Key Benefits:
o Extensibility – Simplify case authoring by compartmentalizing
concerns
o Flexibility - Handle heterogeneous test env. OVS, HW, Quagga, etc.
o Debuggability - Facilitate troubleshooting
#ONOSProject
10
Details on TestON Framework
TestON Core Framework:
Logging, Utilities, …
*.params
*.topo
*.py
Dependency
Test Files
.params File – XML file,ONOS
used to
into
test, such as order of test cases
CLIpass variables
ONOS
Rest
Mininet
.topo
file Driver
– XML File, defines
the components
of the test Quagga Driver
Driver
Driver
Dependency• Folder
Config files or Mininet topology files used by the test
Driver–Files
.py file – Combine
TestON
and CLI
driver
functions
tocustom
defineconnection
your testinterfaces
cases.
• Can
use REST,
via SSH,
or other
Defined
in a hierarchical
structure
Test Cases •are
split into
steps, each
of which should have an assertion in them
• Creates a library of functions to be used in the test
#ONOSProject
11
Example: ONOS Intent Functionality Test
ONOS Intent Feature
• High-level abstraction of setting end to end connectivity
• Can be defined by “Selectors” of MAC, IP, PORT, MPLS label, etc.
System Test Procedure
•
•
•
Set intent through NB REST interface
Validate intent and flows are correctly added
Test end-to-end connectivity in Mininet with the specific selectors
System Test Requirement
•
•
Need a library to send packets with test-specified selector
Leveraging Python Scapy module
#ONOSProject
12
Example: ONOS’ Intent Functionality Test
Building a Scapy Driver
TestON
|--drivers
| |--common
| |--api
class ScapyCliDriver( Emulator ):
| |--cli
def sendPacket( self, iface=None, packet=None, timeout=1 ):| | |--emulator
try:
| | |--ScapyCliDriver.py
sendCmd = 'srp( '
|--tests
(send/recv user-defined packets)
if packet:
sendCmd += packet
else:
...
if iface:
sendCmd += ", iface='{}'".format( iface )
sendCmd += ', timeout=' + str( timeout ) + ')'
self.handle.sendline( sendCmd )
self.handle.expect( self.scapyPrompt )
except:
...
#ONOSProject
13
Example: ONOS’ Intent Functionality Test
Using Scapy Library
- in Mininet Host Handle
TestON
|--drivers
|--tests
|--Example
|--Example.py
|--Example.params
|--Example.topo
for hostName in [ “h1”,”h2” ]:
# Create two host components with the scapy driver4
main.Scapy.createHostComponent( hostName )
main.step( ”Login to scapy on these new components" )
for host in [ main.h1, main.h2 ]:
<Scapy>
<host>192.168.1.101</host>
host.startHostCli()
<user>admin</user>
host.startScapy()
<password></password>
<type>ScapyCliDriver</type>
<connect_order>6</connect_order>
#ONOSProject
</Scapy>
14
Example: ONOS’ Intent Functionality Test
Intent Test Case
- Using Scapy Driver
TestON
|--drivers
|--tests
|--Example
|--Example.py
|--Example.params
|--Example.topo
# Add connectivity from h1 to h2 (unidirectional)
addResult = main.ONOS1.addPointIntent( ingress, egress, ethDst, ipDst)
main.h2.startFilter() # Start packet filter on host2
main.h1.buildEther( dst=main.h2.hostMac ) # build ethernet frame
main.h1.buildIP( dst=main.h2.hostIp ) # Build IP frame
main.h1.sendPacket() # Send built packet
finished = main.h2.checkFilter() # Check if host2 received any packets
#ONOSProject
15
Lessons Learned and Future Work
• Parameterize Test Env & Run Var
– Jenkins to inject at run time
– Test cases deal with “pure” test
logic
• Extensive Exception Handling in
Drivers
– Take care of exceptions, timeout
– Use “retry” method
#ONOSProject
• Test Dependencies
– Run-time Activation
– “Dockerize” Fixture (Future Work)
• Debug Utilities
•
– Test pausing
– Log collation
– More (Future Work)
Scale Test Network (Future Work)
16
Summary
ONOS System Test Highlights
ONOS Test Suites:
Functionality
High Availability
Performance
Scale-out Capability
Longevity
Use Cases
Open Source
Extensibility
Python-Based
Flexibility
Debuggability
For more Information
•
S3 Showcase: “SDN/ONOS System Test”
•
Wiki: https://wiki.onosproject.org/display/ONOS/System+Tests
•
Repo: https://gerrit.onosproject.org/#/admin/projects/OnosSystemTest
•
Github: https://github.com/opennetworkinglab/OnosSystemTest
#ONOSProject
17
Software Defined Transformation of Service Provider Networks
Q&A
Join the journey @ onosproject.org
#ONOSProject
18