Transcript Automated Test of Drivers and Process DB

Automatic Regression Test
Facility for Support Modules
Jon Thompson, Diamond Light Source
Vancouver, 1 May 2009
Introduction
• Four major parts to software engineering:
–
–
–
–
Code: We are good at this.
Design: Should come first!
Document: Doxygen etc.
Test: Often manual and hard to repeat.
• DSC and the US telephone network.
– Google for: DSC communications software fault
• Adding that weird feature to existing code.
The Lessons
• Testing is not an optional extra.
• Tests must be properly repeated at least
before every release and preferably more
often.
• Tests are easier to repeat if they are
completely automatic.
The Requirements
•
•
•
•
•
•
•
•
•
•
Testing of device support modules against a Python simulation of the device. Most recent
stream device support modules have been implemented alongside a python simulation and
an example Linux based IOC.
Testing of device support modules against the device hardware. Getting enough feedback
from the hardware into the test cases may require extra hardware.
Hardware and simulation testing should utilise the same test scripts. Where there are
differences (for example, getting feedback from the device), the test case must take the
target device into account.
Tests scripts to be written in Python utilising standards as appropriate and available.
Automatic identification of support modules that have test scripts. This will require the
establishment of standards regarding the location of test IOCs and test scripts within a
module’s directory.
Cron job to start the test sequence at regular intervals. Ideally, starting in the evening with
results available the following morning.
Test sequence to run without routine user intervention. The operation of, and feedback
from, any hardware must all be automatic. This may require extra hardware to provide
inputs and monitor outputs.
Daily test sequence results to be made available through a web page or email.
Provide information regarding the ‘quality’ or ‘coverage’ of test suites.
Entire test sequence to run in a ‘reasonable’ time. This may mean that test suites must be
run in parallel with some sort of server computer starting tests and collecting the results.
What Do You Get
• A software framework that allows the
creation of automatic test suites for
modules.
• A script that runs test suites and collects
reports.
• A hardware test bench containing ‘high
value’ equipment.
• Software simulations of equipment.
Initial Prototype
Hardware
Software
Motor
Ethernet Switch
Newport XPS
Test IOC
Simulation
Geobrick
Motor
Linux PC
Test Cases
Run Script
The Framework
• Based on PyUnit, the Python unit testing
library.
• Test reports conform to the TAP protocol.
• Monitoring of EPICS database records for
coverage reporting.
• Hardware and simulation targets can use
same test suite.
The Run Script
• Searches a directory tree for modules with
automatic test support.
• Runs tests suites it finds, logging results and
output.
• Can run multiple tests in parallel subject to
resource constraints.
• Level of diagnostic output can be specified.
• Tests run can be restricted to single module,
single target, single case.
Device Simulations
• Based on existing DLS practice.
• Written in Python.
• Currently capable of simulating any device
that communicates through serial or IP
connection.
• Support for instrumentation to allow
protocol coverage reporting.
Directory Structure
• Test suites must exist in a standard place
in a module so that the run script can find
them.
The support module’s ‘TOP’
<moduleName>
dls
test
tests.py
Framework
EpicsRecord
object
X
Classes provided by PyUnit.
X
Classes provided by the framew ork.
-identifier
-fields[*]
records
*
object
EpicsDatabase
epicsDatabase
0..1
unittest.TestSuite
unittest.TestCase
_tests
+runTest()
*
+runTests()
unittest.TestResult
TestResult
TestSuite
«use»
-stream
suite
TestCase
-sim[0..1]
+createTests()
-failures[*]
object
targets
*
Target name
simDevices
0..1
SimDevice
Usage
TestSuite
X
Classes provided by the framew ork.
X
Classes the user must w rite.
suite
TestCase
-sim[0..1]
+createTests()
Target
targets
*
Optional intermediate
class that provides
support functions
common to all cases.
TheTestSuite
+createTests()
The test suite class creates an
instance of DiamondTarget for
each target against w hich the
tests are to run and registers
them w ith the test suite super
class.
CommonSupport
Case1
CaseN
+runTest()
+runTest()
Case2
The test suite class creates a
single instance of each test case
and registers them w ith the
test suite super class.
+runTest()
Example – The Test Suite
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
class Fw102TestSuite(TestSuite):
def createTests(self):
# Define the targets for this test suite
Target("simulation", self,
iocDirectory="example",
iocBootCmd="bin/linux-x86/stfw102Ex_sim.boot",
epicsDbFiles="db/fw102Ex.db",
simDevices=[SimDevice("controller1", 9016)],
guiCmds=['edmMain -m "P=FGZ73762,M=:WHEEL1" -eolc -x data/fw102.edl'])
Target("hardware", self,
iocDirectory="example",
iocBootCmd="./bin/linux-x86/stfw102Ex.boot",
epicsDbFiles="db/fw102Ex.db",
guiCmds=['edmMain -m "P=FGZ73762,M=:WHEEL1" -eolc -x data/fw102.edl'])
# The tests
CaseLocalIncrementSwitch(self)
CaseLocalDecrementSwitch(self)
CasePowerOffOn(self)
CasePvIncrement(self)
CasePvDecrement(self)
CasePvMultipleIncrement(self)
CasePvMultipleDecrement(self)
CasePvSetPosition(self)
CasePvTriggerMode(self)
CasePvPolling(self)
Example – Intermediate Class
•
•
•
•
•
•
•
•
•
•
class Fw102Case(TestCase):
def curDevicePos(self):
''' Get the current wheel position from the device simulation'''
result = 0
self.command("controller1", "getpos")
args = self.recvResponse("controller1", "pos", 1)
if args is not None:
result = int(args[0])
return result
•
•
•
•
•
•
•
def verifyPosition(self, intended):
''' Verify that the wheel is in the intended position'''
if self.simulationDevicePresent("controller1"):
self.verify(self.curDevicePos(), intended)
self.verifyPv("FGZ73762:WHEEL1:POSITION_RBV", intended)
self.verifyPv("FGZ73762:WHEEL1:POSITION", intended)
self.verifyPv("FGZ73762:WHEEL1:INPOS", 1)
•
•
•
def initialPosition(self):
''' Initialise for a test case, return current position'''
...
Example – A Test Case
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
class CasePvIncrement(Fw102Case):
def runTest(self):
'''The PV increment command.'''
# Check the current position of the wheel
before = self.initialPosition()
# Take the wheel round twice
for i in range(12):
# Now advance the wheel using channel access
self.putPv("FGZ73762:WHEEL1:STEPFW.PROC", 1)
after = before + 1
if after > 6:
after = 1
# Check the final position of the wheel
self.sleep(2)
self.diagnostic("Before=%d, after=%d" % (before, after), 1)
self.verifyPosition(after)
before = after
Example - Simulation
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
class fw102(serial_device):
Terminator = "\r"
regexp = re.compile(r'\s*(\w*)([\=\?])(\w*)')
def __init__(self):
'''Constructor. Remember to call the base class constructor.'''
serial_device.__init__(self,
protocolBranches = ["setpos", "settrigger", "getpos", "gettrigger"])
self.position = 1
self.trigger = 0
def command(self, text):
'''Interface function for commands from the test suite.'''
args = text.split()
if args[0] == "incr":
self.position = self.position + 1
if self.position > 6:
self.position = 1
elif args[0] == "decr":
self.position = self.position - 1
if self.position < 1:
self.position = 6
elif args[0] == "getpos":
self.response("pos %d" % self.position)
else:
serial_device.command(self, text)
def initialise(self):
'''Called by the framework when the power is switched on.'''
self.trigger = 0
Example - Simulation
•
•
•
•
•
def reply(self, command):
'''This function must be defined. It is called by the serial_sim system
whenever an asyn command is send down the line. Must return a string
with a response to the command or None.'''
result = None
•
•
•
•
•
•
•
•
# Parse the command
m = self.regexp.match(command)
if m == None:
return result
m = m.groups()
cmd = m[0]
set = m[1] == '='
value = m[2]
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
# Decode and act on the command
if self.isPowerOn():
if set:
if cmd == "pos":
self.position = int(value)
result = "pos=%d" % self.position
elif cmd == "trig":
self.trigger = int(value)
result = "trig=%d" % self.trigger
else:
if cmd == "pos":
result = "pos?\r%d" % self.position
elif cmd == "trig":
result = "trig?\r%d" % self.trigger
return result
•
Example - Test Report
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
[fgz73762@pc0054 diamondtest]$ ./runtests.py -t simulation -f default.config -i -g -b -p 3 -l tests.log -q -m fw102
[1] 1..10
[1] ok 1 - The local increment switch.
[1] ok 2 - The local decrement switch.
[1] ok 3 - Power off and on.
[1] ok 4 - The PV increment command.
[1] ok 5 - The PV decrement command.
[1] ok 6 - Fast increment sync correction.
[1] ok 7 - Fast decrement sync correction.
[1] ok 8 - Movement directly to a position.
[1] ok 9 - Control of the trigger mode.
[1] ok 10 - Control of the polling mechanism.
[1] # ==============================
[1] # Passed 10/10 tests, 100.00% okay, in 260.57s
[1] #
[1] # ==============================
[1] # Sim device controller1 coverage report:
[1] # setpos: ok
[1] # settrigger: ok
[1] # getpos: ok
[1] # gettrigger: ok
[1] #
[1] # ==============================
[1] # EPICS database coverage report:
[1] # FGZ73762:WHEEL1:TRIGGER(mbbo): ok
[1] # FGZ73762:WHEEL1:TRIGGER_RBV(mbbi): ok
[1] # FGZ73762:WHEEL1:RESTART(ai): not touched
[1] # FGZ73762:WHEEL1:INPOS(calc): ok
[1] # FGZ73762:WHEEL1:REINIT1(fanout): ok
[1] # FGZ73762:WHEEL1:POSITION(mbbo): ok
[1] # FGZ73762:WHEEL1:COMMSFAIL(bi): ok
[1] # FGZ73762:WHEEL1:POSITION_RBV(longin): ok
[1] # FGZ73762:WHEEL1:CALCCOMMS(calcout): ok
[1] # FGZ73762:WHEEL1:DISABLEPOLL(bo): ok
[1] # FGZ73762:WHEEL1:STEPBK(calcout): ok
[1] # FGZ73762:WHEEL1:STOP(bo): values not covered: 0, 1
[1] # FGZ73762:WHEEL1:OFFLINEDET(calcout): ok
[1] # FGZ73762:WHEEL1:STEPFW(calcout): ok
[1] # FGZ73762:WHEEL1:POSTRACK(calcout): ok
[1] #
[1] # ==============================
What Next
• Coverage reports for support modules written in
‘C’ may be possible.
• Coverage reports for the Python simulation.
• Improvements to the resource management of
the run script.
• Automatic generation of dependancies.
• Implement cron job systems and web result
reporting.
• More hardware in the test racks.
• Retro-fit test suites to existing support modules