Transcript A RESTful Web Service Interface to the ATLAS COOL Database

A RESTful Web Service
Interface to the ATLAS COOL
Database
Shaun Roe
CHEP'09 Prague
A RESTful Web Service... (Shaun Roe, Atlas)
1
COOL: A database schema and API designed to be technology
neutral
(Oracle/MySQL/SQLite)
Data structure is hierarchical:
Data entry is to a particular
folder, channel and with a
specific Interval Of Validity
(IOV) and tag.
=>These form a unique set of
coordinates for the datum or
resource.
Originally in C++, its methods are exposed in Python,
in the ‘PyCool’ module.
CHEP'09 Prague
High Voltage values from the
Detector Control System of Atlas
SemiConductor Tracker
A RESTful Web Service... (Shaun Roe, Atlas)
2
http://lcgapp.cern.ch/doxygen/COOL/COOL_2_6_0/doxygen/ht
COOL (see talk by A. Valassi, this conference)
Roy Fielding’s dissertation: http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm
REST
REpresentational
State
Transfer
RESTful
A resource is defined by a unique
URL noun
What you ‘do’ to that resource is
defined by the http method: verb
POST: Create resource
GET:
Retrieve resource
PUT:
Update
DELETE: Delete resource
e.g. sending a form
e.g. A browser normally
GETs a web page
‘404 Not Found…’
The reply is given via http codes.
The format of the answer that you
‘Accept: text/html’
want can be further defined by http
headers
CHEP'09 Prague
A RESTful Web Service... (Shaun Roe, Atlas)
3
Motivation
Rewind to 2006…
People want data from Cool on the web!
Solutions:
• Use their own SQL from PHP/Python…
• Use a shell script to set up environment, then
run the Cool api behind some custom code
• Use a cron job to generate local html files
..and people want to insert data from the web, too.
CHEP'09 Prague
A RESTful Web Service... (Shaun Roe, Atlas)
4
Aim: Universal Web Service
Identify resources with url e.g.
http://server/database/schema/dbname/some/folder/path/timespan/0-100/channels/23
Use xml as the data transport format.‘Accept: text/xml’
‘GET’ retrieves the values
‘POST’ creates the channel in a folder
‘PUT’ updates the channel
‘DELETE’ deletes the channel from the folder
Multi-channel uploads/downloads should be possible. DB Modification should be
protected.
Additional resources to be matched to a url:
Database nodelist, folder description, folder payload definition,
list of channels in a folder, list of tags; it should be possible to create and delete
databases, folders.
CHEP'09 Prague
A RESTful Web Service... (Shaun Roe, Atlas)
5
Data format: XML
<channels server="ATLAS_COOLPROD" schema="ATLAS_COOLOFL_DCS" dbname="COMP200"
folder="SCT/DCS/CHANSTAT" since="1226271600000000000" until="1226274125000000000" tag=""
version="single”>
<channel id="138950656" since="1226270581000000000" until="1226274125000000000">
<value name="LVCHSTAT_RECV">193</value>
<value name="STATE">17</value>
</channel>
<channel id="138950656" since="1226274125000000000" until="1226277728000000000">
<value name="LVCHSTAT_RECV">193</value>
<value name="STATE">17</value>
XML Schema Description
</channel>
</channels>
Definition of an XSD schema allows validation of
data format on upload, or type-ahead and
documentation tooltips in an editor:
Namespaces?
JavaScript Object Notation
What about
JSON ? (see
later)
(oXygen editor)
CHEP'09 Prague
A RESTful Web Service... (Shaun Roe, Atlas)
6
Implementation: CherryPywww.cherrypy.org
CherryPy is a Python application server. It is very easy
to install and use.
The ‘Routes’ module (from Ruby) allows automatic or
manual mapping of Python classes and methods to
URLs:
d.connect('folder_payload','/:server/:schema/:dbname/*folderPath/payl
oad',controller=restController,action='folderPayload’)
maps this
http://myserver.cern.ch/cooldb/ATLAS_COOLPROD/ATLAS_COOL_SCT/DCSP200/SCT/DCS/HV/payload
to this
def folderPayload(self, server, schema, dbname, folderPath):
The http headers and the methods are also available and can be used
to route the URL to the appropriate Python method; other modules
for security, caching etc are available.
CHEP'09 Prague
A RESTful Web Service... (Shaun Roe, Atlas)
7
http://curl.haxx.se/
Examples, curl
Command line examples
Create a folder:
curl 'http://server/DEVDB10/ATLAS_SCT_COMMCOND_DEV/ACALTEST' -u
user:pwd -F [email protected] -X POST
Create channels:
curl
'http://server/DEVDB10/ATLAS_SCT_COMMCOND_DEV/ACALTEST/test/gain
/channels' -u user:pwd -F [email protected] -X POST
Insert values:
curl
'http://server/DEVDB10/ATLAS_SCT_COMMCOND_DEV/ACALTEST/test/gain
/timespan/0-9223372036854775807/channels’ -u user:pwd -F
[email protected] -X PUT
cURL (and libcurl, to use from C, C++, Perl) are standard installations.
Other languages have similar libraries: PHP – curl, Java – java.net.url,
Python – urllib2 … all are
standard installations, so accessing Cool via
CHEP'09 Prague
A RESTful Web Service... (Shaun Roe, Atlas)
CherryPy typically does not require any extra setup or libraries.
8
‘In the wild’ examples
/**
* @author formica
*/
public class CherryPyClient {
Java
private String name = "user.name";
private String psswd = "mypass";
private String readnodeUrl = "http://server/ATLAS_SCT_COMMCOND_DEV/ROETEST/Branch/Trial1/payload";
private String writenodeUrl = http://server/ATLAS_SCT_COMMCOND_DEV/ROETEST/Branch/Trial1/timespan/0-100;
public String coolGetCherryPy() {
try {
URL url = new URL(this.readnodeUrl);
java.net.URLConnection conn = url.openConnection();
InputStream is = (InputStream) conn.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(is));
String inputLine = in.readLine();
is.close();
return inputLine;
} catch (Exception e) {
PHP
e.printStackTrace();
}
$header=array('Accept: text/xml');
return "none";
//
}
$ch = curl_init();
Python
httpHeaders={'Accept':'text/xml'}
textData=None
request=urllib2.Request(url,textData, httpHeaders)
u=urllib2.urlopen(request)
xmlDocString = u.read()
return xmlDocString
CHEP'09 Prague
curl_setopt($ch, CURLOPT_HTTPHEADER,$header);
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$format=requested_format();
$http_result = curl_exec($ch);
if ($format != 'text/xml'){
$dom=domxml_open_mem($http_result);
}
curl_close($ch);
A RESTful Web Service... (Shaun Roe, Atlas)
9
Example: Run List
Search for runs matching criteria, display
results;
Uses an ‘Ajax’ request to get the CherryPy
XML.
XML is inserted
directly in the
XHTML page, and
associated with an
XSLT stylesheet to
format it as a table
eXtensible Stylesheet Language
for Transformations
Client-side
CHEP'09 Prague
Local server side
A RESTful Web Service... (Shaun Roe, Atlas)
10
Plans, conclusion
Use of CherryPy allowed rapid development of a web service interface to
Cool which greatly simplified conditions access and adheres to ‘RESTful’
principles.
It was used during 2008 as the engine for the ‘run list’ query page, for various
dcs retrieval services, and for insertion of detector status and calibration data;
its adoption is growing as a part of web information collations in the detector
subsystems.
Future extensions include extra methods for tag viewing, and more flexible
output format (e.g. JSON, zipped format…)
CHEP'09 Prague
A RESTful Web Service... (Shaun Roe, Atlas)
11