Transcript ICAT Integration at ISIS

ICAT- Experience and
activities at ISIS
Tom Griffin, STFC ISIS Facility
4th PaN-Data & CRISP
Harmonisation Meeting
XFEL/DESY
January 2013
[email protected]
Summary
• Data flows
– Business data - Other systems
– Experiment data
• Authentication
• Authorisation
• Data policy
• Problems
• Future plans
IBS-ICAT Link
Getting data into ICAT
• Pre-experiment
• C#
• Creates empty investigations and
datasets ready for data
• Must update ICAT after changes
Getting data into ICAT
• During the experiment
• ‘ICATIngest’
• Subscribes to file system events
ReadDirectoryChangesW()
• Invokes ‘WriteRaw’ and ‘NxIngest’
• Caches groups of files for performance
• Extracts ‘science’ metadata
Data files
• ISIS RAW files
– Custom binary format + custom reader
– Outputs ICAT friendly XML
• Nexus
– Not really application definition compliant
– Use mapping files to convert nexus to
ICAT friendly XML
Getting Data out - Web Access
• Main point of access for ISIS data
• HTTP file serving – pluggable architecture
• Zipped or non-zipped
TopCAT – basic web interface
Getting Data out - Mantid
• More convenient for users
• Must work on and off site
• Derived data upload
• Potential for provenance tracking
Authentication
• ICAT 4.x plugin architecture
• Web services to ISIS user database
• 15 minutes to write
• Single sign on version too
• Much easier to reuse existing auth
systems
Authentication -Umbrella
• For specific permissions will require
local account stub
• Facilities may require local registration
for ‘public’ data
• At ISIS will all go through local user
database
– Single authentication provider in ICAT
Authorisation
• My implementation not complete
– Increasing test coverage….
• So far just under 100 rules
• Not everything is possible
– Limitations around devolving control
Data Policy
• ICAT implements a data policy in rules
• Who can access what data when
• ISIS policy published and implemented
• ICAT is flexible
• http://www.isis.stfc.ac.uk/useroffice/data-policy11204.html
• tiny.cc/isisdp
DOIs – data citation
• When to issue?
Problems
• Tracking reference numbers numbers
• Linking calibration data
• ‘Fixing’ data that has been wrongly
assigned
• Migration from 3.3 -> 4.2.2
Future
• Complete migration to 4.2.2 / 4.3
• Replace current data server with IDS
• Continue integration with Mantid
• Improve and share tools (fixing data)
• Federation of catalogues
– Not just ICATs
– OAI-PMH etc
Summary
• 3.3 in use for several years
• Upgrading to 4.2.2
• Data upload key to future success
• Easy to link to user database and
umbrella
• Hardest part is integration with
business and experiment data
Questions...