Transcript CCP4i developments

Developments with CCP4i
& the Database Handler
Peter Briggs
Outline
• CCP4i developments
• DbCCP4i & database developments
• Where we are now
• Still to do & future plans
CCP4i developments: summary
New tasks: buccaneer, pointless, rapper, pisa, xia2,
mrbump, getax, phaser (experimental phasing)
New utilities: idiffdisp, baubles
Integration of iMosflm and Ctruncate
Very many other bug fixes and improvements
• see newsletter article from January, “New
Developments in CCP4i” (#47):
http://www.ccp4.ac.uk/newsletters/newsletter47/articles/ccp4i.html
Tasklist Presentation
Based on suggestions from Charlie Bond 2 years ago
Intended to provide framework for reorganising
presentation of tasklists
• now implemented & released in 6.0.99 series
Ongoing work doing the actual reorganisation
• received input from Eleanor, Phil, others…
Plugins
Enables you to launch external applications from the
results of a job
• initially applied to launching Coot and CCP4mg after
Refmac 5 jobs
• can be extended programmatically to other tasks and
applications
Other usability additions
Improvements to interactions with job list
• right-click menu, double-click to see logfile, …
Colourisation
• feature currently available in 6.0.2 but now more
easily accessible
Loggraph supports XMGR files from SCALA
• useful for Windows
Let’s see some of these in action…
DbCCP4i & the Database
• DbCCP4i is a server process that takes care of
managing the job database
• Allows many programs to communicate with the job
database at once (not just CCP4i)
• Has an associated visualiser (“DbViewer”)
• Project includes applications to interact with job history
(starKey, reaper…)
Background to DbCCP4i work
• Funded by EU Framework 6 project “BIOXHIT” – due
to end June 2008
• Employed 1 full time developer (Wendy Yang) plus 1 at
50% effort (me)
• Described in various CCP4 newsletter articles:
# 46: http://www.ccp4.ac.uk/newsletters/newsletter46/articles/project-tracking.html
# 45: http://www.ccp4.ac.uk/newsletters/newsletter45/articles/ccp4_bioxhit.html
System Architecture
CCP4i 2.0
XIA
Tcl Client API
Python Client API
dbviewer
- visualiser
Tcl Client API
StarKey
- XML generator
Python Client API
dbccp4i
- database handler
CCP4i db SQLite
…other client
applications
(e.g. consoles,…)
CCP4i 1.4
current version
CCP4i projects
- database.def files
SQL knowledge
database
10
Significant activities since last March
• Integration into MrBUMP and public release (July 2007)
• Expansion of the job database to support “subjobs”
(August 2007)
• Initial implementation of demonstration SQLite
crystallographic knowledge base (August 2007)
• Wendy left (September 2007)
• Integration into CCP4i for 6.1 (ongoing, initial phase
completed March 2008 & released with 6.0.99c)
Integration with MrBUMP
• Work done by Ronan and Wendy
• Uses a customised version of DbCCP4i in parallel with
user’s existing projects
• Creates new CCP4i project database for each run
• Steps in MrBUMP are stored as jobs
• DbViewer allows progress to be monitored and reviewed
Integration with CCP4i
CCP4i 2.0 (in 6.0.99c) uses DbCCP4i as database
backend by default
• seems to work but still some issues to resolve
• update of job list is slow, especially for large projects
& using colourisation
• not tested fully on Windows (expect some problems)
• compatibility issues for MrBUMP
• old “direct access” mode still available as a workaround
• Fully expect to resolve issues before official 6.1 release
Where we are now – to-do for 6.1
• Complete work on tasklist reorganisation – including
updated documentation
• Complete testing and integration of database handler
into CCP4i
• fix know issues with speed of job display
• properly integrate DbViewer into CCP4i
• address compatibility issues for MrBUMP
• Various bug fixes required and minor features still to be
added
• ~50 issues currently logged in bug tracker
Looking slightly further ahead
Bioxhit deliverables (for June 2008):
• “reaper” program to analyse job database and extract
harvesting information (deliverable 5.2.17)
• Make demonstration knowledge base accessible from
DbViewer (deliverable 5.2.16)
MrBUMP and database:
• update MrBUMP to use subjobs mechanism to record
individual steps in each run
Looking ahead even further
Wider adoption of database by other applications or
pipelines?
Expansion of database content?
• Store more data for each job, better treatment of
history links, application-specific data/tags …
Work on SQLite database to store project-based data?
Improve the visualisation options
• Data-centric rather than process-centric view, improve
richness of view …
Acknowledgements
CCP4i
• Originally developed by Liz Potterton
• Current development by DL staff (Peter Briggs, Charles
Ballard, Ronan Keegan, Francois Remacle, Norman
Stein & Martyn Winn)
• Contributions also from Kevin Cowtan, Phil Evans, Airlie
McCoy & Randy Read, Nick Furnham
DbCCP4i
• Wanjuan Yang, Peter Briggs and Ronan Keegan
And Martyn for lending me his laptop for the demos…