Transcript data model
Database Issues in Nutritional Genomics Tony Travis & Peter Gray Rowett Research Institute & University of Aberdeen Jan 2005 Nu GO Un Oslo Rowett Un. Ulster Un Newcastle Un Lund Trinity Un Cork EBI DiFE IFR Rivm Rikilt TNO Un Reading Un Wageningen Un Maastricht Un Krakow Un Munich NuGO Inserm Marseille Un Balearic Illes Un Florence Utopian view • • • • Share data freely Everyone benefits Ideas develop Science prospers Big pharma disagree! • Sell data commercially • Big pharma benefits • Ideas are exploited • Science is a business Scientists are confused… • Intellectual freedom? – Curiosity driven science – Poor funding • Intellectual property? – Commercially driven science – Good funding Preserving intellectual property • Autonomy – Scientist or institution control who their data is disclosed to – Control data sharing by collaborators who share their IP – Needs federated solution • Security – Prevent unauthorised access to data – Prevent unauthorised use of data – Maintain integrity and provenance of data Typical NutriGenomics Use Case • Example of pragmatic solution – DNA microarray work at RRI • Autonomy – Data held locally on PC spreadsheets – Completely under control of investigator • Collaborators – Each create spreadsheet of local results – All collaborators exchange spreadsheets Spreadsheet microarray data Distribution of one spreadsheet A B D C Exchange of all spreadsheets A B D C Manual replication of database • Advantages – Simple peer-to-peer data transfer via email – Each collaborator has entire database locally – Local analysis tools are readily available – Complete control of IP within collaboration • Disadvantages – N(N-1) solution – Does not scale well – Each collaborator must merge data into local database replica – No control over data integrity or provenance Spreadsheet Replicated Data Model • Distributed – Data originates at each collaborator’s site • Replicated – Copy of the entire database at each site • Manually updated – Data and corrections are pushed from each collaborator to all others via email of Excel spreadsheets containing expression data which is merged into a single spreadsheet Local analysis tools: maxd • Microarray Bioinformatics Group University of Manchester (UK) • Java-based • maxdView – Visualise and analyse gene expression data. • maxdLoad2 – Store and curate gene expression data to MIAME standards • Export in MAGE/ML format for submission to ArrayExpress. Import spreadsheet data into maxd Analyse expression profiles • 10,000 genes • Four experiments by one collaborator • Normalised • Clustered • Comparison of gene expression profiles between experiments Upgrade spreadsheet solution • MaxdLoad2 – Replace spreadsheets – Use MIAME standard – JDBC compliant interface – SQL92 (MySQL, Postgres) Candidate Mediator middleware • Maxd – Designed for use with single database • P/FDM – Integration of heterogeneous data sources – Federated union/join of relations • Biomart – MartShell scripting language – Federate database instances Example federated DB MartShell • Command line (text mode) user Interface to BioMart that can be used by programs • Mart Query Language (MQL) • Queries can be executed in ‘batch’ mode using stored procedures in MQL scripts BioArray Software Environment • BASE is a comprehensive database server to manage massive amounts of data generated by microarray analysis • Lund University +Oklahoma University • Data can be analysed using a web-based GUI to server-side PHP scripts or data can be extracted from the BASE database by applications such as Genespring Querying a Federated DB There are two kinds of distributed query that you can send out to the federation: • Federated Join - like adding extra columns with cross-referenced information on the same object or related objects. • Federated Union – like adding extra rows with the same column headings – the same kinds of experiments but done at different sites. Comparing expression profiles (e.g.looking for co-regulation) Lab 1 Rat 1 Rat 2 Rat 3 Lab 2 Rat 4 Rat 5 Rat 6 1.032028 1.320651 5.806003 1.389428 3.625239 … 1.482212 1.157023 6.857593 1.678806 2.142907 … 1.291634 1.06932 1.061052 2.083518 1.146157 … Lab 3 Rat 7 Rat 8 Rat... 1.808716 2.388491 1.47649 0.412969 2.225646 … 1.217205 1.114725 1.218257 2.560339 2.202825 … … … … … … Lab... GeneB 1.152023 1.746107 1.260019 GeneC 6.957593 4.191758 1.079844 GeneD 1.678806 2.666562 1.651717 GeneE 2.172907 3.184208 1.359549 Gene… … … … GeneA 1.488212 0.374015 0.642141 Conditions for making a Federated DB work Needs Common Ontology for data of same type. BEWARE measurements made in different units, or using a very different exptl. procedure, or qualitative measurements such as "large".."medium" Conditions for making a Federated DB work Need Common Unique Identifiers : if no property allows you to tell that one entity instance is the same as another then integration is UNSAFE! (Note - it might be OK for say 95 percent of identifiers...) Conditions for making a Federated DB work Mechanisation of Value mapping : • if data values can only be compared or made compatible with others using the judgement of an experienced scientist, then one must use a Warehouse (as in early PDB), otherwise • if you can mechanise it using rules or equations then it can be done by a view, • or by a mediator accessing the Federation Conditions for making a Federated DB work Need Standard Interchange Formats : • Formats such as MMCIF helped reduce human intervention in PDB. The widely used MIAME format may do the same for MicroArray Data. • However such data is much harder to integrate as it may be measured under different conditions with different technology. Difficulties of Federated Approach • Reliability - Sites must be available continuously, and not crash too often; • Support costs - must be proof against Virus attacks, etc., and have people able to bring them back up again promptly Difficulties of Federated Approach • Compatibility - must provide a common interface - may be able to share development of some downloadable server software (like Java WebStart), responding to SOAP protocol messages and commands, config-urable through web forms that keeps logs of errors. Difficulties of Federated Approach • Performance: Warehouses will provide better performance for data mining programs and others programs with a high hit rate. • Federated systems compete well on more focused queries which allow the use of indexes in remote systems. Having it Both Ways: • A Federated Solution can include some sites that are adopting Warehouse technology to collect and vet large volumes of data of a particular kind. • The NUGO data model and ontologies are bound to change a lot in ways we cannot forsee. Thus it makes sense to be flexible to start, allowing site autonomy, and to delay committing to large warehouses until we understand more about the data model and IPR issues. Discovering the Model Birney & Clamp (2004) say – "the true biological interpretation of data stored in a database will change over time, and discovering new relationships between aspects of the data is an important part of the motivation for storing it..” Conclusion (1) Spreadsheets • Spreadsheets are easy and popular • Integrating Spreadsheets manually is time wasting and can easily lead to errors and wrong conclusions • Scientists need the discipline of a shared Data Model and the automation of data transfer and conversion, usually provided by a Mediator Conclusion (2) – Shared Data Model • Agreement on a shared Ontology is mainly a problem of agreeing Standards for names, units, and specialised types. • Agreeing a shared Data Model is more subtle. It may need experimentation in advance of a standard. • The Data Model, based on Entity-Relationship Model with SubTypes, must be able to evolve not fixed in stone, coping with the unforseen. Conclusion (2) – Shared Data Model • The Data Model must be at Conceptual Level independent of Storage Technique - arrays, ASN-1, XML, tables etc... Otherwise agreeing a Shared Model becomes too hard! • The Data Model must provide External Views both to restrict access and to provide a consistent API to External Applications; these may be Spreadsheets or Statistical Packages or MaxD or Genespring etc... Conclusion (3) – Federating Microarray Data • Usually, a federation is based on a federated Join, through common identifiers, because irrelevant joins can be left out, to speed up the query. • Federated Joins suit integrating other types of data with Microarray data, e.g. physiological, epidemiological data • This is easily done, on the fly; it allows us to evolve the data model and experiment with it without making changes to a centralised warehouse. Once the data model is more stable, parts of it can be stored in warehouse. Conclusion (3) – Federating Microarray Data • Queries that want to compare Gene Expression Profiles across many Experiments need a federated Union of data from different experimenters. • Comparing one profile against those from many experimental sites could be done in parallel. Trusted methods could work with an encrypted profile to keep it confidential. Conclusion (4) – IPR and Federation • Scientists want to retain their autonomy and right to recognised authorship of the data, otherwise they may not share it! • If Database Right (EU proposal) becomes established, scientists may wish to keep data in their own DB in order to take advantage of it. Thus we may need to make more use of federated techniques to bring such data together. • Revenue-Raising Potential may become important (iTunes for example).