Transcript KDD for Science Data Analysis Issues and Examples

KDD for Science Data Analysis
Issues and Examples
Contents



Introduction
Data Considerations
Brief Case Studies







Sky Survey Cataloging
Finding Volcanoes on Venus
Biosequence Databases
Earth Geophysics
Atmospheric Science
Issues and Challenges
Conclusion
Data Considerations





Image Data
Time-series and sequence data
Numerical Vs Categorical values
Structured and sparse data
Reliability of Data
Brief Case Studies





Sky Survey Cataloging
Finding Volcanoes on Venus
Earth Geophysics
Atmospheric Science
Biosequence Databases
Sky Survey Cataloging
The survey consists of 3 terabytes of image data containing an
estimated 2 billion sky objects
The basic problem is to generate a survey catalog which records the
attributes of each object along with its class: star or galaxy
To achieve this scientists developed the SKICAT system
Reasons why SKICAT was successful




The astronomers solved the feature extraction problem
Data mining methods contributed to solving difficult
classification problems
Manual approaches were simply not feasible.
Astronomers needed an automated classifier to make
the most out of the data
Decision tree methods proved to be an effective tool for
finding the important dimensions for this problem
Finding Volcanoes on Venus



Data collected by Magellan spacecraft
The first pass of Venus using the left looking radar
resulted in 30,000 1000 x 1000 pixel images
To help geologists analyze this data set, the JPL
Adaptive Recognition Tool (JARtool) was developed
Motivation for using Data mining methods



Scientists did not know much about image processing or
about the SAR properties. Hence they could easily label
images but not design recognizers
There was little variation in illumination and orientation of
objects of interest. Hence mapping from pixel space to
feature space can be performed automatically
Geologists did not have any other easy means for
finding the small volcanoes, hence they were motivated
to cooperate by providing training data and other help
Earth Geophysics
Two images taken before and after an earthquake and
by repeatedly registering different local regions of the
two images, it is possible to infer the direction and
magnitude of ground motion due to the earthquake.
Example of a geoscientific data mining system is
Quakefinder which automatically detects and measures
tectonic activity in the earths crust by examination of
Satellite data
Atmospheric Science



Data mining tool used is called CONQUEST
Parallel testbeds were employed by Conquest to
enable rapid extraction of spatio-temporal
features for content based access.
Some of the goals of the this tool is the
development of “learning” algorithms which look
for novel patterns, event clusters etc.
Retrieved Sea Level Pressure Fields
Biosequence Databases


The largest DNA database is GENBANK with a database
of about 400 million letters of DNA from a variety of
organisms
The pressing data mining tasks for biosequence are
Find genes in the DNA sequences of various
organisms.
Some of the gene finding programs such as GRAIL,
GeneID, GeneParser, Genie use neural nets and
other AI or statistical methods
Issues and Challenges






Feature Extraction
Minority Classes
High degree of Confidence
Data mining task
Relevant domain Knowledge
Scalable machines and Algorithms
Conclusions
KDD applications in science may in general be easier
than applications in business, finance, or other areas.
This is due to the fact that science end users typically
know the data in intimate detail.