Transcript EVE for NSF
On-Board Mining in the Sensor Web NSF Next Generation Data Mining November 2, 2002 Dr. Rahul Ramachandran [email protected] For Steve Tanner and the EVE Team [email protected] Information Technology and Systems Center University of Alabama in Huntsville 256.824.5157 www.itsc.uah.edu Presentation Outline ITSC/UAH Data Mining Overview Onboard Mining (EVE) – – – – – – – Project Overview System Design Overview The EVE Editor The On-board Components EVE Operations Example Plans Current and Future Directions ITSC and Scientific Data Mining Research primarily focused on – – Developing Mining Environments for Scientific Data Scientific Data Mining Applications Developed Algorithm Development and Mining (ADaM) System – – – – NASA research grant The system provides knowledge discovery, feature detection and content-based searching for data values, as well as for metadata. It contains over 120 different operations that can be performed on the input data stream. Operations vary from specialized atmospheric science dataset specific algorithms to different digital image processing techniques, processing modules for automatic pattern recognition, machine perception, neural networks and genetic algorithms. ADaM Engine Architecture Results Translated Data Data Preprocessed Data Patterns/ Models Processing Input Preprocessing Analysis Output HDF HDF-EOS GIF PIP-2 SSM/I Pathfinder SSM/I TDR SSM/I NESDIS Lvl 1B SSM/I MSFC Brightness Temp US Rain Landsat ASCII Grass Vectors (ASCII Text) Selection and Sampling Subsetting Subsampling Select by Value Coincidence Search Grid Manipulation Grid Creation Bin Aggregate Bin Select Grid Aggregate Grid Select Find Holes Image Processing Cropping Inversion Thresholding Others... Clustering K Means Isodata Maximum Pattern Recognition Bayes Classifier Min. Dist. Classifier Image Analysis Boundary Detection Cooccurrence Matrix Dilation and Erosion Histogram Operations Polygon Circumscript Spatial Filtering Texture Operations Genetic Algorithms Neural Networks Others... GIF Images HDF-EOS HDF Raster Images HDF SDS Polygons (ASCII, DXF) SSM/I MSFC Brightness Temp TIFF Images Others... Intergraph Raster Others... ADaM: Mining Environment Classification Based on Texture Features and Edge Density Science Rationale: Man-made changes to land use cause changes in weather patterns, especially cumulus clouds Comparison between mining techniques based on – Accuracy of detection – Amount of time required to classify Cumulus cloud fields have a very characteristic texture signature in the GOES visible imagery Automated Data Analysis for Boundary Detection and Quantification Analysis of polar cap auroras in large volumes of spacecraft UV images Science rationale: – Indicators to predict geomagnetic storm Damage satellites Disrupt radio connections Developing different mining algorithms to detect and quantify polar cap boundary Polar Cap Boundary Detecting Mesocylone Signatures Detecting mesocyclone signatures from Radar data Mesocyclone is an indicator of Tornadic activity Developing an algorithm based on wind velocity shear signatures – Improve accuracy and reduce false alarm rates “…drowning in data but starving for knowledge” – John Naisbett Data glut affects business, medicine, military, science How do we leverage data to make BETTER decisions??? Information User Community Many On-board Platforms Landsat 7 Terra Aqua Aura ICEsat QuikSCAT Jason-1 Systematic Missions - Observation of Key Earth System Interactions SRTM GRACE QuickTOMS Cloudsat PICASSO GIFTS EO-1 Exploratory - Exploration of Specific Earth System Processes and Parameters and Demonstration of Technologies Many Types of Sensor Data Multispectral Lidar Hyperspectral Synthetic Aperture Radar Thermal Scatterometer A Reconfigurable Web of Interacting Sensors Communications Weather Satellite Constellations Military Ground Network Ground Network Ground Network Project Overview - EVE Requirements • Prototype a processing framework for the onboard satellite environment. • Provide specific capabilities within the framework • • – Data Mining – Classification – Feature Extraction Support research applications – Multi-sensor fusion – Intelligent sensor control – Real-time customized data products Create a ground-based testbed EVE Functional Components EVE Software Architecture Processing Plan Editor On-board Configuration Library Input Analysis Output Modules Modules Modules Inter Process Decision Communcation Support Sensor Model Library Sensor Data Simulations Passive Microwave IR System Specific Modifications Testbed of On-board Systems Control Systems RT Linux Flight Linux etc. XML Based Processing Plans etc. Testbed Control Ground Control EVE Functional Flow: Getting a plan on-board EVE On-board System 3. The on-board system creates the carts for execution Editor 2. The ground station sends the plan on to the appropriate onboard system Ground Station 1. The user edits a processing plan with SMAC and sends an XML description to the ground station Design Overview: What is a Plan? A Processing Plan: Specifies a set of operations and the data stream connections between them Design Overview: What is a Cart? Holds the operations of a plan that will be executed as a single real-time unit Has knowledge of resource limitations on a platform and resource usage of operations Design Overview: Processing Plan Editor Web-Based Editor – – – – – Accessible from everywhere No need to distribute new code for new versions No client installations Easy to build Flexible (drag and drop) Drag and Drop Interface • Developed during ’02 • Java based • Web accessible • Extensible • Much reuse of existing code • Will be incorporated into other projects Close up of Major Editor Features Editing tools Cart building tools Operations Estimated Resource Information Actual On-board Resource Usage Actual On-board Cart Information Design: EVE On-board System Non RT DownlinkComm RT Metrics Module Schedule Conductor Coordinator Schedule Plan Manager Cart Factory Operations Storage Cart Cart Cart Cart Cart Cart Plan Manager System Monitor EVE On-board System Coordinator: – Plan Manager: – Push Carts into the RT environment for execution Conductor: – Start a plan manager for each uploaded plan Schedule and execute Carts and events Cart Factory: – Create Carts based upon the on-board resources and the uploaded plans, and using modules stored in the Operation Storage Design: EVE On-board System The Metrics Module The Coordinator takes The Plan Manager collects resource usage the plan, and creates a parses the plan, and information and sends this Plan Manager process contacts the Cart to the ground station Planplan Manager Factory to create a for thatThe specific then pushes each Cart Cart for each one into the real-time described in the plan kernelModule space and Metrics Schedule inserts schedule information about The Cart Factory Conductor when the Carts should creates an executable Coordinator DownlinkComm Schedule be invoked module for each Cart, Downlink Communications receives a new plan from the ground station Non RT including all described operations and their I/O information Cart Factory Operations Storage RT The Conductor manages both a temporal scheduler and Cart an event Cart scheduler. When a specified time executes orEach eventCart occurs, the as an independent Conductor invokes the process, and can signal appropriate Cart for Cart events by Cart Cart execution sending messages to the Conductor System Monitor The System Monitor watches both real-time Plan Manager and non-real-time system functions, Cart and This information sends status the comes fromtothe ground station PlanOperations ManagerStorage Operations in EVE Each operation is a reusable component capable of functioning in a constrained real-time environment Operation metadata (parameters, input, and output specifications) are specified in the metadata library Plan description files document what and how operations are linked together for a complete plan Operations Currently Available Data I/O Format Conversion Image Processing – – – – Convolve Resample Rotate Etc. Complex number operations (e.g. fft) Signal generator operations Network operations Example Plan: Real–Time Edge Detection Plan branching and recombining Multiple carts, real-time and non-real-time vidop Plan 1 Cart 1 (NRT) Cart 3 (NRT) Cart 2 (RT) user_to_rtf Get sensor data image_to_disk user_from_rtf Store results convolve (vert) from_rtf to_rtf split add Real-time threshold Branch Find edges convolve (horz) Recombine Example Plan: Real–Time Edge Detection • Significant speed improvement - 5+ images per second • Can be used with many sensors • Edge Detection output is used by other processes • Can be the basis for further feature extraction plans Example Plan: Threshold events in AMSU-A Streaming Data Event triggering between plans Channel select Thresholding from_swath Plan 1 AMSUA_detect Get sensor data Save results and signal event save_to_raw_file Plan 2 Read_raw_data convert_to_image Activate on event signal save_image_data Example Plan: Threshold events in AMSU-A Streaming Data EVE Current Issues and Future Enhancements Advanced on-board coordination – Shared memory – Broadcasting from On-Board Event Flagging on Multiple Platforms Enhanced System Tools – Detection of Race Conditions – Monitor operation I/O Year 3 Activities Publish Processing Plan Syntax for use by others Provide public access to web based user interface and beta testing of the EVE system framework Implement and add new operations to the system Incorporate additional operations from other sources Increase data input components based upon known and expected sensors Incorporate intelligent scheduling Port to cluster environment for sensor web prototyping Possibly incorporate EVE into a flight of opportunity (OMNI, UAV, Flight Linux, etc.) Additional Information Website: – eve.itsc.uah.edu Contact Person: – Steve Tanner – [email protected] – (256)-824-6868