Transcript Motivation for a Next-generation 3

Software Tools for Visualization of Multimodality Brain Data for Language Mapping
428.16
Andrew V. Poliakov, PhD1, Eider B. Moore1, David P. Corina, PhD2, James F. Brinkley, MD, PhD1
1Structural
Abstract
Our group has been developing software tools for processing, integrating and
visualizing multimodality data for language mapping. The primary data are collected in
preparation for and during neurosurgery for intractable epilepsy, and include structural
and functional MRI and surgical photographs of cortical stimulation maps (CSM).
Software developed by our group provides tools for reconstructing the cortical surface,
veins and arteries; mapping CSM sites onto the reconstructed surface; visualization and
analysis, including web-based visualization. These tools, however, operate only on
Unix platforms (SGI Irix and Linux), and are non-portable and difficult to maintain.
We present BrainJ3D -- a new software toolkit for processing and visualizing
language mapping and other brain mapping data. BrainJ3D is developed in
Java/Java3D, and is cross-platform (Windows, Linux and Solaris). The toolkit supports
mapping of CSM sites onto the cortical surface, and editing and labeling such maps. It
provides general purpose visualization tools, including image and slice viewers,
interactive visualization of 3D scenes showing cortex, veins, arteries and other surfaces,
and cut-away views of the brain. Functional data (e.g. fMRI, ERP, spectroscopy, etc.)
can be overlaid onto the structural volumes and surfaces. The data are organized in a
coherent and flexible workspace. BrainJ3D utilizes hardware-accelerated surface
rendering (3D graphics card and Java/Java3D are required). In addition to standalone
mode, the program can operate in a client/server mode, allowing remote visualization of
the data stored on a remote server, which performs the rendering and returns them as 2D images to a simple client program on a local computer. Such capabilities will permit
the development of spatial query engines and remote visualization of results obtained
from
distributed
brain
map
databases.
Support: Human Brain Project grant DC02310.
Informatics Group, Department of Biological Structure, 2Department of Psychology, University of Washington, Seattle, Washington USA
Web-based Information System for Language
Mapping
The experiment management component of our information system
allows users to store numeric and text-based data (e.g. patient
demographics, transcripts of experimental trials, etc.) in a relational
database, and edit and manage these data on the web2. The experiment
management component is also used to organize 2-D images (intraoperative photographs etc.).
The 3-D visualization component is used to visualize 3-D volumes
(e.g. structural and functional MRI.), 3-D surfaces derived from such
volumes (e.g. cortex, veins and arteries) and other 3-D data (CSM
maps). These data are loaded into a graphics server, rendered, then
saved as 2-D snapshots, which are sent to the client side. The key
features of this approach include:
•Client-server model
•Server-based data storage and rendering
•Simple web-based clients (e.g. Java or cgi-based)
BrainJ3D is our next generation tool for visualizing brain mapping data.
It builds on our previous experience with 3-D visualization of language
mapping data. It takes advantage of a modern programming environment
provided by Java and Java3D. It is designed to allow us to support the
same functionality yet overcome certain limitations of our current
system. BrainJ3D is designed to be:
• Flexible, making it easy to implement new features and functionality
• Cross-platform, since it is written in Java/Java3D
• Interoperable with several popular brain imaging software packages
Experiment Management Component
Background
As part of the Human Brain Project we are developing software tools
for processing, integrating and visualizing multimodality data for
language mapping. Prior to surgery for intractable epilepsy, structural
(MRI, MRV, MRA) and functional magnetic resonance (fMRI) images
are collected. During surgery Cortical Stimulation Mapping (CSM) is
used to locate areas of language on the cortex that need to be avoided1.
These cases present a unique opportunity for understanding language
function because the CSM data provide a “gold standard” against
which fMRI and other non-invasive methods can be compared.
To process these data we are developing a web-based information
system for managing and analyzing language mapping data, which
includes an experiment management component as well as a 3-D
visualization component.
BrainJ3D
3-D Visualization Component
Client
Client #1
CGI script
Client #2
Java Applet
Client #3
...
Current work
• Integration with Experiment Management component
• Beta-release in the near future
Will require extensive testing as well as improving usability
and refining graphics user interface.
• Improve interoperability
Interoperability proved to be a fast moving target, and is
often beyond our control
• Support for other data modalities
e.g. Spectroscopy, ERP source signal, Transcranial
Magnetic Stimulation
Web Interface
Internet
Server
snapshot
Graphics
Server
snapshot
Relational
Database
Features of BrainJ3D
Data
3-D Image
Volumes
3-D
Models
Stimulation
Sites
2-D
Images
Motivation for a Next-generation 3-D
Visualization Component
Our current production visualization techniques are based on Skandha4,
an in-house general purpose graphics toolkit3. Using this toolkit, we were
able to solve a number of problems, including reconstruction of the
cortical surface, veins and arteries, mapping language data onto these 3-D
models, integrating other data modalities such as functional MRI, and
web-based visualization4. However, as computer programming and
graphics continue to change and evolve, we face several challenges when
maintaining and further developing these 3-D applications.
These
challenges include:
• Relatively slow software rendering,
• Non-portable architecture (SGI Irix and Linux only),
• Large codebase that needs to be maintained and developed,
• Reliance on software tools that are becoming outdated or obsolete
Because of these considerations, we decided to explore alternate modern
techniques. BrainJ3D is the result of these explorations.
•Cut-away mode with functional data overlay
• Mapping and labelling 3D points on the cortex surface surfaces, e.g. CSM
• Surface-based 3D visualization
hardware accelerated, high performance
Slice viewer with functional overlay
• Data is organized in a customisable workspace tree
• Supports popular file types from other software packages (SPM, FSL etc.)
• Standalone and client/server mode
It leverages Java RMI (Remote Method Invocation) to support
client/server mode. The server uses off-screen rendering to generate 2D
snapshots that are then sent to the client.
• Open architecture and file formats
XML files used for all internal configurations and settings
Why Java
 Modern object-oriented language
 Numerous libraries and utilities
 Web-based and remote technologies are well supported.
Generalization to other applications
•
•
•
•
Acknowledgements
This work was funded by Human Brain Project grant DC02310, National
Institute of Deafness and Other Communication Disorders and National
Institute for Mental Health.
References
1.
2.
Why Java3D
Java3D is a high level 3-D programming language that provides
several advantages over lower level approaches such as OpenGL.
 Cross Platform ( Linux, Windows and Solaris)
 Supports hardware acceleration and scene optimisation, so it
performs as well as programs written in OpenGL.
 Supports hardware accelerated off-screen rendering.
 Mature software package that provides needed functionality
Teleradiology
Treatment planning
Experiment management
Online medical record
3.
4.
Ojemann GA. Mapping of neuropsychological language parameters at
surgery. Int Anesthesiol Clin 1986 Fall;24(3):115-31
R. M. Jakobovits and J. F. Brinkley, Managing medical research data
with a Web-interfacing repository manager, Proceedings, AMIA Fall
Symposium, Nashville, pp. 454-458, 1997.
Modayur BR, Prothero J, Ojemann, G, Maravilla K, Brinkley JF.
Visualization-based mapping of language function in the brain.
Neuroimage, 1997: 6: 245-258.
A. V. Poliakov, K. P. Hinshaw, C. Rosse and J. F. Brinkley, Integration
and Visualization of Multimodality Brain Data for Language Mapping,
Proceedings, AMIA Fall Symposium Washington, D.C., pp. 349-53,
1999.