Transcript JRA4 Overview and plans M. Haefele, E. Sonnendrücker

JRA4 Overview and plans
M. Haefele, E. Sonnendrücker
Euforia kick-off meeting
22 January 2008
Gothenburg
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Outline

Presentation of JRA4 tasks

Task 1: Existing visualization tools

Task 2: Visualization & Kepler platform

Task 3: Lossy data compression format

Task 4: 4D and 5D visualization
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Outline

Presentation of JRA4 tasks

Task 1: Existing visualization tools

Task 2: Visualization & Kepler platform

Task 3: Lossy data compression format

Task 4: 4D and 5D visualization
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Purpose
Provide visualization support for project

Unified visualization framework for all codes to be
integrated


Post-processing visualization
Integration of visualization tools in Kepler workflow

Monitoring

Steering
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Outline

Presentation of JRA4 tasks

Task 1: Existing visualization tools

Task 2: Visualization & Kepler platform

Task 3: Lossy data compression format

Task 4: 4D and 5D visualization
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Python Matplotlib

Pure Python library based on numpy

Can be used in


python scripts

web application server that can call python
Syntax close to Matlab visualization tools
=> Replace Matlab visualization for free !

Only 2D plots are available

3D plots will use the VTK library
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Python matplotlib
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
VTK : Visualization Toolkit

Open source C++ library

Community of more than 1000 developers

Architecture managed by a private company : Kitware

Exists and evolves since 1998
=> powerful and stable tool
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
ParaView

Software based on parallel VTK

Can be seen as a GUI for VTK

The user designs the VTK pipeline by himself

The user must know the different pieces of the pipeline

Great flexibility

Parallelization of both processing and rendering

Plugin system based on the use of new VTK classes

Open source software designed by Los Alamos Lab
and Kitware
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Paraview
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
VisIt

Software based on parallel VTK

Tool to visualize different kind of data

The VTK pipeline is hidden from the user




More user friendly

Less flexibility
Parallelization of processing (remote processing)
The plugin system proposes an intermediate layer on
top of VTK
Open source software designed by Lawrence
Livermore Lab
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
VisIt
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Task1: Work plan

List visualization requirements of codes in platform

Develop data (MDS+, UAL) import plugin for



Matplotlib

VisIt
Develop sample visualizations for typical data in
Matplotlib and VisIt
Write tutorials
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Outline

Presentation of JRA4 tasks

Task 1: Existing visualization tools

Task 2: Visualization & Kepler platform

Task 3: Lossy data compression format

Task 4: 4D and 5D visualization
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Monitoring tools

Visualization during the simulation

Check if the simulation is still OK

1D, 2D (light) dataset are enough

Visualization can be performed on the gateway

Visualization must be integrated to Kepler
Which quantities are reliable for code monitoring ?
Which tools have to be integrated into kepler ?
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Interpretation tools

Visualization after the simulation

Huge dataset generated by distributed simulations
1. Copy every dataset to a central visualization resource

Requires big storage and network bandwidth

Requires a single visualization resource
2. Leave dataset on the remote simulation resource

Requires no extra storage or network bandwidth

Requires remote visualization resource
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Visualization strategy
In both case, we suggest to use an existing tool based
on VTK: VisIt (or ParaView)
Example of CSCS (J. Favre)


Shared remote visualization resource strategy
Data reading, processing and rendering are
performed at the computer center

Only resulting pictures go through the network

Interactive visualization performed with ParaView
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Remaining questions


Build a link between visualization tool and Kepler in
order to keep track of generated pictures ?
What are physicists looking for in the data ?
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Outline

Presentation of JRA4 tasks

Task 1: Existing visualization tools

Task 2: Visualization & Kepler platform

Task 3: Lossy data compression format

Task 4: 4D and 5D visualization
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Large dataset management
Size S of a uniformly discretized function in Bytes
d
S= N F

d=2, N=128 => 128 KB

d=3, N=128 => 16 MB

d=4, N=128 => 2 GB

d=4, N=256 => 32 GB

d=6, N=128 => 32 TB
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Large dataset management
Strategy : Use data compression to reduce the size of
the numerical problem

Hierarchical bases of finite elements

Local compression algorithm

Compression integrated into the parallel simulation

Efficient algorithm for slice reconstruction
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Results for a 32 GB function
Particles beam result
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Results for a 32 GB function
Laser-plasma interaction result
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Present status


Interactive exploration of 4D data with 2D slices
Integration of the compression process into the
parallel simulation
=> Allow the export of the distribution function
=> Without damaging the speedup of the simulation

Data reduction with a factor between 8 and 150
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Task 3: Work plan

Enhance compression algorithm

Parallelization of slice reconstruction algorithm

Integration of these tools into VisIt, Kepler...

Handle 5D dataset
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Outline

Presentation of JRA4 tasks

Task 1: Existing visualization tools

Task 2: Visualization & Kepler platform

Task 3: Lossy data compression format

Task 4: 4D and 5D visualization
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Specific difficulties

Visualization of data coming from kinetic simulations

Particle distribution function lives in phase space

The full model requires 6D + time

Objective : visualize functions f: R4  R

How to visualize a 4D function ?

How to represent the large dataset (2-32GB) for an
interactive visualization ?
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Multidimensional visualization

4D distribution function f(r, v//)

Dimension reduction

Representation of a set of slices/projections
Data coming from Gysela5D (CEA Cadarache)
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804
Demonstration
Demo time !
EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804