Transcript I/O and Vizualization - Flash

The Center for Astrophysical Thermonuclear Flashes
What did Flash do?
(Flash2 I/O and Visualization)
By Brad Gallagher
An Advanced Simulation & Computing (ASC)
Academic Strategic Alliances Program (ASAP) Center
at The University of Chicago
Checkpoint Files in Flash2
Checkpoint Files:
1. Tree Data
2. Grid Data
3. Physical Variables Data
(Paramesh unk)
Stored unk:
unk(nvar,nxb,nyb,nzb,total_blocks)
Tree Data:
(mostly Paramesh data structures, some data
structures derived by Flash)
Grid Data:
(Paramesh data structures that define the AMR grid,
tree data used to properly interpret grid data)
Physical Variable Data (unknowns):
(5d array containing solution data, only the interior
cells are written. There can be up to 24 variables per
nxb by nyb by nzb by total_blocks)
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Checkpoint Files in Flash2 Cont..
 Header data:


Misc header data, setup call, build time, geometry name, Flash modules used etc..
(many more)
Simulation parameters (total_blocks, simulation time, dt, redshift, nxb, nyb,nzb , number
of steps)
 Timer Data:

Fully nested timer data for various modules and subroutines
 Particle Data:

Full snapshot of particle tracers data
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Restarting a Flash simulation from Checkpoint
 How to do it:



Set “restart” to true in flash.par
Provide the correct checkpoint number (cpnumber) to start from in flash.par
Be sure to provide what will be the right subsequent plotfile number (ptnumber)
 What it does:




First the simulation parameters are read
Total_blocks from simulation parameters is used along with the number of processors to calculate the
approximate number of blocks to put on each processor
Global offsets are again calculated by determining first the number of blocks on processor i, then the
number of blocks on processor i+1 is the number of blocks on i plus the number of blocks left of
processor i
Once all the data is read, the gid array (global block ids) is used to re-build the tree structure
(paramesh’s neigh, child and parent), then guardcells can be exchanged and you are ready to take
another timestep.
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Flash2
Plotfiles
Plotfiles:
 Contain same tree and grid data as checkpoints
 Contain only a subset of the unknown data (as defined by flash.par)
 Reals are stored in single precision (both unknowns and real grid data)
 Data can be cell-centered or corner interpolated
 Plotfiles are dumped based on tplot parameter as specified in flash.par
 In general they are dumped more frequently than checkpoints
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Particle Plot Files in Flash2
 Outputted like plotfiles, if time_elapsed since last output > ptplot
 Can also be outputted every timestep with a filter
subset of particles are selected for dumping at each timestep based on user definition
 Flexible output style control



Single or multiple files or multiple files after N particles have been dumped
Particle data can be written as a structure of mixed types or as arrays of the same type
Underlying binary I/O libraries restrict the above (pnetcdf must write arrays of the same type)
 Runtime parameters that control particleplotfile output





pptype : LOGICAL : if true output to a new file for each particleplot call, if false write to the same file
nppart : INTEGER: if pptype is false still create a new file after nppart particles have been dumped
pfilter : LOGICAL: if true output particleplotfile every timestep (user defined filter in place), if false dump
based on ptplot
ptplot : REAL : time elapsed between particleplotfile dumps
pstruct : LOGICAL: if true dump particle data as mixed data type structure, if false split structure into arrays of
the same type
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Details Cont.. (Tree Data)
Tree Data: oct-tree in 3d quad tree in 2d
Tree Data:
lrefine: The level of refinement of a block

Dimension: (ndim,total_blocks)
nodetype: describes the relationship of a
block to others




1 : Block is a leaf node (no children)
2: Block is a parent (at least 1 leaf node)
3: Block has children (no leaf nodes)
Dimension: (ndim,total_blocks)
gid: global block id array




Derived by Flash using child,nfaces, neigh and
parent
Used upon restart to rebuild parent data structure
Parent data structure needed to do guardcell
exchange after restart
Dimension: (nfaces+1+nchild,total_blocks)
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Details Cont.. (Grid Data)
Grid Data:
 coordinate data: xyz coordinates of the
center of a block

Dimension(ndim,total_blocks)
size data: The xyz extents of a block

Dimension(ndim,total_blocks)
bounding box data: the upper and
lower edge of a block along the jth
coordinate axis

Dimension(2,ndim,total_blocks)
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Flash2 flash.par (sedov explosion)
#
Runtime parameters for the Sedov explosion problem.
p_ambient
= 1.E-5
rho_ambient
= 1.
exp_energy
= 1.
r_init
= 0.013671875
xctr
= 0.5
yctr
= 0.5
zctr
= 0.5
gamma
= 1.4
geometry = "cartesian"
xmin
= 0.
xmax
= 1.
ymin
= 0.
ymax
= 1.
zmin
= 0.
zmax
= 1.
xl_boundary_type = "outflow"
xr_boundary_type = "outflow"
yl_boundary_type = "outflow"
yr_boundary_type = "outflow"
zl_boundary_type = "outflow"
zr_boundary_type = "outflow"
cfl
= 0.8
lrefine_max
= 6
basenm
= "sedov_2d_6lev_"
restart
= .false.
trstrt
= 0.01
tplot
= 0.001
nend
= 10000
tmax
= 0.05
run_comment
= "2D Sedov explosion, from t=0 with r_init = 3.5dx_min"
log_file
= "sedov_2d_6lev.log"
eint_switch
= 1.e-4
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Output Flow in Flash2
Flow of Ouput in Flash2
call OUTPUT
happens everytime step
call checkpoint_wr
call plotfile_wr
call particleplot
if time_elapsed > trstrt
if time_elapsed > tplot
if time_elapsed > ptplot
or pfilter .eq. true
hdf5_serial
hdf5_parallel
pnetcdf
hdf5_serial
hdf5_parallel
pnetcdf
hdf5_serial
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
hdf5_parallel
pnetcdf
Serial vs. Parallel I/O in Flash2
 Serial I/O
Master process gathers data from all others in the communicator and dumps the data
Supported in HDF5
 Parallel I/O
Each process writes its own data in a fully non-contiguous manner using global offsets
global offsets calculated by gathering all the blocks from all processes and
each process determines the amount of blocks “to the left” of itself
Same thing is done for particles
Supported in HDF5 and Pnetcdf
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
How does Flash do parallel I/O really??
 Paramesh data structures are local to a processor


For example the lrefine data structure has a dimension of (ndim,local_blocks)
Using a global offset number that is local to each processor along with the local blocks on each
processor the full lrefine(ndim,total_blocks) data can be written to a file in parallel non-contiguously
 Real Examples (writing level of refinement in parallel)
void FTOC(h5_write_lrefine)(hid_t* file_identifier,
call h5_write_lrefine(file_id,
int lrefine[],
lrefine, &
int* local_blocks,
lnblocks, &
int* global_offset)
int* total_blocks,
{
tot_blocks, &
rank = 1;
dimens_1d = *total_blocks;
global_offset)
dataspace = H5Screate_simple(rank, &dimens_1d, NULL);
dataset = H5Dcreate(*file_identifier, "refine level", H5T_NATIVE_INT,
dataspace, H5P_DEFAULT);
start_1d = (hssize_t) (*global_offset);
stride_1d = 1;
count_1d = (hssize_t) (*local_blocks);
status = H5Sselect_hyperslab(dataspace, H5S_SELECT_SET, &start1d,&stride1d, &count_1d,
NULL)
dimens_1d = *local_blocks;
memspace = H5Screate_simple(rank, &dimens_1d, NULL);
status = H5Dwrite(dataset, H5T_NATIVE_INT, memspace, dataspace, H5P_DEFAULT, lrefine)
}
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Parallel I/O: putting it all together
Top Down view of implementing parallel I/O
Parallel Filesystem
PVFS
MPI Implementation
MPI-IO (ROMIO)
Native binary (uses raw mpi-io)
parallel HDF5
parallel netCDF
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
HDF5 vs. Parallel NetCDF



Parallel-NetCDF is more lightweight and more tightly coupled to mpi-io.
Writing attribute/header data in parallel-netcdf is done in one call, while in hdf5 multiple single
processor writes must occur to complete these operations
There is some good indication that parallel-netcdf is faster
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Visualizing Flash Data
 2d / 3d – slice tool FILDR3






Uses IDL
Can make 2d images and 3d-slices of Flash data
Has support for particles
Can give 1-d slices of 2d data
Can show the amr grid
Both hdf5 and pnetcdf currently supported
 Flashviz









Developed by Mika Papka and Randy Hudson (MCS Argonne)
Renders 3d iso-surfaces of Flash data
Fully parallel (both reading the file and geometry calculations are done in parallel)
Parallelism allows for the ability to look at very large datasets interactively
Supports rendering multiple iso-surfaces of multiple variables or the same variable
Can show the amr grid in 3d
Has xyz cut plane features so that the entire range of data for a variable can be projected on a plane
Future support for parallel rendering
Currently only supports hdf5 format
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Example of 3d slice taken by FILDR3
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Example of an isosurface (single variable)
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Another example with two variables
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Another example/ isosurface with cut plane
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Isosurface with Grid
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Issues with Cell-centered vs. Corner Interpolated Data
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago
Issues with cell-centered vs. corner interpolated data
The ASC/Alliances Center for Astrophysical Thermonuclear Flashes
The University of Chicago