Transcript pptx - Galformod Project Portal

Millennium Data Dissemination
databases and other services
2012-12-17
Millennium Workshop
1
FOF groups and Subhalos
Raw data:
Particles
Mock images
Density fields
Subhalo merger trees
Synthetic galaxies (SAM)
Mock catalogues
2012-12-17
Millennium Workshop
2
MRObs
Simulation
Particles
Gadget
FOF Groups
Scaling
(Sub-)Halos
Halo Merger
Trees
FOF Group
Finder
Tree
Builder
SUBFIND
SAM
SSP
SExtractor
SExtractor
Catalogues
2012-12-17
SkyMaker’
MoMaF’
Synthetic
Images
Light-cone
Galaxies
PSF, Noise,
Background
IGM
Millennium Workshop
SAM
Galaxies
3
Virgo-Millennium Database
• Online access to Millennium simulations etal
– portal: http://www/mpa-garching.mpg.de/millennium
• Databases
– public: http://gavo.mpa-garching.mpg.de/Millennium
– private + MyDB:
http://gavo.mpa-garching.mpg.de/MyMillennium
• Improved database access
– MillenniumTAP batch queries
– http://galformod.mpa-garching.mpg.de/millenniumtap
2012-12-17
Millennium Workshop
4
Millennium data model (old)
5
5
Trees in Database
last progenitor
2012-12-17
main leaf and root
Millennium Workshop
6
Merger trees
particles
2012-12-17
halos
Millennium Workshop
7
Joining Garching and Durham
8
8
2012-12-17
Millennium Workshop
9
Most recent data sets
(@MPA)
• New simulations:
– MR-XXL: Angulo etal 2012
– Millennium WMAP7 (Thomas etal, MilliGas?)
• Updated semi-analytics, with (scaling to) WMAP7, MCMC
– Guo etal 2011, 2012
– Henriques etal 2012/3
• New synthetic light cone catalogues
– Henriques etal 2012
– also some all-sky (SDSS depth)
• Millennium Run Observatory
– Overzier etal 2012, images+Sextractor catalogues
• Most loaded in new databases
– soon: MR-XXL trees, Millennium WMAP7 (sim +scaled, halos + galaxies)
• New DB functions:
– spatial queries
– possibly: scaling, …
2012-12-17
Millennium Workshop
10
AllSky(Paritosh Verma)
“All”
smoothed
“All”
“Red”
2012-12-17
Millennium Workshop
“Blue”
11
Most recent data sets
(@MPA)
• New simulations:
– MR-XXL: Angulo etal 2012
– Millennium WMAP7 (Thomas etal, MilliGas?)
• Updated semi-analytics, with (scaling to) WMAP7, MCMC
– Guo etal 2011, 2012
– Henriques etal 2012/3
• New synthetic light cone catalogues
– Henriques etal 2012
– also some all-sky (SDSS depth)
• Millennium Run Observatory
– Overzier etal 2012, images+Sextractor catalogues
• Most loaded in new databases
– soon: MR-XXL trees, Millennium WMAP7 (sim +scaled, halos + galaxies)
• New DB functions:
– spatial queries
– possibly: scaling, …
2012-12-17
Millennium Workshop
12
Spatial queries in the database
Spatial3D library (with Tamas Budavari)
• Searches for objects in spatial subsets of
simulation box is hard
• C# library for optimising such spatial queries
using PH space filling curve
• Soon in Millennium DBs (needs documentation)
2012-12-17
Millennium Workshop
13
with cover as (
-- some details left out
select *
from dbo.fSimulationCoverSphere(
Simulation::Millennium,
Sphere3D::newInstance(0,0,0,20),6) )
select f.*
from cover c
,
millennium..fof f
where f.snapnum=67
and f.phkey between c.keymin and keymax
and c.shape.ContainsPoint(f.x, f.y, f.z)=1
14
Query shape syntax
• Primitives :
– box[ 0,0,0,10,10,10 ], sphere[10,10,10,20]
– cone[0,0,0,1,1,1,.1,600] vertex,direction,angle,min-depth,max-depth,
– conesegment[0,0,0,1,1,1,.1,500,600]
vertex,direction,angle,min-depth,max-depth
• Combinations :
– union/intersection/difference [ shape (, shape)* ]
15
used in MXXL browser
courtesy RaulAngulo
In the future: light cones in SQL
(requires galaxy catalogues with dM/dz)
2012-12-17
Millennium Workshop
17
Usage statistics
• 589 registered users
• > 17.3 million queries (7532/day)
• > 287 billion rows (137 million/day)
queries,rows,users
2012-12-17
Millennium Workshop
18
Distributions over users
2012-12-17
Millennium Workshop
19
Millennium TAP
• New DB access service
– All MyMillennium users have access
• Batch queue
– longer timeout
– capability to kill queries
– history
• Implements IVOA’s Table Access Protocol (TAP)
– AJAX frontend: TAPFace
– TAP aware tools can query it, TOPCAT
• Some differences
– schema vs database (Non SQLServer DBs can be served)
– no mydb (yet)
• Demo this afternoon
2012-12-17
Millennium Workshop
20
It’s not always as simple as it seems.
One should try to understand the data when using the database
thanks to
Kevin Bundy, Tommaso Treu, Richard Ellis
http://astro.berkeley.edu/~kbundy/millennium/
2012-12-17
Millennium Workshop
21
Other Millennium data
dissemination services
• portal: http://galformod.mpa-garching.mpg.de
• Simulation service(s)
– SeSam online (Neistein, Weinmann , Bourges)
– L-Galaxies online (see talk by Matthias Egger)
• Image Browsers:
– may help define where to point your virtual telescope.
– MR-XXL browser (public | interactive) (Angulo),
– MRObs image browser (public | private) (Overzier
etal)
2012-12-17
Millennium Workshop
22
Service vs other data sets
• Databases
– HydroClusters database (Dolag)
• Virtual telescopes
– Smac (Dolag, Reynecke, Bourges)
– PHOX (Biffi etal)
• Image Browsers:
– Phoxy browser + PHOX
2012-12-17
Millennium Workshop
23
Thank you.
2012-12-17
Millennium Workshop
24
24