Transcript Assistant tool for checking Monte Carlo results

A general assistant tool
for the checking results
from Monte Carlo simulations
Koi, Tatsumi
SLAC/SCCS
Contents
•
•
•
•
•
•
Motivation
Precision and Accuracy
Central Limit Theorem
Testing Method
Current Status of Development
Summary
Motivation
• After a Monte Carlo simulation, we get an answer.
However how to estimate quality of the answer.
What we must remember is
• Large number of history does not valid result of
simulation.
• Small Relative Error does not valid result of
simulation
Motivation (Cont.)
• To provide “statistical information to
assist establishing valid confidence
intervals for Monte Carlo results”
for users, something like MCNPs did.
Subject of this study
• Precision of the Monte Carlo
simulation
• Accuracy of the result is NOT a
subject of this study
At first we have to define
Precision and Accuracy of simulations
Precision and Accuracy
• Precision: Uncertainty caused by statistical
fluctuation
• Accuracy: Difference between expected value and
true physical quantity.
Accuracy
True Value
Precision
Mote Carlo Results
Subject of this study
(Cont.)
• Precision of the Monte Carlo simulation is
subject of this study.
• To state accuracy of simulations, we
should consider details of simulation, i.e.,
uncertainties of physical data, modeling of
physical processes, variance reduction
techniques and so on.
• To make a generalized tool, we have to
limit subjects only for precision.
Accuracy is a subject for most of
presentations in this workshop.
Principal of this study is
Central Limit Theorem
Central Limit Theorem
• Every data which are influenced by
many small and unrelated random
effects has normally distribution.
• The estimated mean will appear to be
sampled from normal distribution
with a KNOWN standard deviation 
when N approaches infinity.

N

Central Limit Theorem
(Cont.)
• Therefore, We have to check that N
have approached infinity in the sense
of the CLT, or not.
• This corresponds to the checking the
complete sampling of interested
phase space has occurred, or not.
This is not
a simple static test
but
check of results from
nature of Monte Carlo
simulations
Checking Values
• Mean
1
x
N
• Variance and Standard
Deviation
• Variance of Variance
x
R
S x2
i
i 1
N
S2 
• Relative error
N
 x  x 
i 1
2
i
N 1
2
S
, where S x2 
x
N
S 2 S x 
VOV 
4
Sx
Checking Values (Cont.)
• Figure of Merit
• Scoring Efficiency
Rintrinsic and Refficiency
• Shift
• SLOPE
1
FOM  2
RT
number of NON ZERO histories
q
N
SHIFT    xi  x 
Fit to the Largest history scores
3
2S N 
2
What we check?
•
•
•
•
•
•
•
•
•
Behavior of MEAN
Values of R
Time profile of R
Values of VOV
Time profile of VOV
Time profile of FOM
Behavior of FOM
Value of SLOPE
Value of SHIFT
Boolean
Answer
• Effect of the largest
history score occurs
on the next history.
–
–
–
–
–
MEAN
R (Rintrinsic and Refficiency)
VOV
FOM
SHIFT
Numeric
Answer
Of cause, Boolean check is
carried out mathematically
(statistically)
value
behavior
time profile
For behaviors and
time profiles check
• Derive Pearson’s r from data (results and
theoretical values)
– r=1(-1), perfect positive (negative) correlation
– r=0, uncorrelated
• null hypothesis is set to uncorrelated
• Then, t  r N  2 1  r 2 follows student t
distribution of degree of freedom   N  2
• Checking significance of r with null hypothesis.
• Rejection level of null hypothesis is 68.28% (1σ)
Example
• Checking value: Observable Energy of Sampling Calorimeter.
• Material
– Pb (Lead)-Scinitillator
• Thickens
– Pb: 8.0 mm/layer, Sci: 2.0 mm/layer
• Layers
– 120 layers
– 1 m x 1 m – interaction surface
• Beam
– Electon 4 GeV
• Range Cuts
– 1 mm
e-
2mm
Pb
8mm
・・・・・・・・
Sci.
Example
100 histories
MEAN
SD
mean
80
SD
10
9.9
79.5
9.8
79
9.7
78.5
mean
9.6
SD
9.5
78
9.4
77.5
9.3
77
9.2
0
20
40
60
R
0.02
0.018
0.016
0.014
0.012
0.01
0.008
0.006
0.004
0.002
0
80
100
120
0
20
40
60
80
100
120
VOV
R
VOV
0.04
0.035
0.03
0.025
R
0.02
vov
0.015
0.01
0.005
0
0
20
40
60
80
100
120
0
20
40
60
80
100
Does not pass most of Boolean tests
120
Example
1,000 histories
MEAN
SD
mean
80
79.5
79
78.5
mean
78
77.5
77
0
200
400
600
R
0.007
800
1000
1200
200
400
R
0.003
0.002
0.001
0
600
600
800
1000
1200
VOV
vov
0.004
400
SD
R
0.005
200
11
10.8
10.6
10.4
10.2
10
9.8
9.6
9.4
9.2
9
0
0.006
0
SD
800
1000
1200
0.005
0.0045
0.004
0.0035
0.003
0.0025
0.002
0.0015
0.001
0.0005
0
vov
0
200
400
600
800
1000
Does not pass some of Boolean tests
1200
Example
10,000 histories
MEAN
SD
mean
80
79.5
79
78.5
mean
78
77.5
77
0
2000
4000
6000
R
8000
10000
12000
4000
SD
0
2000
6000
4000
6000
8000
10000
12000
VOV
vov
R
2000
11
10.8
10.6
10.4
10.2
10
9.8
9.6
9.4
9.2
9
R
0.002
0.0018
0.0016
0.0014
0.0012
0.001
0.0008
0.0006
0.0004
0.0002
0
0
SD
8000
10000
12000
0.0005
0.00045
0.0004
0.00035
0.0003
0.00025
0.0002
0.00015
0.0001
0.00005
0
vov
0
2000
4000
6000
8000
10000
12000
Does not pass one of Boolean tests (SLOPE check)
How to apply Energy
Spectrum estimation etc.
P1
P2
P3
V/E
P4
E
• Checking each
confidence level of
P1, P2, P3, P4,,,,
• Of course, scoring
efficiency becomes
low.
Unfortunately, this tool
does not work well with
some deterministic variance
reduction techniques.
This is come from limitation
of CLT (means some
variance are required for
distribution), so that we
can not over come.
And some simulations
becomes deterministic
without awaking of users.
Please check your
simulation carefully.
Current Status of
Development
• Most part of developments has been
done.
• Following items are remained under
development.
– Output of testing result
– Class or function for minimization of
multi dimensional functions
We want to include this tool in
Geant4
but
what category is suite for this
tool?
Run, SD, Hits and its collections,
Tally??
Summary
• We have successfully developed a
general assistant tool for the
checking the results from Monte
Carlo simulations like MCNPs.
• Through this tool, users easily know
the confidence intervals for Monte
Carlo results.