Transcript Diapositiva 1

The use of information from
experts for agricultural official
statistics
Ballin Marco, Carbini Riccardo,
Loporcaro Maria Francesca, Lori Massimo,
Moro Roberto, Olivieri Valeria, Scanu Mauro
Istituto Nazionale di Statistica
Roma, 10 July 2008
Q 2008
Aim of the work and summary of the talk
• The main aim of this work is to investigate if
we can assess the quality of statistics based
on experts opinion and if these statistics can
be used as official statistics
• The talk
– The Italian experience
– Some reminders on elicitation and a proposal for an
elicitation scheme
– Some results belonging to an experimental
elicitation
– Quality report and next steps in the research
Roma, 10 June 2008
Q 2008
Introduction
Up to now, expert opinion has been widely used to produce short term
statistics on crops.
Main users:
•
•
•
Eurostat
System of National Accounts
Economic Operators
Summarizing, local authorities supply an evaluation of area and yield on the
different crops (Jannuary: estimates on areas for winter cereals, May:
estimates on area for maize, June estimates on yields for winter crops ,….).
Roma, 10 June 2008
Q 2008
Example
Example: common wheat area (national level) according to the
Farm Structure Survey (sample survey) and aggregated
evaluations from local authorities
Common wheat area (ha)
800.000
700.000
600.000
500.000
400.000
FSS
300.000
Evaluat ions f rom local aut horit ies
200.000
100.000
0
1997
Roma, 10 June 2008
1998
1999
2000
2003
2005
Q 2008
Example
Example 2: maize area (national level) according to the Farm
Structure Survey (sample survey) and aggregated evaluations
from local authorities
Maize area (ha)
1200000
1150000
1100000
1050000
1000000
950000
FSS
Evaliations from local authorities
900000
850000
1997
Roma, 10 June 2008
1998
1999
2000
2003
2005
Q 2008
Why a survey on expert opinions?
…a first judgement
• Up to now in Italy these “statistics” seem to reproduce quite
well estimates produced by sample surveys for many crops
…some other important strengths:
• Timeliness (results available before traditional survey
estimates)
• Analytic data (estimates with a high geographical detail)
• Inexpensive
Roma, 10 June 2008
Q 2008
Why a survey on expert opinions?
… but also some weaknesses:
• Heterogeneity of the process at local level
• Lacking of accountability
• Problems in assessing data quality
In other words
• difficulties in defining and filling up a quality report
Roma, 10 June 2008
Q 2008
Proposal
From evaluation to structured elicitation
To overcome these difficulties and to transform experts opinion in an
additional tool for official statisticians useful to investigate
phenomena difficult to observe with traditional surveys we propose
to adopt a formal expert elicitation
“Expert elicitation in the context of uncertainty quantification aims at a
credible and traceable account of specifying probabilistic information
regarding uncertainty in a structured and documented way.” (Hora, 1992)
Roma, 10 June 2008
Q 2008
Proposal
From evaluation to structured elicitation
Elicitation is currently and successfully applied in many experimental
situations, as weather forecasting, biomedical applications, nuclear
risks assessment, attributing foodborne pathogen illness to food
consumption …
Protocols on the elicitation process have already been introduced in
many research fields. These allow to evaluate the quality of the
elicitation process, making the users aware about the use of the
statistical results
…….a proposal of elicitation in the official statistics context
Roma, 10 June 2008
Q 2008
Proposal
• 1. Organization
– Definition of the problem (and the questionnaire)
– Finding out at least one expert and one facilitator
– To train the experts and the facilitators
• 2. Elicitation
–
–
–
–
–
Carry out the interviews
Transform expert opinions into probability distributions
Combine opinions of different experts (if available)
Feedback
Production of final results
Roma, 10 June 2008
Q 2008
Proposal
• 3. Evaluation of the elicitation results (quality report)
–Timeliness, Coherence, Relevance, Comparability, Accessibility
(as usual statistics)
–Accuracy
•
•
•
•
Variability of elicited distributions (if more then one distribution have
been elicited; this indicator should replace the usual variance)
Effect of feedback (this should replace the indicators on effect of
re-interview)
Fiducial interval (this should replace the confidence interval)
Assessment of expert knowledge by means of control/seed
variables (this indicator should replace indicators on bias)
–Furthermore
•
•
•
Self assessment of expert knowledge and of his/her sources of
information (by expert)
Assessment of how the questions in the questionnaire are phrased
(by expert)
meta-information on the interview (by facilitator)
Roma, 10 June 2008
Q 2008
An experience: the saffron case
Step 1. Organization
Problem definition: structure of saffron sector in Italy (Saffron
is a rare crop, not observable by sample surveys)
Questions on the following phenomena: production and total
area of saffron in Abruzzo, in Italy, in the world; export and
import quantities; number of planted bulbs in the last year;
forecast on production, number of operators,…
Selection of experts: president of the consortium of saffron
producers in one of the two most important areas for the
production of saffron in Italy (Navelli county)
Selection of facilitators: Istat personnel with
agricultural background belonging to regional office
strong
Training: document on how the elicitation process is
conducted and on the main biases that can affect the elicitation
process. Mail and phone contacts
Roma, 10 June 2008
Q 2008
An experience: the saffron case
Step 2. Elicitation
face to face Interview (about one hour) on
•
•
Minimum, Maximum, Mode (most probable value) for each
phenomenon of interest
Distribution shape using fixed interval method (Ten “X” to be put
in five intervals of equal length)
Check of data and first results
• Check of data (compatibility, units measure, etc.,…)
• Fitted distribution (Rectangular or Beta)
• Point estimates
• Fiducial interval
Feedback from expert and facilitator
•
•
Update of elicitation by expert and facilitator
Final results
Roma, 10 June 2008
Q 2008
An experience: the saffron case
Final result (concerning production of saffron in 2007)
120 Kg is the point estimation (mode)
(105 kg -130 kg) is the range within the production of saffron in 2007 lies
with probability one for the expert
111-126 Kg is the 95% fiducial interval according to the expert distribution
Expert
info
Point
estimate
Fitted
distribution
Fiducial
interval
Roma, 10 June 2008
Q 2008
An experience: the saffron case
Step 3. Evaluation of elicitation results
Some indicators among those proposed before,
•Effect of feedback (expert confirmed his point of view after the
production of the first report)
•Fiducial intervals (e.g. 111-126 Kg for the production of saffron in
Italy)
•Self assessment of expert knowledge and of his/her sources of
information
Confidence in the
reference sources
4
Index (sum of points/max)
81,2%
3
2
1
statistical culture
0
Contacts with local and
national producers
Roma, 10 June 2008
degree of agreement
with other
experts/producers
Q 2008
An experience: the saffron case
Step 3. Evaluation of elicitation results
• Assessment of how the questions are phrased (by expert)
•Results on number of planted bulbs could produce
misunderstanding if they are not integrated with information
about type and quality of bulbs
• Meta-information on the interview (by facilitator):
•Expert declared and seemed to be honoured to collaborate
with Istat
•He appeared well trained and showed a true mastery of the
topics
•He collaborated during the feedback too
•He has years of experience and has written many publications
on the topic
Roma, 10 June 2008
Q 2008
Final summary and Future developments
Summary
1) The elicitation process formalizes the uncertainty of one or more
experts on particular phenomena
2) Statistics based on experts have some good properties
3) In some cases expert opinion is the only available sources of
information
4) It is possible to define protocols and quality reports to upgrade
“numerical evaluations” to statistics
5) some indicators have been proposed for accuracy
6) A small experience has been (partially) illustrated
Future developments
1) Combination of elicited distribution belonging to two or more experts
(by feedback, mathematical aggregation, …)
2) Combination of elicited distributions and data belonging to surveys
(linear combination, bayesian framework, ….., used as early
estimates to be replaced when data are available).
3) Development of protocols for fields different from agriculture
4) …..
Roma, 10 June 2008
Q 2008
Bibliography
Anthony O'Hagan, Caitlin E. Buck, Alireza Daneshkhah, J. Richard Eiser, Paul H.
Garthwaite, David J. Jenkinson, Jeremy E. Oakley, Tim Rakow (2006) Uncertain
Judgements: Eliciting Experts' Probabilities. Wiley
Di Bacco (1990). l’aggregazione di valutazioni diprobabilità: una rassegna…non
imparziale , 8, ed. Pitagora. Bologna
Garthwaite P. H., Kadane J. B., and O'Hagan A. (2005): Statistical methods for
eliciting probability distributions, Journal of the American Statistical Association,
100, 680-701.
ISTAT (1993): Manuale delle statistiche agricole rilevate con le tecniche estimative.
Note e relazioni, 1993, 1. Roma: ISTAT.
Mortera J.(1990). “Aggregazione delle opinioni: una panoramica” Rassegna di
metodi statistici ed applicazioni, 8, ed. Pitagora. Bologna
SHELF (Sheffield Elicitation Framework): http://www.tonyohagan.co.uk/shelf
J.P. van der Sluijs, P.H.M. Janssen, A.C. Petersen, P. Kloprogge, J.S. Risbey, W.
Tuinstra, J.R. Ravetz (2004):Tool Catalogue of the RIVM/MNP Guidance for
Uncertainty Assessment and Communication
Roma, 10 June 2008
Q 2008
Saffron: a flower – many recipies
Saffron is obtained from the yellow “core” of the flower
Crocus. It is mainly produced in Iran and India, with
minor productions in Spain, Greece, and Italy.
Italy produces about 120 kg of saffron per year.
It is used in many regional recipes, as the Risotto alla
Milanese
Roma, 10 June 2008