Transcript Comments by Saša Žiković

Comments on
“Incorporating Uncertainties into Economic Forecasts:
an Application to Forecasting Economic Activity in
Croatia”
Dario Rukelj and Barbara Ulloa
by Saša Žiković
16th DEC, Dubrovnik, Croatia, 2010
Motivation
• Although forecasts are by their nature probabilistic most
economic predictions quote just a single value without giving
attached probabilities.
• As many countries started their policies of inflation targets,
the incorporation of the uncertainty in economic variable
predictions served to show that there is uncertainty about
shocks to affect the economy.
• The approach proposed by Garrat et al. (2003) regarding
uncertainty forecasting, is applied in forecasting economic
activity in Croatia. The incorporation of uncertainty about the
unobserved future shocks on forecasts and robustness of
forecasts to the choice of the parameters is done for Rukelj
(2010) model.
• Rukelj (2010) model is used assuming that it balances
economic theory and consistency with data, and so has good
forecasting capabilities.
Good points of the paper
• The authors present density forecasts of economic activity in
Croatia using stochastic simulations of the random errors with
parametric and non-parametric approaches, as well as the
evaluation of density forecasts,
• Ecoometric are nicely and correctly done
• Parametric and non-parametric methods are used in generation of
shocks and the probability bands are constructed from the
obtained set of simulated values.
• I liked the use of bootstrap for non parametric errors I hope that
the bootstrapping was applied to IID errors – since I don’t see any
mention of this in the paper)
• Introduces skewness in future distributions in order to represent
potential risks in Croatian economy
• The study is legitimate as an applied study if the procedure is
relevant to a Croatian policy question the authors have in mind .
Comments and suggestions
• A bit difficult to read because of numerous errors - needs to checked
by a professional...
• The literature review is unsatisfactory in terms of the econometric
methods used, the theoretical basis of the study and policy
discussions.
• The small macro-model (Rukelj 2010) is not properly introduced,
evaluated and discussed. All we have is the author’s word that his
own model performs well.
• The methodology section is a not-so-competent summary of the
appendix to Garett et al (2003). Equations 1-5 are essentially
equations 27-31 in Gatett et. al. (2003), the only difference being the
letters used in notation. Moreover the transcription is incomplete and
wrong at some points. For example on page 5 the autors talk about ϕi
but this set of coefficients do not exist in their notation, ϕi exist in
Garett et. al.(2003).
• In section 2.2 the number of replications are denoted of period T+h
forecasts is denoed by both x and r interchangably.
• In most places summation notation is not properly specified, no
definitions and range for summation subscripts like x, h, t and s.
Rukelj (2010) SVAR in Croatia
paper
• Rukelj, D. 2010, "Modelling Fiscal and Monetary Policy Interactions in
Croatia using Structural VECM"
- very simple theoretical framework with 3 equations and 3 variables
(government expenditures, M1 and velocity of money) and one
common stochastic trend which drives the whole system.
• There are NO characteristic Croatian features so I don’t understand
why this would be a preferred model?
• “...we chose the model proposed by Rukelj (2010) as it accurately
accounts for Croatian’s macroeconomic features, and it performs well
when forecasting” – Rukelj 2010 paper can only serve to establish
benchmark parameters - the model is a basic SVAR. I don’t see any
Croatian macroeconomic features and any sign of forecasting
performance – there is absolutely no backtesting or comparison with
any other model???
Comments and suggestions
• Following eq. 3 on page 5 the predictive probability distribution
function is not presented and similarly no predictive probability
distribution function is presented for the case of parameter
uncertainty after eq. 5.
• The bullet points explaining how probability bands are defined look
like they are adopted from somewhere because the notation does
not follow the preceding equations (especially w.r.t. x and r
superscripts).
ˆ ( s )is not properly explained. No information
• Following equation 6, 
is given on whether simulated errors differ with or without
parameter uncertainty. The authors were better off just citing
Gratett et al. (2003) inviting the curious reader to look at the
original exposition of the technique.
• The notation mistakes and lack of explanations of the procedure
being followed make this section very problematic. There is no
methodological contribution here. This is just an application of the
Garett et al. methodology to Croatian data.
• Why is there no comparison with other approches? You can use
Clements and Smith (2000) methodology for competing models.
Comments and suggestions
• Introduction of skewness produce very poor results – not U
distr and increases the chances of observing a higher index
in the time of crisis(???) – it is as useful as a fifth wheel!
• Asymmetric shocks idea used by Bank of England in Inflation
Report since 1996, but is there any asymetry? Have you
tested for it - here the moments values are taken ad hoc and
give strong positive skew – this is obviously the authors’
subjective view of the Croatian economy although I don’t
understand why?
• Why would the balance of risks be constant over time? – look
at the current crisis
• There have been a lot of advances since 1996 look at GP
and skewed Student’s t distribution
• What about structural breaks?
Comments and suggestions
• Density forecasts suggest the economic activity index would have
increased the first trimester of 2009, regardless the method for
errors simulation. – SVAR needs to be improved only 3 variables
(government expenditures, M1 and velocity of money) – overly
simplistic – what about structural breaks, price and wage
rigidities, exchange and interest rates, consumption, exports,
imports, revenues, small open economy - influence of the state of
the economy of the major trading partners, would have been
particularly useful in predicting the effects of the current crisis on
Croatia
• Your bad forecasting results are not surprising if you notice that
the calibration of your model was done on a time period which
showed positive economic activity, without considering alternative
states of the economy and/or changes in the global surrounding
the results will always be equal to driving while looking at the rear
view mirror.
Comments and suggestions
• There is no dynamics in the generation of errors, no volatility
modelling, the authors assume IID, how come? Authors are sure
that there is no ARMA or GARCH terms in empirical errors? – look
at your KS results. Density forecasts make most sense when
combined with ARMA-GARCH framework
• You only show the results - there is absolutely no discussion of
the results - what are the implications of this results, what is the
reason behind obtained results - this would be interesting to
researchers and forecasters
• The results are not very compelling because you can’t reject the H
of outturns being U-dist in only 7/12 (only future uncertainty –
parametric errors) and 5/12 (only future uncertainty – nonparametric errors) – and these are your best approaches!
• What is the message of the paper – it is not worth including
parameter uncertainty in forecasts? Why is this so? Make an
argument! Provoke debate, attack the whole concept – BUT with
strong arguments! What do the other authors say?
Comments and suggestions
• limitations of KS test - tends to be more sensitive near the centre
of the distribution than at the tails; and the distribution must be
fully specified, i.e., if location, scale, and shape parameters are
estimated from the data, the critical region of the K-S test is no
longer valid. It typically must be determined by simulation.
• ➨ Lilliefors test -uses sample estimates of parameter values
instead of parameter values that are assumed to be known
• ➨ Instead of directly testing if the errors are IID U(0,1) you can
transform observations to make them normal under the null
hypothesis. This can be done by applying an inverse normal
transformation to the uniform series.
If xt is IID U(0,1), then zt = Ф-1(xt) is IID N(0,1). When the data is
transformed to follow normal distribution, a wider array of powerful
statistical tools can be applied, than under uniform distribution.
You can test the null hypothesis that zt is IID N(0,1) against a fairly
general first-order autoregressive process with a possibly different
mean and variance.
Comments and suggestions
• It is not clear why exactly you are introducing parameter and future
uncertainty into the Rukelj 2010 model and why we should care
whether the inclusion of only one type of uncertainty or both types
is more appropriate.
• Is the model a benchmark model for the Croatian economy? Is it
used in policy design by the policy maker?
• Is there an events scenario to be simulated using the models
developed here?
• Without the answers to these questions the finding that the model
with future uncertainty marginally passes the KS test does not
mean much.
• It does not tie into any policy discussion or theoretical discussion.
• Thus far you only showed that this approach is not good at
forecasting economic activity in Croatia???
• What are the impacts of uncertainty on government policies?
Comments and suggestions
• 4 pages of unexplained graphs in a 16 page article on
top of 3-4 pages which are practically transcribed from
another article is not good use of space. This leaves very
little room for original contributions. Moreover I am not
following why the table on page 10 has 12 columns. I
believe a summary measure could have been
developed. Needless to say, that table needs a a title
and a number.
• In the references Billix and Sellin 1998 does not have
proper citation info.
• Figures would have been more informative if realized
values were reported alongside the forecasts so that
model performances could be visually compared.
Comments and suggestions
• For your future research look at...
1) Measuring forecast uncertainty: A review with evaluation based on a
macro model of the French economy (C. Bianchi, G. Calzolari)
• Five alternative techniques have been applied to measure the degree of
uncertainty associated with the forecasts produced by a macro-model of
the French economy, the Mini-DMS developed at INSEE. They are
bootstrap, analytic simulation on coefficients, Monte Carlo on coefficients,
parametric stochastic simulation and re-estimation, a residual-based
procedure. Due to the complexity and the size of the model (nonlinear
and with more than 200 equations), several associated technical
problems had to be solved. The remarkable convergence of results which
has been obtained for all the main endogenous variables suggests that
forecast confidence intervals are likely to be quite reliable for this model.
2) Norwegian RIMINI model (30 stochastic equations, and about 100
exogenous variables)
To conclude
• Correct econometric exercise.
• If the goal is to formulate forecasts and policy
responses this is a first step on a loooong journey!
Comment about these comments...
• “It is always easier to criticize than to write”!
A word of warning!
• “Theoretical models are necessary tools in our attempts to
understand and “explain” events in real life but whatever
“explanations” we prefer, it is not to be forgotten that they
are all our own artificial inventions in a search for an
understanding of real life; they are not hidden truth to be
discovered.” (Haavelmo, 1944)
• “The disposition of present-day statistical theorists to
suppose that all ‘error’ distributions are exactly normal can
be ascribed to their ontological perception that normality is
too good not to be true” (Anscombe, 1967)