Autoregressive moving-average model explained

In the statistical analysis of time series, autoregressive–moving-average (ARMA) models are a way to describe of a (weakly) stationary stochastic process using autoregression (AR) and a moving average (MA), each with a polynomial. They are a tool for understanding a series and predicting future values. AR involves regressing the variable on its own lagged (i.e., past) values. MA involves modeling the error as a linear combination of error terms occurring contemporaneously and at various times in the past. The model is usually denoted ARMA(p, q), where p is the order of AR and q is the order of MA.

The general ARMA model was described in the 1951 thesis of Peter Whittle, Hypothesis testing in time series analysis, and it was popularized in the 1970 book by George E. P. Box and Gwilym Jenkins.

ARMA models can be estimated by using the Box–Jenkins method.

Mathematical formulation

Autoregressive model

See main article: Autoregressive model. The notation AR(p) refers to the autoregressive model of order p. The AR(p) model is written as

where

are parameters and the random variable is white noise, usually independent and identically distributed (i.i.d.) normal random variables.^{[1] [2]}

In order for the model to remain stationary, the roots of its characteristic polynomial must lie outside the unit circle. For example, processes in the AR(1) model with

are not stationary because the root of lies within the unit circle.^[3]

The augmented Dickey–Fuller test assesses the stability of IMF and trend components. For stationary time series, the ARMA model is used, while for non-stationary series, LSTM models are used to derive abstract features. The final value is obtained by reconstructing the predicted outcomes of each time series.

Moving average model

See main article: Moving-average model. The notation MA(q) refers to the moving average model of order q:

where the

are the parameters of the model, is the expectation of (often assumed to equal 0), and , ..., are i.i.d. white noise error terms that are commonly normal random variables.^[4]

ARMA model

The notation ARMA(p, q) refers to the model with p autoregressive terms and q moving-average terms. This model contains the AR(p) and MA(q) models,^[5]

In terms of lag operator

In some texts, the models is specified using the lag operator L. In these terms, the AR(p) model is given by

where

represents the polynomial

The MA(q) model is given by

where

represents the polynomial

Finally, the combined ARMA(p, q) model is given by

or more concisely,

or

This is the form used in Box, Jenkins & Reinsel.^[6]

Moreover, starting summations from

and setting and , then we get an even more elegant formulation:

Spectrum

The spectral density of an ARMA process is$$S(f)\; =\; \backslash frac\; \backslash left\backslash vert\; \backslash frac\; \backslash right\backslash vert^2$$where

is the variance of the white noise, is the characteristic polynomial of the moving average part of the ARMA model, and is the characteristic polynomial of the autoregressive part of the ARMA model.^{[7] [8]}

Fitting models

Choosing p and q

An appropriate value of p in the ARMA(p, q) model can be found by plotting the partial autocorrelation functions. Similarly, q can be estimated by using the autocorrelation functions. Both p and q can be determined simultaneously using extended autocorrelation functions (EACF).^[9] Further information can be gleaned by considering the same functions for the residuals of a model fitted with an initial selection of p and q.

Brockwell & Davis recommend using Akaike information criterion (AIC) for finding p and q.^[10] Another option is the Bayesian information criterion (BIC).

Estimating coefficients

After choosing p and q, ARMA models can be fitted by least squares regression to find the values of the parameters which minimize the error term. It is good practice to find the smallest values of p and q which provide an acceptable fit to the data. For a pure AR model, the Yule-Walker equations may be used to provide a fit.

ARMA outputs are used primarily to forecast (predict), and not to infer causation as in other areas of econometrics and regression methods such as OLS and 2SLS.

Software implementations

• In R, standard package stats has function arima, documented in ARIMA Modelling of Time Series. Package astsa has an improved script called sarima for fitting ARMA models (seasonal and nonseasonal) and sarima.sim to simulate data from these models. Extension packages contain related and extended functionality: package tseries includes the function arma, documented in "Fit ARMA Models to Time Series"; packagefracdiff contains fracdiff for fractionally integrated ARMA processes; and package forecast includes auto.arima for selecting a parsimonious set of p, q. The CRAN task view on Time Series contains links to most of these.
• Mathematica has a complete library of time series functions including ARMA.^[11]
• MATLAB includes functions such as arma, ar and arx to estimate autoregressive, exogenous autoregressive and ARMAX models. See System Identification Toolbox and Econometrics Toolbox for details.
• Julia has community-driven packages that implement fitting with an ARMA model such as arma.jl.
• Python has the statsmodelsS package which includes many models and functions for time series analysis, including ARMA. Formerly part of the scikit-learn library, it is now stand-alone and integrates well with Pandas.
• PyFlux has a Python-based implementation of ARIMAX models, including Bayesian ARIMAX models.
• IMSL Numerical Libraries are libraries of numerical analysis functionality including ARMA and ARIMA procedures implemented in standard programming languages like C, Java, C# .NET, and Fortran.
• gretl can estimate ARMA models, as mentioned here
• GNU Octave extra package octave-forge supports AR models.
• Stata includes the function arima. for ARMA and ARIMA models.
• SuanShu is a Java library of numerical methods that implements univariate/multivariate ARMA, ARIMA, ARMAX, etc models, documented in "SuanShu, a Java numerical and statistical library".
• SAS has an econometric package, ETS, that estimates ARIMA models. See details.

History and interpretations

The general ARMA model was described in the 1951 thesis of Peter Whittle, who used mathematical analysis (Laurent series and Fourier analysis) and statistical inference.^{[12] [13]} ARMA models were popularized by a 1970 book by George E. P. Box and Jenkins, who expounded an iterative (Box–Jenkins) method for choosing and estimating them. This method was useful for low-order polynomials (of degree three or less).^[14]

ARMA is essentially an infinite impulse response filter applied to white noise, with some additional interpretation placed on it.

In digital signal processing, ARMA is represented as a digital filter with white noise at the input and the ARMA process at the output.

Applications

ARMA is appropriate when a system is a function of a series of unobserved shocks (the MA or moving average part) as well as its own behavior. For example, stock prices may be shocked by fundamental information as well as exhibiting technical trending and mean-reversion effects due to market participants.

Generalizations

There are various generalizations of ARMA. Nonlinear AR (NAR), nonlinear MA (NMA) and nonlinear ARMA (NARMA) model nonlinear dependence on past values and error terms. Vector AR (VAR) and vector ARMA (VARMA) model multivariate time series. Autoregressive integrated moving average (ARIMA) models non-stationary time series (that is, whose mean changes over time). Autoregressive conditional heteroskedasticity (ARCH) models time series where the variance changes. Seasonal ARIMA (SARIMA or periodic ARMA) models periodic variation. Autoregressive fractionally integrated moving average (ARFIMA, or Fractional ARIMA, FARIMA) model time-series that exhibits long memory. Multiscale AR (MAR) is indexed by the nodes of a tree instead of integers.

Autoregressive–moving-average model with exogenous inputs (ARMAX)

The notation ARMAX(p, q, b) refers to a model with p autoregressive terms, q moving average terms and b exogenous inputs terms. The last term is a linear combination of the last b terms of a known and external time series

. It is given by: where are the parameters of the exogenous input .

Some nonlinear variants of models with exogenous variables have been defined: see for example Nonlinear autoregressive exogenous model.

Statistical packages implement the ARMAX model through the use of "exogenous" (that is, independent) variables. Care must be taken when interpreting the output of those packages, because the estimated parameters usually (for example, in R^[15] and gretl) refer to the regression:

where incorporates all exogenous (or independent) variables:

See also

• Autoregressive integrated moving average (ARIMA)
• Exponential smoothing
• Linear predictive coding
• Predictive analytics
• Infinite impulse response
• Finite impulse response

Further reading

• Book: Mills, Terence C. . Time Series Techniques for Economists . Cambridge University Press . 1990 . 0521343399 . registration .
• Book: Percival . Donald B. . Andrew T. . Walden . Spectral Analysis for Physical Applications . Cambridge University Press . 1993 . 052135532X .
• .
• Shumway, R.H. and Stoffer, D.S. (2017). Time Series Analysis and Its Applications with R Examples. Springer. DOI: 10.1007/978-3-319-52452-8

Notes and References

1. Book: Box, George E. P. . Time series analysis : forecasting and control . 1994 . Prentice Hall . Gwilym M. Jenkins, Gregory C. Reinsel . 0-13-060774-6 . 3rd . Englewood Cliffs, N.J. . 54 . en . 28888762.
2. Book: Shumway, Robert H. . Time series analysis and its applications . 2000 . Springer . David S. Stoffer . 0-387-98950-1 . New York . 90–91 . en . 42392178.
3. Book: Box . George E. P. . Time series analysis : forecasting and control . Jenkins . Gwilym M. . Reinsel . Gregory C. . 1994 . Prentice Hall . 0-13-060774-6 . 3rd . Englewood Cliffs, N.J. . 54–55 . en . 28888762.
4. Book: Box . George E. P. . Time series analysis : forecasting and control . Jenkins . Gwilym M. . Reinsel . Gregory C. . Ljung . Greta M. . 2016 . John Wiley & Sons, Incorporated . 978-1-118-67492-5 . 5th . Hoboken, New Jersey . 53 . en . 908107438.
5. Book: Shumway, Robert H. . Time series analysis and its applications . 2000 . Springer . David S. Stoffer . 0-387-98950-1 . New York . 98 . en . 42392178.
6. Book: Box . George . Time Series Analysis: Forecasting and Control . Jenkins . Gwilym M. . Reinsel . Gregory C. . Prentice-Hall . 1994 . 0130607746 . Third.
7. Book: Rosenblatt, Murray . Gaussian and non-Gaussian linear time series and random fields . 2000 . Springer . 0-387-98917-X . New York . 10 . en . 42061096.
8. Book: Wei, William W. S. . Time series analysis : univariate and multivariate methods . 1990 . Addison-Wesley Pub . 0-201-15911-2 . Redwood City, Calif. . 242–243 . en . 18166355.
9. Web site: Missouri State University. Model Specification, Time Series Analysis.
10. Book: Brockwell . P. J. . Davis . R. A. . Time Series: Theory and Methods . 2nd . Springer . New York . 2009 . 273 . 9781441903198 .
11. http://www.wolfram.com/products/applications/timeseries/features.html Time series features in Mathematica
12. Book: Hannan, Edward James . Edward James Hannan . Multiple time series . John Wiley and Sons . 1970 . Wiley series in probability and mathematical statistics . New York.
13. Book: Whittle, P. . Hypothesis Testing in Time Series Analysis . Almquist and Wicksell . 1951.

    Book: Whittle, P. . Prediction and Regulation . English Universities Press . 1963 . 0-8166-1147-5.

    Republished as: Book: Whittle, P. . Prediction and Regulation by Linear Least-Square Methods . University of Minnesota Press . 1983 . 0-8166-1148-3.

14. Book: Hannan . E. J. . Edward James Hannan . Statistical theory of linear systems . Deistler . Manfred . John Wiley and Sons . 1988 . Wiley series in probability and mathematical statistics . New York.

15. http://search.r-project.org/R/library/stats/html/arima.html ARIMA Modelling of Time Series