OR-94-17-1 -- Bayesian Nonlinear Modeling for the Prediction Competition PDF

The 1993 energy prediction competition involved the prediction of a series of building energy loads from a series of environmental input variables. Nonlinear regression using neural networks is a popular technique for such modelling tasks. Since it is not obvious how large the input time-window should be or what preprocessing of inputs is best, this can be viewed as a regression problem in which there are many possible input variables, some of which may actually be irrelevant to the prediction of the output variable. Because a finite data set will show random correlations between the irrelevant inputs and the output, any conventional neural network (even with regularisation or "weight decay") will not set the coefficients for these junk inputs to zero. Thus, the irrelevant variables will hurt the model's performance. The automatic relevance determination (ARD) model puts a prior probability distribution over the regression parameters that embodies the concept of relevance. This is done in a simple and "soft" way by introducing multiple regularisation constants - one associated with each input. Using Bayesian methods, the regularisation constants for junk inputs are automatically inferred to be large, preventing those inputs from causing significant overfitting. An entry using the ARD method won the competition by a significant margin.

KEYWORDS: artificial intelligence, expert systems, calculating, heat load, accuracy, energy consumption, competition