Combining Load Forecasts from Independent Experts
Experience at NPower Forecasting Challenge 2015
Jingrui Xie, Bidong Liu, Xiaoqian Lyu,Tao Hong
Energy Production and Infrastructure Center
University of North Carolina at Charlotte
Charlotte, NC, USA
Abstract - The NPower Forecasting Challenge 2015 invited
students and professionals worldwide to predict daily energy
usage of a group of customers. The BigDEAL team from the Big
Data Energy Analytics Laboratory landed a top 3 place in the
final leaderboard. This paper presents a refined methodology
based on the implementation during the competition. We first
build the individual forecasts using several forecast techniques,
such as Multiple Linear Regression (MLR), Autoregressive
Integrated Moving Average (ARIMA), Artificial Neural Network
(ANN) and Random Forests (RF). We then select a subset of the
individual forecasts based on their performance on a validation
period, a.k.a. post-sample. Finally we obtain the final forecast by
averaging the selected individual forecasts. The forecast
combination on average yields a better result than the forecast
from a single technique.
Keywords—electric load forecasting; forecasting combination;
forecast competition; NPower Forecasting Challenge 2015
I. INTRODUCTION
Energy industry is one of the many industries that integrate
forecasting to the day-to-day business operations. Accurate
forecasts help the energy companies to purchase the proper
amount of electricity with the least cost on a daily basis. The
NPower Forecasting Challenge 2015 was organized by
NPower with the primary purpose of recruiting summer
interns, though the competition was open to students and
professionals worldwide. The competition topic was six-month
ahead ex post daily energy forecasting, which falls into the
category of retail energy forecasting [1]. In this paper, we
present a refined methodology based on the implementation
during the competition.
During the past many decades, extensive literature has been
devoted to load forecasting [2]. A recent review of short term
load forecasting is in [3]. Common load forecasting techniques
include regression analysis [4], time series models [5], artificial
neural networks [6], and fuzzy regression [7]. While the load
forecasters have not yet found a technique that can dominate
all the others, forecast combination has been regarded by the
forecasting community as a practical method to enhance the
forecast accuracy [8][9]. The literature in load forecast
combination is limited. Some researches tackled the problem
by combining weather stations [10] [11], while others have
focused on combining forecasts from different modeling
techniques [12]–[15]. A recent load forecast combination study
is presented in [16], where the author combined forecasts from
David Basterfield
Volume Position Management
RWE npower
Solihull, UK
several sister models to enhance the accuracy of point load
forecast.
In the nPower Forecasting Challenge 2015, we first
generated individual forecasts using different techniques, such
as Multiple Linear Regression (MLR), Autoregressive
Integrated Moving Average (ARIMA), Artificial Neural
Network (ANN), and Random Forecast (RF). We then
averaged a subset of these individual forecasts to achieve a
more robust and accurate point forecast than each individual
forecast.
The rest of the paper is organized as follow: Section II
introduces the data used in the competition. Section III presents
the modeling techniques and the forecast combining strategy.
Section IV discusses the results. The paper is then concluded in
Section V.
II. DATA DESCRIPTION
The competition included three rounds. In each round, the
contestants were asked to submit a six-month ahead ex post
forecast of daily energy consumption, which is similar to the
sliding simulation method discussed in [17]. At the beginning
of the first round, two sets of data were provided. One dataset
included the historical data used for modeling, such as the daily
energy consumption, and the weather and the calendar
information as summarized in Table I. The other dataset
included six months of actual weather and calendar
information in the forecasting period. At the beginning of each
of the next two rounds, the organizer provided the actual daily
energy consumption over the forecasting period of the previous
round, as well as the actual weather and calendar information
of the next six months.
For each round of the competition, we dissect the data into
three pieces as summarized in Table II: 1) the training data for
parameter estimation; 2) the validation data for model
selection; and 3) the test data (i.e. the forecast data), which is
the forecast period designated by the competition organizer
NPower. Fig. 1 shows the actual daily energy consumption
data from the historical period of the first round to the forecast
period of the third round. Fig. 2 shows the scatter plots
between the daily energy consumption and the weather
variables, such as temperature, wind speed, precipitation, and
solar radiation.
 Fig. 1. Daily energy consumption (Apr. 2011 – Sep. 2014)

Fig. 2. Scatter plots of daily energy consumption (kWh) and weather variables by day type

TABLE I. LIST OF VARIABLES

Variable Name Variable Description

Volume Daily energy consumption (kWh)
Temp Daily average temperature (C°)
WindSpeed Daily average wind speed (m/s)
PrecipAmt Rainfall equivalent in the preceding half hour (mm)
SolarRad Solar radiation (kJ/m²)
Weekdays: WE (weekday) / SA (Saturday) / SU (Sunday)
DayType
Special holidays: A2 / CD / BD / GF / J1 / M1 / M2 / SD / EM.
SchoolHoliday Dummy variable: 1 if school holiday, 0 otherwise
Winter Dummy variable: 1 if winter, 0 otherwise
Trend A linear trend variable
Weekday With value 1 to 7 to represent the 7 days of a week
Month With value 1 to 12 to represent the 12 months of a year
TABLE II. TRAINING, VALIDATION, AND TEST DATA IN THE THREE ROUNDS The backward selection strategy starts with the following
model:
Round Training Validation Test
1 Apr. 2011 - Mar. 2012 & Apr. 2012 - Apr. 2013 - Volume = β 0Trend + β1Temp + β 2Temp 2 + β 3Temp 3
Oct. 2012 - Mar. 2013 Sep. 2012 Sep. 2013 + β 4 Month * Weekday + β 5 DayType + β 6WindSpeed (3)
2 Apr. 2011 - Sep. 2012 & Oct. 2012 - Oct. 2013 -
Apr. 2013 - Sep. 2013 Mar. 2013 Mar. 2014 + β 7 PrecipAmt + β 8 SolarRad + β 9 SchoolHoliday
3 Apr. 2011 - Mar. 2013 & Apr. 2013 - Apr. 2014 - + β10Winter
Oct. 2013 – Mar. 2014 Sep. 2013 Sep. 2014
From Eq. (3), we try to exclude variables from the model one
at a time. If the MAPE value of the validation period does not
Mean Absolute Percentage Error (MAPE) as defined get worse due to excluding a variable from the model, we
below is used by the competition organizer to evaluate the eliminate the variable from the model. Otherwise, we keep the
submitted ex-post forecast in each round. We also used it as variable in the model. We repeat this backward selection
the measure for model selection: process until all variables in Eq. (3) have been enumerated. We
1 n y − yˆ i then consider grouping the similar weekdays and treating
MAPE = ∑ | i | (1)
n i =1 yi holidays as special day(s) as summarized in Table IV. The best
where yi and yˆ i are the actual daily energy and the forecasted model selected from the backward selection strategy is denoted
daily energy of day i, respectively. as MLR2.

III. MODELING PROCESS TABLE IV. DAY CODE MODIFICATION (BACKWARD SELECTION MLR)

A. Multiple Linear Regression Original Day Code Modified Day Code

Tuesday Thursday
For the MLR technique, we implement two variable Wednesday Thursday
selection strategies, a forward selection strategy and a Saturday Sunday
backward selection strategy. Dec. 23rd to Jan. 1st (next year) Special weekday 8

The forward selection strategy in principle follows the

variable selection methodology proposed in [3]. We first build
a model with four quantitative variables (Trend, Temp, Temp2
and Temp3), one class variable (Weekday) and the interactions
between Weekday and temperature variables. The model can be
specified as follows:
Table V lists the best MLR models from these two variable
selection strategies for each of the three rounds of the
competition.
TABLE V. MULTIPLE LINEAR REGRESSION MODELS

Volume = β 0Trend + β1Temp * Weekday + β 2Temp * Weekday

2 Round Model Independent Variables
(2) Trend, Temp*Weekday, Temp2*Weekday,
+ β 3Temp * Weekday
3
MLR1
Temp3*Weekday
1
Trend, Temp, Temp2, Temp3, WindSpeed,
We then try to add other variables listed in Table I into Eq. MLR2
PrecipAmt, DayType, Month*Weekday
(2) one at a time. If an additional variable improves the MAPE Trend, School_Holiday, Temp*Weekday,
value of the validation period, we keep the variable in the MLR1
Temp2*Weekday, Temp3*Weekday
2
model. Otherwise, we abandon it. We repeat this process until MLR2
Trend, Temp, Temp2, WindSpeed, DayType,
enumerating all variables listed in Table 1. Finally, we consider Month*Weekday
grouping the similar weekdays and treating some special Trend, Day_Type, School_Holiday,
MLR1 Temp*Weekday, Temp2*Weekday,
holidays as weekend days, which is again similar to the 3 Temp3*Weekday
methodology proposed in [3]. The modification of the day code Trend, Temp, Temp2, DayType, Month*Weekday,
for Round 1 is summarized in Table III. For each round of the MLR2
SchoolHoliday
competition, the same modeling process was conducted based
on the validation data as specified in Table II. Therefore, the
final model and the day type code modification may vary from B. ARIMA
one round to another. The MLR model selected from this ARIMA is a widely used technique in time series
forwarding selection strategy is denoted as MLR1. forecasting. In this competition, we used several ARIMA
models to generate some of the individual forecasts. We use a
TABLE III. DAY CODE MODIFICATION (FORWARD SELECTION MLR) trial-and-error method to identify the order of the auto-
Original Day Code Modified Day Code regressive model (p), the degree of the differencing (d), and the
Thursday Wednesday order of the moving average term (q). For each of these three
A2 Sunday parameters (i.e. p, d and q), we test the order from 1 to 10,
EM Sunday which in total yields 1000 different ARIMA models. We select
GF Saturday the best combination of parameters based on the MAPE in the
M1 Saturday
validation period. The selected p, d and q values for each of the
M2 Sunday
SD Sunday
three rounds of the competition are listed in Table VI.
 TABLE VI. PARAMETERS OF THE ARIMA MODELS selected individual forecasts to obtain the combined forecasts.
We enumerate various options of grouping individual
Rounds p d q
forecasts. Based on the MAPE values in the validation period
1 5 7 1 of each round, we select the group of individual forecast(s) that
2 7 4 9 returns the lowest MAPE after simple averaging. Table VII
3 1 7 9 presents the validation MAPE values of the 5 individual
models and the forecast combination in each round.

C. Artificial Neural Netwrok IV.

ANN models have been widely used in the load forecasting
literature. We use a three-layer feed-forward neural network
model in this competition. The input variables include the
following variables: Trend, Temp, SchoolHoliday and
DayType. Both SchoolHoliday and DayType are categorical
variables, a.k.a. classification variables, which are
implemented as multiple indicator variables when serving as
the inputs to the ANN models.
D. Random Forests
RF is an ensemble learning method that builds a large
number of decision trees in the training period, and outputs the
mode or mean of the individual trees. In the competition, we
develop random forest models with 500 trees to forecast the
daily energy consumption. During the first two rounds, we
include the four quantitative variables Temp, WindSpeed,
PrecipAmt and SolarRad as features. For the 3rd round, we add
Weekday defined in Table IV as an additional feature.
E. Combined Forecast
Among various forecast combination methods, simple
average is the easiest one to implement and usually
outperforms many other complicated combination schemes
[18]. Simple average is also one of the top performers in the
load forecasting case study presented in [16]. Considering its
effectiveness and simplicity, we take the simple average of the
FORECASTING RESULTS
The MAPE values of each individual model and the
forecast combination are presented in Table VIII. On average,
the combination yields the MAPE of 2.40%, which is better
than each individual model. While the performance of a single
model could be quite volatile, the performance of the
combination is relatively stable. For instance, the MAPE
values of ARIMA models in the three rounds range from
3.13% to 13.24%. We also list our original MAPE values from
the implementation during the competition.
V. CONCLUSION
The NPower Forecasting Challenge 2015 attracted more
than 40 teams worldwide. The forecast combination method
from BigDEAL team won a top 3 place in the final
leaderboard. BigDEAL was the only team managing to have
less than 3% MAPE in all three rounds; the only team ranking
top 2 twice; one of the only three teams ranking top 10 in all
three rounds; and one of the only two teams with consistently
improved MAPE throughout the three rounds. This paper
presents the refined forecast competition method based on the
BigDEAL’s implementation in the NPower Forecasting
Challenge 2015. The method leverages the results from
individual models. On average, the forecast combination yields
better and more stable results than each individual model.

TABLE VII. VALIDATION MAPE (%) OF INDIVIDUAL MODELS AND COMBINED RESULTS.
Round MLR1 MLR2 ANN ARIMA RF Combined Combined Option
1 2.34 2.45 2.25 3.38 4.76 2.18 MLR1+ANN+MLR2
2 2.39 2.02 3.18 7.6 3.75 1.83 MLR1+MLR2
3 2.45 2.13 2.63 3.12 3.25 2.13 MLR2

TABLE VIII. MAPE (%) Values of the Forecast Period.

Round MLR1 MLR2 ANN ARIMA RF Combined Competition
1 3.46 3.42 2.98 8.31 3.57 3.01 2.84
2 3.05 2.24 4.23 13.24 3.25 2.39 2.39
3 2.05 1.75 2.42 3.13 4.78 1.75 2.21
AVG. 2.85 2.47 3.27 8.23 3.87 2.40 2.48

[3] T. Hong, “Short term electric load forecasting,” Ph.D. Dissertaion,

REFERENCES
[1] J. Xie, T. Hong, and J. Stroud, “Long-term Retail Energy Forecasting
with Consideration of Residential Customer Attrition,” IEEE
Transactions on Smart Grid, vol. pp, no. 99, 2015.
[2] T. Hong, “Energy forecasting : past, present, and future,”
Foresight:The International Journal of Applied Forecasting, no. 32, pp.
43–48, 2014.
Graduate Program of Operation Research and Dept. of Electrical and
Computer Engineering, North Carolina State University, 2010.
[4] T. Hong, P. Wang, and H. Willis, “A naïve multiple linear regression
benchmark for short term load forecasting,” 2011 IEEE Power and
Energy Society General Meeting, Detroit, MI.
[5] M. T. Hagan and S. M. Behr, “The time series approach to short term
load forecasting,” IEEE Transactions on Power Systems, vol. 2, no. 3,
pp. 785–791, 1987.
[6] H. S. Hippert, C. E. Pedreira, and R. C. Souza, “Neural networks for
short-term load forecasting: a review and evaluation,” IEEE
Transactions on Power Systems, vol. 16, no. 1, pp. 44–55, 2001.
[7] T. Hong and P. Wang, “Fuzzy interaction regression for short term load
forecasting,” Fuzzy Optimization and Decision Making, vol. 13, no. 1,
pp. 91–103, Sep. 2013.
[8] J. S. Armstrong, Principles of Forecasting: a handbook for researchers
and practitioners. Springer Science & Business Media, 2001.
[9] J. Nowotarski, E. Raviv, S. Trück, and R. Weron, “An empirical
comparison of alternate schemes for combining electricity spot price
forecasts,” Energy Economics, vol. 46, pp. 395–412, 2014.
[10] S. Fan, L. Chen, and W.-J. Lee, “Short-term load forecasting using
comprehensive combination based on multimeteorological information,”
IEEE Transactions on Industry Applications, vol. 45, no. 4, pp. 1460–
1466, 2009.
[11] D. Fay and J. Ringwood, “On the influence of weather forecast errors in
short-term load forecasting models,” IEEE Transactions on Power
Systems, vol. 25, no. 3, pp. 1751–1758, 2010.
The author has requested enhancement of the downloaded file. All in-text references underlined in blue are linked to publications on ResearchGate.
[12] D. Smith, “Combination of forecasts in electricity demand prediction,”
International Journal of Forecasting, vol. 8, no. 3, pp. 349–356, 1989.
[13] J. Wang, S. Zhu, W. Zhang, and H. Lu, “Combined modeling for
electric load forecasting with adaptive particle swarm optimization,”
Energy, vol. 35, no. 4, 2010.
[14] J. Taylor, “Short-term load forecasting with exponentially weighted
methods,” IEEE Transactions on Power Systems, vol. 27, no. 1, pp.
458–464, 2012.
[15] M.Matijas, J.Suykens, and S.Krajcar, “Load forecasting using a ˇ
multivariate meta-learning system,” Expert Systems with Applications,
vol. 40, no. 11, pp. 4427–4437, 2013.
[16] J. Liu, “Combining sister load forecasts,” University of North Carolina
at Charlotte, 2015.
[17] L. J. Tashman, “Out-of-sample tests of forecasting accuracy: an analysis
and review,” International Journal of Forecasting, pp. 437–450, 2000.
[18] V. Genre, G. Kenny, A. Meyler, and A. Timmermann, “Combining
expert forecasts: Can anything beat the simple average?,” International
Journal of Forecasting, vol. 29, no. 1, pp. 108–122, 2011.