METHODOLOGIES for ERROR MEASUREMENT

Researched and Presented by:

Christopher Thompson

Which forecast error measuring tool is the best?

Of the four error tracking tools we have learned about today (MAD, MSE, MAPE and seasonality), which method provides the best way to track forecast error? Give examples.

Mean Absolute Percent Error (MAPE)

A problem with both the MAD and MSE is that their values depend on the magnitude of the item being forecast. If the forecast item is measured in thousands, the MAD and MSE values can be exceptionally large. To avoid this problem, we can use the mean absolute percent error (MAPE). This is computed as the average of the absolute difference between the forecasted and actual values, expressed as a percentage of the actual values. That is, if we have forecasted and actual values for n periods, the MAPE is calculated as:

Mean absolute percent error (MAPE)

The average of the absolute differences between the forecast and actual values, expressed as a percent of actual values.

“Three commonly used measures for summarizing historical errors are the mean absolute deviation (MAD), the mean squared error (MSE), and the mean absolute percent error (MAPE). MAD is the average absolute error, MSE is the average of squared errors, and MAPE is the average absolute percent error.

MAD is the easiest to compute, but weights errors linearly. MSE squares errors, thereby giving more weight to larger errors, which typically cause more problems. MAPE should be used when there is a need to put errors in perspective.”

I think it is a judgment call when to use those method because each method is good for certain situation. However, I do prefer to use MAPE method to calculate forecast errors.

Retrieved from:

http://highered.mcgraw-hill.com/sites/dl/free/0073525251/886181/stevenson11e_sample_ch03.pdf

The mean squared error is arguably the most important criterion used to evaluate the performance of a predictor or an estimator. (The subtle distinction between predictors and estimators is that random variables are predicted, and constants are estimated.) The mean squared error is also useful to relay the concepts of bias, precision, and accuracy in statistical estimation. In order to examine a mean squared error, you need a target of estimation or prediction, and a predictor or estimator that is a function of the data. Suppose that the target, whether a constant or a random variable, is denoted as  . The mean squared error of the estimator or predictor   for   is

  One use for these measures is to compare the accuracy of alternative forecasting methods. For instance, a manager could compare the results to determine one which yields the lowest MAD, MSE, or MAPE for a given set of data. Another use is to track error performance over time to decide if attention is needed.  Overall, the operations manager must settle on the relative importance of historical performance versus responsiveness and whether to use MAD, MSE, or MAPE to measure historical performance. MAD is the easiest to compute, but weights errors linearly. MSE squares errors, thereby giving more weight to larger errors, which typically cause more problems. MAPE should be used when there is a need to put errors in perspective.

The first measure of the overall forecast error for a model is the mean absolute deviation (MAD). This value is computed by taking the sum of the absolute values of the individual forecast errors (deviations) and dividing by the number of periods of data. The mean squared error (MSE) is a second way of measuring overall forecast error. MSE is the average of the squared differences between the forecasted and observed values. A drawback of using the MSE is that it tends to accentuate large deviations due to the squared term. For example, if the forecast error for period 1 is thrice as large as the error for period 2, the squared error in period 1 is nine times as large as that for period 2. Hence, using MSE as the measure of forecast error typically indicates that we prefer to have several smaller deviations rather than even one large deviation. A problem with both the MAD and MSE is that their values depend on the magnitude of the item being forecast. If the forecast item is measured in thousands, the MAD and MSE values can be exceptionally large. To avoid this problem, we can use the mean absolute percent error (MAPE). This is computed as the average of the absolute difference between the forecasted and actual values, expressed as a percentage of the actual values. The MAPE is perhaps the easiest measure to interpret. For example, a result that the MAPE is 6% is a clear statement that it is not dependent on issues such as the magnitude of the input data.

References: Heizer, Jay H. Operations management – 10th Edition Textbook

The MAPE is the easiest measurement to interpret because using percentages give a straight forth picture of a forecast.  And we all know that when you are in meetings all managers throw out percentages to make their point.

According to the text on page 116, the MAPE is perhaps the easiest measure to interpret. For example, a result that the MAPE is 6% is a clear statement that is not dependent on issues such as the magnitude of the input data. 

A problem with both the MAD and MSE is that their values depend on the magnitude of the item being forecast. If the forecast item is measured in thousands, the MAD and MSE values can be very large. To avoid this problem, we can use the mean absolute percent error (MAPE). This is computed as the average of the absolute difference between the forecasted and actual values, expressed as a percentage of the actual values. That is, if we have forecasted and actual values for n periods, the MAPE is calculated as:

The average of the absolute differences between the forecast and actual values, expressed as a percent of actual values.

Ref: Operations Management, tenth Edition

What other forecast measuring tool(s) can we use besides MAD, MSE, and MAPE to better show the direction of the forecast error?

There are several very efficient tools used to find errors in forecasts.  One of those tools, are using a tracking signal.  A Tracking signal is defined in our textbook as:

A tracking signal is a measurement of how well a forecast is predicting actual values. As forecasts are updated every week, month, or quarter, the newly available demand data are compared to the forecast values. The tracking signal is computed as the cumulative error divided by the mean absolute deviation (MAD). 

Our textbook further explains the meaning of positive and negative tracking signals:

Positive tracking signals indicate that demand is greater than forecast. Negative signals mean that demand is less than forecast. A good tracking signal—that is, one with a low cumulative error—has about as much positive error as it has negative error. In other words, small deviations are okay, but positive and negative errors should balance one another so that the tracking signal centres closely around zero. A consistent tendency for forecasts to be greater or less than the actual values (that is, for a high absolute cumulative error) is called a bias error. Bias can occur if, for example, the wrong variables or trend line are used or if a seasonal index is misapplied.

This information is clear and concise.  I would like to see this application performed in excel.

Render, Jay Heizer and Barry (). Operations Management [10] (VitalSource Bookshelf), Retrieved from

http://devry.vitalsource.com/books/9781256081487/id/ch4box38

Forecast bias is a tendency for a forecast to be consistently higher or lower than the actual value. Forecast bias is distinct from forecast error in that a forecast can have any level of error and be completely unbiased.

Bias= the sum of error/the number of forecast errors

Running sum of forecast errors (RSFE)- provides a measure of forecast bias

RSFE= the sum of forecast error for the number of periods

 Information retrieved from

http://www.scmfocus.com/demandplanning/2012/02/forecastbias/

What about Trend Projections and the Least Squares Method?

Trend projection

A time-series forecasting method that fits a trend line to a series of historical data points and then projects the line into the future for forecasts. (Render 119)

If we decide to develop a linear trend line by a precise statistical method, we can apply the least-squares method. This approach results in a straight line that minimizes the sum of the squares of the vertical differences or deviations from the line to each of the actual observations. (Render 119)

A least-squares line is described in terms of its y-intercept (the height at which it intercepts the y-axis) and its expected change (slope). If we can compute the y-intercept and slope, we can express the line with the following equation:

y(hat) = a + b x

(Render 119)

Render, Jay Heizer and Barry. Operations Management, 10th Edition. Pearson Learning Solutions. <vbk:9781256081487#outline (8.8.8)>.

 Another forecast measuring tool is Trend Projections which is a time-series forecasting method that fits a trend line to a series of historical data points and then projects the line into the future for forecasting  the technique fits a trend line to a series of historical data points and then projects the line into the future for medium to long-range forecasts.

 Trend projection:  This technique fits a trend line to a series of historical data points and then projects the line into the future for medium to long-range forecasts. Several mathematical trend equations can be developed (for example, exponential and quadratic), but in this section, we will look at linear (straight-line) trends only.

http://devry.vitalsource.com/#/books/9781256081487/pages/55879738

 Tracking signal is one such tool that shows the direction of the forecast and compare with the actual values. A tracking signal is a measurement of how well a forecast is predicting actual values. As forecasts are updated every week, month, or quarter, the newly available demand data are compared to the forecast values. As used in forecasting, the tracking signal is the number of mean absolute deviations that the forecast value is above or below the actual occurrence. A tracking signal (TS) can be calculated using the arithmetic sum of forecast deviations divided by the mean absolute deviation:

TS = RSFE/MAD

Where RSFE = The running sum of forecast errors, considering the nature of the error. (For example, negative errors cancel positive errors and vice versa.

MAD = The average of all the forecast errors (disregarding whether the deviations are positive or negative). It is the average of the absolute deviations.

 Once tracking signals are calculated, they are compared with predetermined control limits. When a tracking signal exceeds an upper or lower limit, there is a problem with the forecasting method, and management may want to re-evaluate the way it forecasts demand.

How does a bias and trend adjustment affect MAD, MSE, and MAPE?

Forecast accuracy can be determined by computing the bias, mean absolute deviation (MAD), mean square error

(MSE) or mean absolute percent error (MAPE) for the forecast using different values for alpha. Bias is the sum of the forecast errors [Σ(FE)]. For the exponential smoothing example above, the computed bias would be:

(60 – 41.5) + (72 – 54.45) + (58 – 66.74) + (40 – 60.62) = 6.69

If one assumes that a low bias indicates an overall low forecast error, one could compute the bias for a few potential values of alpha and assume that the one with the lowest bias would be the most accurate. However, caution must be observed in that wildly inaccurate forecasts may yield a low bias if they tend to be both over forecast and under forecast (negative and positive). For example, over three periods a firm may use a particular value of alpha to over forecast by 75,000 units (-75,000), under forecast by 100,000 units (+100,000), and then over forecast by 25,000 units (-25,000), yielding a bias of zero (-75,000 + 100,000 25,000 = 0). By comparison, another alpha yielding over forecasts of 2,000 units, 1,000 units, and 3,000 units would result in a bias of 5,000 units. If normal demand was 100,000 units per period, the first alpha would yield forecasts that were off by as much as 100 percent while the second alpha would be off by a maximum of only 3 percent, even though the bias in the first forecast was zero.

A safer measure of forecast accuracy is the mean absolute deviation (MAD). To compute the MAD, the forecaster sums the absolute value of the forecast errors and then divides by the number of forecasts (Σ |FE| ÷N). By taking the absolute value of the forecast errors, the offsetting of positive and negative values is avoided. This means that both an over forecast of 50 and an under forecast of 50 are off by 50. Using the data from the exponential smoothing example, MAD can be computed as follows:

(|60 – 41.5| + |72 – 54.45| + |58 – 66.74| + |40 – 60.62|) ÷ 4 = 16.35

Therefore, the forecaster is off an average of 16.35 units per forecast. When compared to the result of other alphas, the forecaster will know that the alpha with the lowest MAD is yielding the most accurate forecast.

Mean square error (MSE) can also be utilized in the same fashion. MSE is the sum of the forecast errors squared divided by N-1 [Σ(P(FE)) ÷ (N-1)]. Squaring the forecast errors eliminates the possibility of offsetting negative numbers, since none of the results can be negative. Utilizing the same data as above, the MSE would be:

[(18.5) + (17.55) + (–8.74) + (–20.62)] ÷ 3 = 383.94

As with MAD, the forecaster may compare the MSE of forecasts derived using various values of alpha and assume the alpha with the lowest MSE is yielding the most accurate forecast.

The mean absolute percent error (MAPE) is the average absolute percent error. To arrive at the MAPE one must take the sum of the ratios between forecast error and actual demand times 100 (to get the percentage) and divide by N[(Σ | Actual demand —forecast | ÷Actual demand) × 100 ÷ N]. Using the data from the exponential smoothing example, MAPE can be computed as follows:

[(18.5/60) + 17.55/72 + 8.74/58 + 20.62/48) × 100] ÷ 4 = 28.33%

As with MAD and MSE, the lower the relative error the more accurate the forecast.

It should be noted that in some cases the ability of the forecast to change quickly to respond to changes in data patterns is more important than accuracy. Therefore, one’s choice of forecasting method should reflect the relative balance of importance between accuracy and responsiveness, as determined by the forecaster.

http://www.encyclopedia.com/topic/Forecasting.aspx

Bias is something that is often introduced into the raw data by the way the data is obtained. It is something that you can discover and remove. The problem is sometimes it is introduced without knowing, often from not having clearly defined procedures as to how to obtain something.

 Post Problem 4.13b Here

B. Use a three-year moving average to forecast demand in years 4,5,6 Year A Moving Average Error 

1 45   

2 50   

3 52   

4 56 45+50+52/3=49 7 

5 58 50+52+56/3=52.7 5.3 

6 52+56+58/3=55.3  

Made=12.3/2=6.2   

Question: Post Problem 4.13a Here

 4.13 (a) Exponential smoothing, a = 0.6:

                                                                                                Absolute

Year      Demand    Exponential   Smoothing a = 0.6         Deviation

1            45         41                                                            4.0

2            50         41.0 + 0.6(45–41) = 43.4                              6.6

3            52         43.4 + 0.6(50–43.4) = 47.4                           4.6

4            56         47.4 + 0.6(52–47.4) = 50.2                           5.8

5            58         50.2 + 0.6(56–50.2) = 53.7                           4.3

6            ?         53.7 + 0.6(58–53.7) = 56.3

                                                                                                {= 25.3

                                                                                               MAD = 5.06

(b)  Exponential smoothing, a = 0.9:

                                                                                Absolute

Year       Demand Exponential Smoothing a = 0.9    Deviation

1             45             41                                                   4.0                                 

2             50             41.0 + 0.9(45–41) = 44.6                    5.4

3             52             44.6 + 0.9(50–44.6 ) = 49.5                2.5

4             56             9.5 + 0.9(52–49.5) = 51.8                   4.2

5             58             51.8 + 0.9(56–51.8) = 55.6                 2.4

6             ?             55.6 + 0.9(58–55.6) = 57.8

                                                                                        {= 18.5

                                                                                        MAD = 3.7

As you can see in the following table, demand for heart transplant surgery at Washington General Hospital has increased steadily in the past few years: The director of medical services predicted 6 years ago that demand in year 1 would be 41 surgeries. a) Use exponential smoothing, first with a smoothing constant of .6 and then with one of .9, to develop forecasts for years 2 through 6.

Year Heart Transplants Forecast Using alpha = 0.6 Forecast Using alpha = 0.9

1 45 41.000 41.000

2 50 43.400 44.600

3 52 47.360 49.460

4 56 50.144 51.746

5 58 53.658 55.575

6   56.263 57.757

My calculations came out the same as yours in the forecast table. The MAD calculated by Willian of 5.06 and 3.7 were also the same.

 While MAD, MSE and MAPE are great ways to track forecasts, nothing to me does it better than seasonality. Working in the car business, I have realized that the seasons play a huge role in the inventory that is purchased. During the wintertime, most people are looking for vehicles that will be able to handle snow, so sports cars will not likely be purchased. Likewise, the summertime is the perfect time to purchase that two-seater that you have always wanted. After reviewing the four tracking tools MAD, MSE, MAPE and seasonality I have determined that MAPE provides the best way to track forecast error. When using the other tools there is a greater risk of making errors using large numbers. MAPE allows you to use smaller numbers and exposes you less to the risk of making errors in your populations. Also, MAPE tells you a percentage error unlike the other ones and people most likely respond high percentages faster than some other numbers.