The Role of Sampling Errors in Explaining Recent Surprises in USDA Corn Stocks Estimates: Part I

In our farmdoc daily post of February 7, we identified and evaluated a number of explanations that have been offered to explain the large surprises in some USDA quarterly corn stocks estimates since the 2006-07 marketing year. Based on a set of four criteria, we found that each of those potential explanations fell short of providing a complete explanation for the frequency, timing, and magnitude of surprises that have been experienced, particularly beginning with the 2009-10 marketing year. We indicated that one additional explanation, sampling error, may have more merit for explaining the pattern of surprises.  In particular, we identified that unresolved sampling errors associated with the final corn production estimate may provide a more complete explanation for the surprises in the stocks estimates.  In addition to sampling errors associated with production estimates, however, there can also be sampling errors associated with the estimates of stocks.  Here, we address the issue of sampling errors associated with stocks estimates and in tomorrow’s post will take up the issue of sampling errors associated with the final production estimates released in January following the corn harvest. This is the fourth in a series of farmdoc daily posts discussing the findings of our recent report, which can be found here. The research was funded by the Office of the Chief Economist of the USDA.

The survey results for estimating on-farm stocks of corn (and all other commodities) are subject to sampling error because only a sample of farm operators are surveyed.  While the sample is of sufficient size and is drawn randomly in such a way as to be representative of all farm operations, it is not a census of all farm operations.  As a result, a different sample could produce different results and any sample might not accurately reflect the entire set of farm operators.  In contrast, the survey of off-farm stocks is a near census.  All known commercial operations that may have crops in storage are surveyed.  The survey includes about 4,800 entities that represent nearly 8,800 storage facilities.  In the most recent survey cycle, the response rate to the survey was about 82 percent of the entities.  However, the response rate for larger entities was even higher so that more than 82 percent of the storage capacity was represented by the respondents to the off-farm survey.

The USDA acknowledges and quantifies the potential sampling error for the on-farm stocks estimates in each of the quarterly stocks reports.  The following statement, for example, was included in the January 2014 report that contained the estimates of stocks as of December 1, 2013:

“Reliability of the on-farm and off-farm stocks must be treated separately because the survey designs for the two surveys are very different.The on-farm stocks estimates are subject to sampling variability because all operations holding on-farm stocks are not included in the sample. This variability, as measured by the relative standard error at the United States level, is approximately 1.7 percent for corn, 2.3 percent for soybeans, and 2.6 percent for all wheat. This means that chances are approximately 95 out of 100 that survey estimates for stocks will be within plus or minus 3.4 percent for corn, 4.6 percent for soybeans, and 5.2 percent for all wheat of the value that could be developed by averaging the estimates produced from all possible samples selected from the same population and surveyed using the same procedures.”

Both the on-farm and off-farm stocks estimates may also contain non-sampling errors, which cannot be directly measured and are not considered in our analysis.  The Grain Stocks reports contain a statement relative to these potential errors.  The January 2014 report, for example contained the following statement:

“Survey indications are also subject to non-sampling errors such as omission, duplication, imputation for missing data, and mistakes in reporting, recording, and processing the data. Off-farm, as well as on-farm stocks, are subject to these types of errors. These errors cannot be measured directly,   but they are minimized through rigid quality controls in the data collection process and a careful review of all reported data for consistency and reasonableness.”

With this background, we can turn our attention to the sampling error in the USDA quarterly corn stocks estimates.  When the USDA releases a point estimate of quarterly corn stocks, it needs to be recognized that the true magnitude of on-farm stocks may differ from the point estimate.  With 95 percent confidence, for example, on-farm stocks on December 1, 2013, which were estimated at 6.38 billion bushels, may have been as much as 217 million bushels larger or smaller than the point estimate (plus or minus 3.4 percent).  The size of this 95 percent confidence interval has varied over time and varies from report to report.  The interval tends to be the smallest, in percentage terms, for the December 1 estimate and largest for the September 1 estimate.  Since December 1 stocks are much larger than September 1 stocks, the confidence interval measured in bushels is largest for the December 1 estimate, declines through the marketing year, and is smallest for the September 1 estimate.  Note that sampling error only applies to the on-farm stocks estimates since the off-farm stocks estimates are not based on a randomized survey design.

As a result of the presence of sampling error, some of the market surprise provided by USDA stocks estimates may come from the fact that actual on-farm stocks differ from the point estimate released by the USDA.  Market participants, then, may have correctly anticipated stock levels, but those stocks were not confirmed due to sampling error, resulting in a surprise to the market.  To judge whether sampling error could explain the magnitude of market surprises in recent years, we calculated the magnitude of the 95 percent confidence interval for each quarterly estimate of on-farm stocks from December 2009 through December 2013 and compared that to the magnitude of the market surprise in each of those reports.  The confidence interval is that reported by the USDA in each of the Grain Stocks reports.  The period starting in 2009 was selected due to the prevalence of large surprises beginning that year. The magnitude of the surprise is measured as the difference between the USDA estimates of stocks and the average expectation for the estimates as revealed by a survey of market participants and reported by Dow Jones Newswire.  For the December 1 report, the magnitude of the surprise in the stocks estimate is adjusted for any unexpected change in the corn production estimate in the January Annual Crop Production report.  That unexpected change is calculated as the USDA production estimate minus the average trade guess for the size of the estimate.

The comparison of the magnitude of the 95 percent confidence interval for the on-farm stocks estimate and the surprise provided by the estimate of all stocks for the last 17 stocks estimates is presented in the last two columns of Table 1.  For 9 of the 17 quarters, the confidence interval is nearly as large, or larger, than the magnitude of the surprise in the stocks estimate.  Those results suggest that sampling error could be a potential explanation for those surprises.  That is, actual stocks may have been closer to the expected level than implied by the USDA point estimate.  For 8 of the 17 quarters highlighted in Table 1, however, the magnitude of usage implied by the quarterly stocks estimate was larger than the 95 percent confidence interval around the estimate of on-farm stocks.  In these cases, then, sampling error does not appear to be an explanation for the magnitude of surprises.

It might be argued that the large surprise in the stocks estimate for those eight quarters might be explained by sampling errors of opposite directions in two consecutive quarters.  The sum of the sampling errors in those two estimates, then, might explain the surprise for the most recent quarter.  However, all of the eight large surprises exceed the magnitude of the 95 percent confidence interval for the current and previous on-farm stocks estimate.

Conclusions

While sampling errors associated with USDA estimates of on-farm corn stocks may explain some of the surprises in the quarterly stocks estimates, they clearly do not explain all of the large surprises.  In addition, the presence of sampling error would not satisfy the criteria for explaining surprises in the corn stocks estimate described in our previous post: 1) Why corn and not soybeans? 2) Why 2006-2012 and not previous marketing years? 3) Why only in particular marketing years? and 4) Why a pattern of reversals in surprises during marketing years?  Non-sampling error for both the on-farm and off-farm stocks estimates might also explain some of the surprise, but the potential size of such errors cannot be measured.  It is unlikely that such errors would be large enough to explain the largest surprises in stocks estimates observed since 2009.  Tomorrow’s post will take up the issue of sampling errors in corn production estimates as an explanation of surprises in corn stocks estimates.