Evaluation of Selected USDA WAOB and NASS Forecasts and Estimates in Corn and Soybeans

A number of agencies in the United States Department of Agriculture (USDA) are involved in collecting, analyzing, forecasting, and disseminating information about the production and consumption of the corn and soybean crops.  Market participants rely heavily on estimates and forecasts provided by these agencies in order to form price expectations and to make business decisions. The estimates and forecasts are also used to guide and administer a variety of government commodity programs.  In spite of on-going efforts to maintain the quality of information provided and the transparency of the methodology used, misunderstanding, concerns, or complaints about the information periodically arise.  More recently (since 2006) those concerns have centered on the accuracy of the quarterly estimates of corn inventories and to a lesser extent on the methodology and accuracy of early season yield forecasts for corn and soybeans.  The purpose of today’s post is to summarize the results of a research report   that evaluates USDA forecasts and estimates for corn and soybeans. The full report can be found here.  This is the first of a series of farmdoc daily posts that will discuss the findings in the report. The research was funded by the Office of the Chief Economist of the USDA.

The statistical analysis in the report focuses on three parts of the USDA forecasting system for corn and soybean supply and demand.  A summary of each part of the statistical analysis is found below:

• (1) National corn and soybean yield forecasts presented in the May, June, and July World Agricultural Supply and Demand Estimates report provided by the World Agricultural Outlook Board (WAOB).

• The evidence suggests that WAOB corn and soybean yield forecasts made in May, June, or July do not have a substantial bias.  The accuracy of the forecasts also has not changed markedly over the 1993-2012 time period for either corn or soybeans.  With a few exceptions, WAOB corn and soybean forecast errors since 2006 generally are within the historical range of errors.  While there are instances of large forecast errors, these are readily explained by unusual weather conditions or insect problems that occurred after the forecasts were released.

• (2) National corn and soybean yield and production forecasts in the August, September, October, and November Crop Production reports provided by the National Agricultural Statistics Service (NASS).

• NASS yield forecasts for corn reveal no evidence of bias in any month over 1990-2012 and forecast errors since 2006 are well within the historical range of errors.  There is some evidence of improvement in the accuracy of NASS corn yield forecasts over time.  Soybean forecast errors since 2006 are also within the historical range, except for September and October 2012, and there is no statistical change in the magnitude of forecast errors for soybean yields over time.  However, there is also a general tendency for soybean forecasts to be conservative, in the sense of under-estimating the final yield.  The downward bias is statistically significant in October and November for the entire sample period and especially pronounced in 2004-2012, when about two-thirds of the forecasts across all four release months under-estimated final yield.  In addition, market analysts consistently under-estimated NASS production during the first half of the sample and over-estimated production during the second half.

• (3) National corn and soybean stocks estimates in the December, March, June, and September quarterly Grain Stocks reports provided by NASS.

• The statistical evidence indicates there has been a sharp decline in market analysts’ ability to anticipate actual quarterly corn usage as implied by NASS Grain Stocks reports since the start of the 2006 marketing year.  The standard deviation of market surprises for implied corn usage in 2006-2012, 6.2 percent, was twice the standard deviation over 1990-2005, 3.1 percent.  Double-digit implied usage surprises occurred three times during 2006-2012 and each one substantially exceeded the largest surprise observed over 1990-2005.  The most problematic corn stocks estimates occurred in the 2009, 2010, and 2012 marketing years.  Within these three problematic years there was also a clear tendency towards reversal of the surprises from quarter-to-quarter.  In contrast, there was no compelling evidence of deterioration in market analysts’ ability to anticipate actual quarterly soybean usage as implied by NASS Grain Stocks reports over 1990-2012.
• Given the evidence of a sharp decline in market analysts’ ability to anticipate actual quarterly corn usage since the start of the 2006 marketing year, we considered a number of potential explanations for the decline offered by various market participants.  Based on the statistical evidence regarding implied usage surprises we developed four criteria that an explanation needs to satisfy in order to be credible: i) Why corn and not soybeans?, ii) Why 2006-2012 and not earlier?,  iii) Why only in particular marketing years?,  and iv) Why a pattern of reversals during marketing years?  Using these criteria, we showed that all but one of the potential explanations offered to date clearly fails to satisfy at least one of the criteria.  The explanation with the most merit is that unresolved errors in production estimates for corn led to the large surprises.  NASS stocks estimates undoubtedly encompass sampling errors for both production and stocks estimates and it is likely that unresolved sampling errors for corn production estimates are large enough to explain the surprises.  It is more difficult to pin down exactly why unresolved sampling errors for corn production were concentrated in 2009, 2010, and 2012 and caused the quarter-to-quarter reversal pattern in surprises, but reasonable arguments can be put forward.  Nonetheless, it is important to emphasize there is no “smoking gun” in terms of the available evidence on the impact of unresolved production sampling errors, with some of the evidence best described as circumstantial.

Based on our statistical analysis and evaluation, we offer four sets of recommendations regarding USDA corn and soybean forecasts and estimates:

• (1) WAOB corn and soybean yield forecasts:

• While there are no obvious problems with the accuracy of these yield forecasts, the forecasts have been subject to criticism from time-to-time because of changing methodology, perceived inappropriate period for calculating trend, or lack of sensitivity to other potential yield indicators such as crop conditions.  Some of the criticism also probably reflects a lack of understanding of the forecasting methodology.  In particular, some data users don’t appear to be aware of the difference between the WOAB forecasting methodology and that of NASS as reflected in the Crop Production reports issued later in the growing season.  A substantial change was apparently made in 2013 with the adoption of a crop weather regression model as the basis for producing WAOB corn and soybean yield forecasts.  It is not unreasonable to anticipate that this changing menu of forecasting methods creates some confusion on the part of market participants. [emphasis type=”danger”]We recommend that the WAOB describe in a written document the exact process used to determine corn and soybean yield forecasts for each month, including the roles of crop weather regression forecasts, subjective judgment, and any other inputs, and this document be available on the WAOB website and explicitly referenced and hyperlinked in the footnotes of the relevant supply and demand tables in May, June, and July World Agricultural Supply and Demand Estimates (WASDE) reports.[/emphasis]

• (2) NASS corn and soybean yield forecasts:

• There is no evidence of problems with the accuracy of NASS corn yield forecasts, which have actually shown improved accuracy in recent years.  However, a tendency for soybean forecasts to be conservative, in the sense of underestimating final yield, has developed in recent years.  The magnitude of the bias, while not large is also non-negligible.  The downward bias in soybean yields has also led to market analysts consistently being surprised in the opposite direction.  Consequently, soybean forecasts by market analysts have been more accurate than NASS forecasts at times, but market prices may have been nudged in the wrong direction given the benchmark status of NASS forecasts. [emphasis type=”danger”]We recommend that NASS institute an internal review of soybean yield forecasting procedures to determine the source of any bias and make changes needed to insure it is eliminated.[/emphasis]
• Our review also highlighted the ongoing problem of widespread misunderstanding and confusion about the forecasting procedures used by NASS to generate corn and soybean yield forecasts.  While this problem likely can never be fully solved, and we are well aware of and applaud the ongoing efforts that NASS makes to communicate with market participants, we believe more can and should be done to address the misunderstanding and confusion. [emphasis type=”danger”]We recommend that NASS “open up the black box” for each monthly corn and soybean yield forecast as much as possible. This should include: i) presentation of state and national yield forecasts derived from the agricultural yield survey (AYS) and the objective yield survey (OYS), as well as the usual composite forecast derived from the two surveys, ii) presentation of assumptions regarding fruit weights used in deriving OYS yield forecasts during forecast months when these measurements are not available, and iii) some form of recognition of the degree to which weather and crop condition data influence composite forecasts.[/emphasis]
• We understand that these changes would represent a major shift in disclosure for NASS and that arguments can be made that forecast users prefer more informative point estimates to a wider but more accurate range of competing forecasts or that releasing the underlying AYS and OYS forecasts would only sow more confusion.  Nonetheless, it is our view that most market participants would welcome this additional information and it would greatly improve not only the understanding but the usefulness of NASS forecasts.  The recommendation is especially important with respect to publishing yield forecasts based on the AYS and OYS surveys.  Few market participants understand the crucial role that compensating biases play in the historical accuracy of NASS corn and soybean yield forecasts.  We see no reason not to explicitly acknowledge this part of the forecasting process, especially in light of the recent publication of historical data on the biases in this publication. Finally, recent studies have developed new methods for combining the AYS and OYS survey yield forecasts along with information based on weather and crop conditions.  To date, our understanding has been that NASS corn and soybean yield forecasts did not consider either weather or crop conditions data in determining final yield forecasts.  Since one of the co-authors of these studies is the Chief of the Statistical Methodology Research Branch in NASS, the research obviously has the official sanction of NASS.  If NASS has changed its procedures for determining final published yield estimates or is considering a change in procedures for the future, it is very important that market participants be made fully aware of the change.
• A review of NASS corn and soybean yield estimates would be incomplete without some discussion of the ongoing technological revolution widely known as “precision agriculture.”  One part of this revolution is the widespread use of yield monitors to provide “real-time” data on grain yields as crops are harvested.  While there certainly are issues with the accuracy of this source of yield data, there is no reason not to expect the accuracy and availability of this data to increase substantially over time.  We are unaware of any research in the public domain about how this potentially valuable source of yield data could be used in the future to improve the accuracy of NASS crop yield estimates, as well as potentially increasing the frequency of estimates. [emphasis type=”danger”]We recommend that NASS initiate a research project to study how yield monitor data could be incorporated into crop yield estimation procedures.[/emphasis]

• (3) NASS corn and soybean stock estimates:

• We thoroughly reviewed NASS survey procedures for stock estimation and determined that off-farm survey respondents are instructed to report the number of 56-pound bushels of corn and 60-pound bushels of soybeans, while similarly specific instructions are not provided to on-farm respondents. [emphasis type=”danger”]To improve the accuracy of on-farm stock estimates, we recommend that the same instructions regarding weight per bushel that NASS provides to off-farm survey respondents also be provided to on-farm survey respondents.[/emphasis]
• Statistical evidence indicates there has been a sharp decline in market analysts’ ability to anticipate actual quarterly corn usage as implied by NASS Grain Stocks reports since the start of the 2006 marketing year.  We carefully vetted numerous explanations that have been offered by market participants for the decline and found all but one lacking in terms of one or more criteria.  The explanation with the most merit is that unresolved sampling errors in production estimates for corn led to the large surprises. [emphasis type=”danger”]We nonetheless recommend that NASS initiate an internal review of corn stock estimation procedures in an effort to determine whether methodological problems are apparent.[/emphasis]
• We recognize that this (and any other reviews) may require additional resources.  In addition, we recognize that given the difficulty involved in tracking the source of corn stocks estimation errors, full resolution may require some form of “ground-truthing” project whereby on- and off-farm measurements are cross-checked versus quantities reported for quarterly surveys.  This could be a very expensive undertaking. [emphasis type=”danger”]As an alternative, we recommend that NASS investigate the possibility of adding grain stocks questions to the Agricultural Census.[/emphasis]. This would at least provide the opportunity to assess the accuracy of on-farm grain stocks estimates once every five years.
• There is also an education issue raised by our analysis of stock/usage surprises for corn.  It appears there may be limited understanding among market participants that NASS grain stocks estimates encompass sampling errors in both production and stocks, which means that considerable caution needs to be used when deriving the implications for usage of a NASS point estimate of stocks.  Our analysis shows just the opposite for corn, in that stocks surprises of only a few hundred million bushels have at times changed corn prices by as much as 10 percent. [emphasis type=”danger”]We recommend that NASS engage market participants in a discussion of this particular issue and consider what means might be available to improve the general understanding of the limits of stock estimates for implying usage.[/emphasis]

• (4) Domestic usage estimates:

• Implied usage surprises in soybeans over 2006-2012 were unquestionably less problematic than in corn, and we believe the evidence strongly suggests this can be traced to the ability of NASS to revise January soybean production estimates after the end of the marketing year.  NASS has not historically revised the previous year’s production estimate for corn in association with the September stocks estimates.  Since feed usage of corn is not measured, implied feed and residual use of corn in the fourth quarter of the marketing year does not provide enough information to quantify a change in the production estimate for the previous year.  Regardless, it is reasonable to expect that January corn production estimates contain errors of a similar percentage magnitude to soybeans, given that a very similar estimation procedure is used for both crops.  However, the corn revisions (estimation errors) are never realized because the accounting for corn feed and residual usage is not precise enough to provide NASS with a reliable basis for making the revisions. [emphasis type=”danger”]We recommend that the WAOB and NASS evaluate the potential costs and benefits of adding a survey of corn feed use that would allow a fuller accounting of corn usage similar to what has been historically possible for soybeans.[/emphasis]
• We believe this is the single most important change that NASS should consider implementing in light of our report.  The need for a feed usage survey in corn has long been discussed but never implemented due to the perceived expense and complexity of such a survey.  Westcott and Norton (2012, p.5) recently noted the practical challenges presented in developing a feed survey for corn, “USDA does not survey for grain used in feeding and relies on survey-based production and quarterly stocks estimates to indicate grain disappearance.  Surveying for feeding would require a substantial increase in resources.  Such a survey might also provide impractical, given that vast array of feeding operations that make the Nation’s swine, beef cattle, dairy, broiler, layer, and turkey production sectors.  These operations are diverse and use a wide and complex variety of feed production in addition to grains.  There is also a significant amount of feed use of grain that occurs outside the conventional livestock, dairy, and poultry sectors and includes feeding of horses, household pets, and aquaculture, all of which present additional challenges to any comprehensive survey of grain used in feeding.”  While we concur that the challenges appear to be daunting, we also note that consolidation of the livestock production industry in the last 20 years may allow a feed usage survey to be implemented in a more practical and cost-effective manner than is commonly perceived.  For example, a useful data series could be based on a sample of larger operations from each of the major livestock species. This would not provide an estimate of total feed usage of corn but the series would be a valuable indicator of changes over time. In any event, if it is deemed important to improve the ability of market participants to better anticipate and interpret NASS corn stock estimates, some type of a corn feed survey is likely to be the only way to accomplish this goal.
• Another issue related to accounting for corn usage is the average yield of ethanol per bushel of corn.  As we noted in the body of the report, variation in the industry average ethanol yield of corn is not likely to be important in explaining corn usage surprises in recent years.  Nonetheless, better estimates of corn used in ethanol production would contribute to the overall accuracy of corn supply, demand, and price forecasts generated by the WAOB. [emphasis type=”danger”]We recommend that WAOB and NASS investigate the costs and benefits of adding a survey of ethanol plants to provide more accurate estimates of corn used in ethanol production.[/emphasis] Compared to a feed survey, the cost of surveying ethanol plants to gather this data would be relatively low given that only 211 ethanol plants are currently operating in the U.S.  A good model for this type survey is the U.S. Energy Information Administration’s (EIA) monthly survey of biodiesel production.
• The full accounting of usage in soybeans in the past has depended on monthly Census Bureau reports of the activity of the domestic soybean crushing industry.  Since the discontinuation of the Census Bureau reports in July 2011, National Oilseed Processors Association (NOPA) estimates have been used to gauge the rate of the domestic crush, the strength of soybean and product demand, and to anticipate NASS quarterly soybean stock estimates.  As time progresses, the historic relationship between NOPA and Census crush estimates may no longer provide an accurate estimate of the domestic crush on a quarterly basis now that the Census crush estimates are no longer available.  This may add to the difficulty of anticipating NASS soybeans stocks estimates. [emphasis type=”danger”] We recommend that WAOB and NASS seek funding to replace the former monthly Census Bureau M311J Fats and Oils: Oilseed Crushings report.[/emphasis]