Sugi-90-187 Friend A..
Contents
1. Ratio lt 0 Ratio gt 2 Revisions failing within each class may be interpreted in the following manner Ippolito 1979 A ratio equal to 1 indicates a perfect revision equal to the forecast error in the original time period A ratio between 1 and 2 indicates a revision in the correct direction but too large The closer the value is to 1 the better the revision A ratio between 0 and 1 indicates a revision in the right direction but too small A ratio equal to zero means that no revision was made while a negative ratio indicates that the revision was in the wrong direction A ratio greater than or equal to 2 means a very bad revision was made Note that 1087 large revision ratios may result whenever the initial forecast itself was a very good forecast even for revisions made in the right direction Prediction Realizations There are several steps involved in computing prediction realization statistics the first of which is a call for an additional year of input data In order to eliminate seasonal bias computations are made using the actual values of one time period and the values of the same relative time period of the previous year hence the need for the additional data To reduce the possibility of misunderstanding the nature of the calculations the input data including the additional year are printed whenever prediction realization values are requested Given a frequency of observation f 4 for quarterly data2. PREDICTION REALIZATION BESCHE BESCPUI3 BESCPUIG PERIOD ACTUAL PREDICT PREDICT2 1981081 5 9099 HS 5 7348 1381802 3 50483 2 87564 6 60444 19818833 2 91665 4 47673 1 09584 1980004 1 84777 1 52133 1 52199 1982881 1 906013 1 15887 8 19781 198288 1 32475 1 5634 2 94388 1902003 3 41094 2 86952 1 96715 2 48971 2 16564 18 1764 SAS 18 89 Thursday March 15 1998 4 THEIL STATISTICS FOR BESCPUS BESCPUI3 STAT VALLE Ut 68 3817203595 V2 8 85 9893678 18 134 SAS 10 89 Thursday March 15 1998 11 REVISION RATIO ANALYSIS TABLE BESCPUS BESCPUI3 BESCPUIG COLE cotz Ratio 1 8 iCRtio 2 1 8CRatioC 1 4 Ratio lt 8 2 Ratio 2 1 200 18 18AM SAS 18 89 Thursday March 15 1938 14 DIRECTIOMAL ACCURACY CRITICAL VALLE IS 1 8 BESCPUS BESCPUI3 BESCPUIG PERIOD PEEDICTI PREDICT 1981601 8 B 1981862 H H 1381883 8 Z 1381984 H H 1982801 2 1 1992082 8 8 1932003 D a 1382884 8 1 1090
3. function which had been served by a Macro Library on our mainframe system Until then we had not had a mechanism for sharing code among users in the PC environment It was decided to use the EVAL system under development as a prototype application to be incorporated within the SAS ASSIST framework to serve as an example for others interested in developing such multi user projects 1085 2 Development of EVAL as an ASSIST application The first step in linking an application to SAS ASSIST is to tell ASSIST where to find it This is done by editing or creating if it does not already exist a data set called SASAPPL in the SASUSER directory which indicates the catalog where the application is filed Directions for doing this can be obtained by choosing APPLICATIONS from the SAS ASSIST Primary Menu and choosing HELP on the next Menu that appears In terms of writing the SCL code to provide the front end to the application we found it easiest to adapt the programs already included in SAS ASSIST While in general it is often more difficult to decipher someone else s code than to write your own in the case of the SAS AF software used to develop ASSIST we found it to be a good learning experience to go through the programs that already existed In that way we could see the results of certain programming techniques and learn how to incorporate some of the existing programs for general functions such as puiling in the variable list of a data
4. 12 for monthly and so on a prediction realization is computed as the percent change between the actual or forecast value at time t and the actual value observed f periods one year previously That is RA e t f PRedR 100 A A PRedR 100 LEEF t f A directional accuracy table which relies upon the PR calculations is also available through the PR request However you cannot get a directional accuracy table without the accompanying prediction realization table This was built into the system to reduce possible misinterpretations of the resuits The directional accuracy table is an interpretation of the prediction realizations and it uses a critical value specified by the user used to set a range within which changes are considered to be essentially zero for the purpose of directional accuracy Predicted and actual values that fall within the range CriticalValue 2 CriticalValue lt PRedR lt 2 are considered as no change If no critical value is specified a default value of 1 0 is used The possible values in the directional accuracy table are 0 1 and 2 and they should be interpreted in the following manner a value of 0 indicates a correct directional prediction 1 indicates an incorrect directional prediction and 2 indicates a truly bad directional prediction Since no change is usually considered a direction in forecasting analysis there are two steps from a correct forecast to a horrible fo
5. EVAL An Application of SAS ASSIST Software to Forecast Evaluation Philip E Friend and Linda P Atkinson Economic Research Service USDA Abstract SAS ASSIST software currently distributed with base SAS software of the SAS System for personal computers gives applications developers a platform from which to use SAS Software to build systems for end users This example demonstrates such a system SAS ASSIST is used as a user friendly front end for a forecast evaluation system Full screen menus guide a user into developing a processing request without needing to have a knowledge of SAS syntax Help screens and automatic validation techniques provide assistance as the user pulls data in from SAS data sets DOS files or keyboard entry and then chooses from groups of forecast evaluation statistics including descriptive measures such as mean square error Theil forecast error statistics forecast revision ratios and directional accuracy tabies 1 Introduction Analysts at USDA s Economic Research Service are frequently called upon to compare and evaluate alternate sets of forecasts such as predicted scenarios concerning agricultural production prices and so on The EVAL Forecast Evaluation System was developed at researchers requests for a comprehensive easy to use package for evaluating price forecasts Such a system was developed and was in use during the 1970s but it fell into disuse for several reasons The database which
6. corporated into ASSIST As such a parameters data set had to be created to pass to the macros information about the data and variables to be used and the analysis desired To store values in the macro symbol table when the data step was executed required use of the SYMPUT routine All of this was no longer necessary when EV AL was moved into ASSIST Screen Control Language SCL variables defined as fields on the EVAL display panel pass information obtained through interactive user input to the macros which perform the requested computations 3 Hardware software and data requirements The software requirements for EVAL to run on an IBM or compatible personal computer are SAS version 6 03 or higher with the following products installed SAS BASE SAS STAT SAS ML SAS ETS and SAS ASSIST Extended memory is necessary two megabytes is suggested SAS AF is needed to develop an application and link it to SAS ASSIST however it is not needed to run it since the programs are compiled Although EVAL was developed with a particular data base in mind we have tried to keep the requirements as general as possible The SAS data set which is to be analyzed must have one or more variables containing actual values of series of interest and one or more variables containing forecast values which are to be compared to the actual observed data In addition there must be a variable to indicate the time period represented by each observation It sh
7. d forecasted values with M and B estimated by PROC REG in SAS STAT For perfect forecasts F A so that the resulting regression would have zero intercept B and unit slope M 1 Parameters of the regression would be tested to see if they differed significantly from zero and one Kost 1980 Revision Ratios table and a Revision Ratio Analysis table These are only appropriate for examining forecast variables from the same source for the same actual variable over the same time period when the forecasts are made at different times Thus more than one forecast variable name must be specified The revision ratios are a measure of how well a forecast is revised as the forecast is made at a time closer to the period being predicted The revision ratio between two forecasts F and F where Fs is the forecast made later is calculated as the ratio of the difference between the two forecasts the revision and the forecast error the difference between the original forecast and the actual observed value That is A forecast made 3 months ago may be a good bad or indifferent revision of a forecast made six months ago for example The revision ratio analysis table summarizes the revision ratios for the specified time period It provides a count of the Revision ratios which fall into each of five categories for each of the ordered pairs of forecasts The classes of revision ratios are Ratio 1 1 lt Ratio lt 2 0 lt Ratio lt 1
8. he regression specification screen also had a number of Yes No options for analyzing residuals etc We modified these to indicate options for EVAL analysis such as computing Theil statistics By seeing how these switches were implemented we were then able to add additional options move things around on the screen and so on Some references to other ASSIST programs expected them to be in the current catalog so we needed to make their calls more explicit For example call display customiz program was changed to cal display sashelp assist customiz program In terms of adding Help screens for the forecast analysis specification screen in general and for particular fields it was noticed that in SAS ASSIST CBT entrics were used instead of HELP It turns out that with a CBT window displayed a user only needs to press ENTER to exit rather than needing to know to type END This seemed more intuitive for a novice user so we also coded CBT type entries in our catalog As we became more accustomed to using SAS AF software we also designed an introductory panel of information for the system and a preliminary menu before bringing the user into the main EVAL specification screen A Dam Management option branches the user to the same place that choosing DATA from the SAS ASSIST Primary Menu would again by a call to the same program sashelp assistdatmenu program EVAL was first coded as a set of SAS macros before being in
9. ons uhich include any data editing you vould lihe to do or ruming the Forecast Evaluation program Put an X by the appropriate option and press EXTER Once in the menu for EVAL you will nove your cursor around the screen to fill in such information as the data set you uish to analyze the variable uhich contains the actual values for the series of interest the variable or variables uhich contain forecasts you would like to conpare to the sctual values and the types of analyses you wish to perform Press PFt at any point at Wich you reed help Press ENTER to continue IST Application Select jonas 10 IN Select the application you uant to run Place any character 1n the field next to the application description Press ENTER to run the selected application use GOBACK to return plication Selectio Qu WI Privata Applications _ FORECAST EVALUATION SYSTEM Forecast Evaluation Syste 18 1268 variable of actual values is required ENTER to make a selection ACTIVE data set AGDATA PRICES DATE indicator TIME FREQUENCY of data INITIAL TIME PERIOD ACTUAL value variable LAST TIME PERIOD FORECAST variables DISPLAY input data HG TEIL statistics M REGRESSION of actual en predicted values W REVISION ratios M PREDICTION realization and directional accuracy statistics NO Critica value for directional accuracy computations 1 8 ASSIST Prinary fenum lf 116A SAS ASSIST P
10. or on ACTIVE data set and pressipg ENTER and then selecting from a list of data sets presented according to any libraries specified in ASSIST SAS Also by default the program expects to use a variable called TIME as a date indicator The user can override this by positioning the cursor on DATE indicator pressing ENTER and selecting from the variables shown By working within the ASSIST framework the user also has available all of the other menus for commonly performed tasks within the SAS System If graphics or tabular displays are wanted these can be obtained by making appropriate choices from the SAS ASSIST Primary Menu before or after choosing APPLICATIONS 5 The Statistics The statistics which the current version of EVAL can calculate and display follow Printing of the input data Each analysis group is specified by a dataset name initial time period to be analyzed last period to be included in analysis Actual variable name and Forecast variable names PROC IML is used to retrieve the data and PROC PRINT to display it Statistics of Fit These include the mean error mean percent error mean absolute error mean absolute percent error root Mean square error and root mean square percent error These are generated by PROC SIMLIN Theil statistics There are many Theil inequality coefficients in use The ones EVAL calculates are based on relative change forecast errors as recommended in for example Leuthold 1975
11. ould be a numeric variable of the form yyyyppp with a value of 1979004 for example indicating the fourth period month quarter etc of 1979 in the future this may be generalized to handle true SAS date variables 4 Running EVAL To run the EVAL Forecast Evaluation System a user moves the cursor on the SAS ASSIST Primary Menu to APPLICATIONS and presses ENTER The next screen provides a choice of PUBLIC which we interpreted to mean applications provided by SAS Institute since they currently distribute some SAS OR examples with SAS ASSIST or PRIVATE which is where we put EVAL an application developed at our site After choosing PRIVATE the user secs a list of applications of which currently EVAL is the only one Placing an X by Forecast Evaluation System and pressing ENTER places the user on another screen with options for running the Forecast Evaluation Program data management and exiting After indicating that EVAL should be run a final screen is presented for specification of such information as the data set to be analyzed the variables to be 1086 used the time periods to limit the analysis to the periodicity of the data monthly quarterly etc and the types of analyses to be performed Context specific help is available by pressing PF1 while positioned on any field By default the last analyzed data set is presented on the screen as the one to be used for EVAL The user can change this by positioning the curs
12. recast hence the three possible values as explained above 6 Anticipated enhancements EVAL is an application under development rather than a finished product Several additional tests that EVAL should perform have already been discussed with economists at USDA ERS They include Fisher Forecast Evaluation Fisher Forecast Comparison Wilcoxin Signed Rank Kendall Coefficient of Concordance and the Mincer Zarnowitz test The scope of the system will thus be expanded to allow comparisons of forecasts from different sources as well as evaluations of forecasts In order to improve performance somewhat and to eliminate the need for the presence of SAS ETS and possibly SAS STAT when running the application we intend to recode several EVAL modules The menus help screens and flow of the system will be altered as suggestions are made and time and resources allow Output will be improved as needs or complaints demand 7 Condusion We hope that EVAL will serve as a prototype application in SAS ASSIST so that others can use it as an example for sharing applications of general interest This framework Should be a powerful vehicle for disseminating and supporting such programs so that we can develop a good library of useful procedures References Granger C W J and P Newbold Some Comments on the Evaluation of Economic Forecasts Applied Economics 5 1973 Ippolito Pauline and Linda Lynn Forecast Evaluation System EVAL U
13. rimary Menu pplicatio Tab and press EATER to select Gh BN Type of Application PUBLIC PRIVATE FORECAST EVALUATION SYSTEM MAIN HEHU Place an X next to your selection Rum Forecast Evaluation Progran Data Hanagenent select edit etc if not already done Return to Previous Menu L Forecast Evaluation Syste 18 14AK Tab and press ENTER to select ACTIVE data set AGDATA PRICES DATE indicator TIME FREQUENCY of data 4 INITIAL TIME PERIOD 1991001 ACTUAL value varlable BESCPUS LAST TIME PERIOD 1982094 FORECAST variables RSR BESCPUI6 DISPLAY input data VES THEIL statistics VES REGRESSION of actual on predicted values VES REVISION ratios YES PREDICTION realization and directional accuracy statistics YES Critical value for directional accuracy computations 1 8 1089 Appendix B Selected EVAL Output SAS 18 83 Thursday March 15 1998 THEIL ENDOGENOUS BESCPUS EXOGENOUS BESCPUI3 SIILIN Procedure Statistics of fit Mean Mean Mean Abs Mean fbs Variable Mo Error Error Error X Error Error BESCPUS 8 43759 8 077 117 9158 2 13949 157 1531 8 180 SAS 18 89 Thursday March 15 1938 10 REVISION RATIOS FOR BESCPUS HESCPUI3 BESCPUIG PERIOD nz 1981000 13 6304 138182 8 3738 135183 1 3883 1381084 6 0000 1982881 8 7835 1982882 8 8523 198293 0 6258 1382004 8 8475 SAS 18 89 Thursday March 15 1398 13
14. ser s Manual Data Services Center Working Paper USDA ESCS March 1979 Kost William E Model Validation and the Net Trade Model Agricultural Economics Research Vol 32 No 2 April 1980 Leuthold Raymond M On the Use of Theil s Inequality Coefficients Amer J Agr Econ May 1975 p 344 346 Pindyck Robert S and Daniel L Rubinfeld Econometric Models amp Economic Forecasts McGraw Hill 1981 SAS Institute Inc SAS AF User s Guide Release 6 03 Edition 1988 SAS Institute Inc SAS AF Software Applications Using Screen Control Language Course Notes 1989 Theil Henri Economic Forecasts and Policy North Holland N Y 1961 Thomson James M Analysis of the Accuracy of USDA Hog Farrowings Statistics Am J Agr Econ December 1974 SAS SAS AF SAS ASSIST SAS ETS SAS IMI SAS STAT and SAS OR are registered trademarks of SAS Institute Inc Cary NC USA 1088 Appendix A Selected EVAL Screens The Forecast Evaluation System has been coded as an application in SAS ASSIST To access it nove the cursor to APPLICATIONS on the ASSIST main wa ard press EXTER The next screen uill give you a cholce of PUBLIC applications provided by SAS Institute or PRIVATE applications developed at our site Choose PRIVATE You will then see a list of applications to select fron Ccurrently EVAL is the only one Put an X by Forecast Evaluation Systex and press EATER You uill be given another fist of opti
15. set without having to reinvent the wheel ourselves In order to modify parts of SAS ASSIST you must compile a set of macros that are used throughout ASSIST We copied MACROS PROGRAM from the SASHELP ASSIST catalog into our own catalog EVAL ASSIST and added the following to our ASSIST SAS file dm af c eval assist macros program This made the macros available to us during development work When we were not planning to make changes we would comment out this line before running the application leaving it in the file in case we needed to make later modifications As a starting point for the specification screen for EVAL on which a user would indicate details for an analysis to be performed we looked for a program in ASSIST that did something similar In the REGRESSION option chosen from STATISTICS on the SAS ASSIST Primary Menu the user is presented with a panel on which to indicate a dependent and one or more independent variables to use in a regression This was similar to our framework of needing to specify an Actual variable of observed valucs and onc or more Forccast variables to compare to it We copied SASHELP ASSIST STREG PROGRAM into EVAL ASSIST FCST PROGRAM and edited it to change the labelling to suit our purposes This gave us already defined options along the top of our screen for providing HELP electing to RUN the SAS program generated saving the program customizing the output and going back to a previous screen T
16. to overcome sensitivity to an additive transformation The calculations are performed in PROC IML according to the following gt F A NE gt GF ALA a 4 AAt t c n by F A 2 taa X L AL d t 2 These statistics compare the predictive ability of the model generating the forecasts to the naive no change model where the forecast for the next period is this period s actual Observation Both U1 and U2 have a lower limit of 0 which refers to an ideal forecast F A thus the smaller these statistics the better For U1 the upper boundary is 1 when F A so that all models appear to predict at least as well as the no change model The U2 siatistic removes this limitation so that its interpretation is U2 0 perfect forecast 0 lt U2 lt 1 forecast is better than naive model U2 1 naive no change extrapolation U2 gt 1 forecast is worse than naive model Leuthold 1975 Note In some cases the computed values of the Theil statistics differ from those that would have been produced by the SAS procedure PROC SIMNLIN We have been unable to duplicate by hand calculation the values that PROC SIMNLIN prints Regression of actual on predicted values y Mx B as a method of testing goodness of fit Here y is the endogenous actual variable A x is the exogenous forecast variable F and M and B are the slope and intercept respectively The solution is the result of an ordinary least squares fit of the actual an
17. was the source of the price data and with which the system interfaced was replaced with a new database and the raw Fortran code in which the system itself was written was lost When researchers requested a replacement for the system advantages and disadvantages of several languages and packages were considered and the SAS system with its SAS ETS econometric software and SAS IML matrix language software seemed to offer the most power with the most flexibility for the least code In addition the multi platform development which SAS is following and the Institute s famous support suggested that a forecast evaluation system written in SAS would bc thc most accessible to users with a variety of forecast evaluation problems While some of the forecast evaluation measures desired were available in various places in SAS procedures others were not We decided to develop a comprehensive package by combining forecast evaluation tools from SAS procedures already available with statistics calculated by macros using SAS IMI software We wanted to make the system flexihle enough so that newer more accurate evaluation techniques could be added as they are developed by econometricians At about the same time an update to the SAS ASSIST front end to the SAS System added an Applications option to its Primary Menu It was envisioned that this could serve as the vehicle by which applications of general interest could be shared within the Agency a
Download Pdf Manuals
Related Search