Home

RAND`s Portfolio Analysis Tool

1. Weight 0 25 0 25 0 5 0 25 0 25 0 5 High or Low values High High High High High High desired Threshold value 2 2 2 2 2 2 Goal value 7 7 7 7 7 7 Level 2 value for 0 0 0 0 0 0 Threshold Level 2 value for Goal 1 1 1 1 1 1 Option A Weighted score Option B Weighted score Option C Weighted score Details of the Methodology 69 Rankings Method In our example the summary and effectiveness weights are identical with the third submea sures of each measure having twice the weight of the other two as shown in Table 5 11 The average rank and measure score calculations are compressed slightly and all values are rounded to two decimal places For the measure score the sum of the submeasure scores is divided by the product of the maximum submeasure score per measure 2 and the sum of the weights 1 The effectiveness value is just the average of the measure scores Because Invest ment Option C has the best value for four of six submeasures it is clearly the best option under the Rankings method Table 5 11 Illustrative Results for the Rankings Method Measure Weight High or Low values High High High desired Threshold value Level 2 value for Threshold Level 2 value for Goal rank x weight score x weight Avg Effec rank tiveness it i a af af rank x weight aa score x weight
2. 2 500 2 000 Option A Option B Option C A H u o oO T 1 000 Cumulative Total Cost M 500 0 Ot 2010 2011 2012 2013 2014 2015 Year NOTE The example uses current i e then year dollars but the user can specify use of constant dollars instead 4 An example of this complexity might be a set of Level 3 measures related to risk PAT Output Worksheets 45 Scatter Plot Sheet for Cost Effectiveness Landscapes PAT s built in Scatter Plot sheet allows the user to construct cost effectiveness landscapes i e plots of effectiveness or another measure of performance versus cost The user can choose any one cost metric and any two evaluation metrics from menus on the Scatter Plot sheet The cost metrics on the menu are generated from the cost categories and any specialized cost metrics used on the Summary sheet The evaluation metrics may be any of the Level 2 measures or an effectiveness score calculated by any of the built in methods e g Threshold or Weakest Link The chart presents points representing the options located by cost x axis and effectiveness metric vertical y axis or axes Mousing over a point reveals which option the point refers to These cost effectiveness landscapes are quite useful when evaluating choices with eco nomic constraints They can illustrate both the classic phenomenon of diminishing returns and e g the chunkiness phen
3. Option C vI rank x weight an m CE i i pee P E EEEa pf ie 5 eee i fl o0 vI CHAPTER SIX Marginal and Chunky Marginal Analysis Introduction In marginal analysis small changes to key variables in a system are considered one at a time Marginal analysis is often used to find which variables are most responsible for affecting the outcomes of a system When the key variables are investments marginal analysis helps to determine what the next dollar or million dollars should be allocated to or removed from in order to maximize the capability of the resulting collection of investments Ideally the small changes in each variable are equivalent in magnitude to enable relevant comparisons between options Although marginal analysis can be used for a variety of systems we restrict our discus sion to the marginal analysis of investments Marginal analysis is not appropriate for all situations For example small deviations in investments may have no effect on system outputs This happens particularly when invest ments are tied to purchases of discrete objects An extra dollar invested in the acquisition of a radar system has absolutely no impact when a single component costs thousands or millions of dollars Similarly investments in systems that require a large buy in before they become effec tive have no impact until the buy in is reached at which point there may be a large discontinu ity in
4. or radars interceptors or human capital equipment A word of caution If you have numerous items here the data require ments quickly become very burdensome If you don t want to bother you might replace Item 1 with Stuff or Miscellaneous e List the investment categories for which you want to track costs separately Main is the placeholder The usual choices here might be Total meaning don t bother with categories or e g the items R amp D acquisition and O amp S Blocks A B and C require little explanation but let us elaborate on Block D The syntax for entering data in the first three columns is shown in Table A 1 We assume here that you have three measures as in the text with only the second measure having third level data Measure 1 1 appears to the immediate right of Measure 1 and Measure 2 2 1 appears to the immediate right of Measure 2 2 The name of a measure should not be repeated if the next row is simply filling in a submeasure You of course will want to have more interesting names for your various measures and submeasures The following are some examples of the kinds of measures you might use at the first second and third levels of detail separated by semicolons Force Structure Regions of the World Within each environment shaping versus war fighting within warfighting expectations for different test cases Additional measures coul
5. Davis Paul K James H Bigelow and Jimmie McEver Exploratory Analysis and a Case History of Multiresolution Multiperspective Modeling Santa Monica Calif RAND Corporation RP 925 2001 As of September 2 2009 http www rand org pubs reprints RP925 Davis Paul K David C Gompert and Richard L Kugler Adaptiveness in National Defense The Basis of a New Framework Santa Monica Calif RAND Corporation Issue Paper IP 155 1996 As of September 2 2009 http www rand org pubs issue_papers IP155 Davis Paul K and Richard Hillestad Families of Models That Cross Levels of Resolution Issues for Design Calibration and Management Proceedings of the 1993 Winter Simulation Conference 1993 Davis Paul K Richard Hillestad Duncan Long Paul Dreyer and Brandon Dues Reflecting Warfighter Needs in Air Force Programs Santa Monica Calif RAND Corporation forthcoming Davis Paul K and Reiner Huber Variable Resolution Combat Modeling Issues Principles and Challenges N 3400 DARPA 1992 As of September 2 2009 http www rand org pubs notes N3400 Davis Paul K Stuart Johnson Duncan Long and David C Gompert Developing Resource Informed Strategic Assessments and Recommendations Santa Monica Calif RAND MG 703 JS 2008 As of September 2 2009 http www rand org pubs monographs MG703 Davis Paul K Richard L Kugler and Richard Hillestad Zssues and Options for the Quadrennial Defense Review QDR
6. The names entered under Cost Categories have nothing to do with the rest of the sheet The names entered under Investment Items have nothing to do with the rest of the sheet The combination of Cost Categories and Investment Items dictates the structure of the Cost Data sheet The Build Sheets command causes PAT to generate structures in numerous input and output sheets The user will be prompted by a sequence of query messages that ask whether in rebuilding he wishes to preserve option specific data already in the Level 2 and Level 3 Data sheets the Perspectives data sheet and the Cost Data sheet In many cases saying Yes will save a great deal of time and trouble because data entry is time consuming and prone to errors Saying Yes is appropriate if for example the purpose of the rebuilding is to change the names of some measures or options or to add some additional measures or options In the first case name changing the resulting data sheets will have holes wherever newly named options or measures appear In the second case adding new measures or options the same will be true Obviously data cannot be retained if they do not exist 30 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Hint If the user has made substantial changes in the data structure or if errors have appeared that troubleshooting has not resolved it may be better to rebuild entirely saying No rather than Yes to retaining data
7. e Overall risk derived from technical strategic and political risks Cost effectiveness The ratio of an effectiveness score to a measure of cost Effectiveness A composite score formed by e Combat effectiveness of a force structure based abstracting from the scores of one or on model outcomes for diverse scenarios more measures Measure A way of evaluating something a e The size of attack that saturates a defensive dimension of an assessment similar system to a metric e The likelihood of campaign success in a specified planning scenario Method A procedure used to map raw scores Linear weighted sums into scores or to calculate higher level e Threshold modified linear weighted sums scores from lower level scores e Weakest link Raw value of a An unscaled value of a submeasure e The size of attack that saturates a defensive submeasure system e The number of simultaneous conflicts with which U S forces could deal Relative cost The ratio of an option s cost effectiveness effectiveness to that of the option with the highest cost effectiveness Score A value between 0 and 1 derived from The value of a brigade in comparison with that of raw values and goals to convey a sense a standard brigade of goodness Submeasure One of the factors determining a e The size of attack that saturates a defensive parent measure system in a particular case e g with attacker countermeasures 5 All of PAT s relevant func
8. 5 10 5 11 6 1 6 2 6 3 6 4 AT A 2 A 3 AA Core Built In Aggregation Methods a sscssa1 iets cxasedsawes ekesvaadvciaers ves esveedeczessis 10 PAT Output and Input Sheets i 2 ess ccsacnsseaivedes cxbunnpeesgvicdsweusnnenalntivedseneswcnsets 12 Default Ordered Listing of Tabs for PAT Sheetssarsonmirori i n E N igarixeas 13 A Glossary of PAT Terminology pieier en E n E E E 55 Notation for Defining Scoring Methods i0isi csvavvantumars ienanieivaeitwinssbecadeevenbuies 58 Mapping Measure Scores into Colors uuiisernsiciieiiinipinini ia i baleen 64 Color Coding in the Rankings Method siisccionesknediondsannorerdonins eeiamanieawsaieonmur 64 Summary of Methods neriionninneneip iin EE E E E 65 Illustration of Scoring Methods iscsriicane ina a A 65 Illustrative Results for the Goals Method 5 socsscciseatyesserionsdavesseaieeravesdeeienecsees 66 Illustrative Results for the Thresholds Method 0 0 00 ccceeceeeeeeeeeeeee eee eees 67 Illustrative Results for the Weak Thresholds Method sc0sc0s sserssassvevsenesveveosss 68 Illustrative Results for the Weakest Link Method 0c ccc ceeceeeeeeeeeeeee ees 68 Illustrative Results for the Rankings Method si scccsceisedscnsdsvecees erie eisendeasdaueds 69 Notional Probabilities of Intercept for Illustrative Problem 0 0 00 ceeee eee 73 Performance of Options Probabilities of Intercept by Mission 00000085 73 Costs and Ef
9. THE ARTS CHILD POLICY CIVIL JUSTICE EDUCATION ENERGY AND ENVIRONMENT HEALTH AND HEALTH CARE INTERNATIONAL AFFAIRS NATIONAL SECURITY POPULATION AND AGING PUBLIC SAFETY SCIENCE AND TECHNOLOGY SUBSTANCE ABUSE TERRORISM AND HOMELAND SECURITY TRANSPORTATION AND INFRASTRUCTURE WORKFORCE AND WORKPLACE NATIONAL DEFENSE RESEARCH INSTITUTE This PDF document was made available from www rand org as a public service of the RAND Corporation Jump down to document w The RAND Corporation is a nonprofit research organization providing objective analysis and effective solutions that address the challenges facing the public and private sectors around the world Support RAND Purchase this document Browse Books amp Publications Make a charitable contribution For More Information Visit RAND at www rand org Explore the RAND National Defense Research Institute View document details Limited Electronic Distribution Rights This document and trademark s contained herein are protected by law as indicated in a notice appearing later in this work This electronic representation of RAND intellectual property is provided for non commercial use only Unauthorized posting of RAND PDFs to a non RAND Web site is prohibited RAND PDFs are protected under copyright law Permission is required from RAND to reproduce or reuse in another form any of our research documents for commercial use For information on reprint and li
10. tug bar located at the bottom right hand corner of the display see Figure A 3 3 e Generate the scatter plot of effectiveness versus cost If you have done everything correctly the results should look like those shown in Figures A 4 through A 7 3 Microsoft documentation refers to splitting or freezing panes when discussing these matters Precise terminology may differ in different versions of Excel 90 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Figure A 4 Figure A 3 Tug Bar for Viewing Separated Portions of an Excel Spreadsheet Summary Sheet Excerpt for Exercise Problem Total Cost Effect in Effect in Other 2010 2030 Measures ScenarioA Scenario B Measures M Detail Detail Detail Detail Detail Investment Options 1 1 1 Baselne Option 1 0 Option 2 10 Option 3 80 Option 4 160 Option 5 200 Effectiveness 0 1 0 23 0 23 0 37 0 43 Relative Cost Effectiveness 0 12 0 1 0 09 Quickstart on Using PAT 91 Figure A 5 Level 2 Drilldown for Exercise Problem Effect in Scenario Level 1 Measure B Best Worst Best Case Case Estimate Outcome Outcome Level 2 Measure Outcome Upside Risk Warning Baseline Option 1 Option 2 Option 3 Option 4 ao on Option 5 Technolo Figure A 6 Level 3 Drilldown for Exercise Problem Level 1 Measure Effect in Scenari
11. 1 1 1 1 Option C 1 00 1 00 1 00 1 00 ranking The investment option with the highest absolute cost effectiveness for each perspec tive is given a relative cost effectiveness of 1 with all other cost effectiveness values measured against it The same scaling is used with effectiveness not shown CHAPTER FIVE Details of the Methodology This chapter repeats some of the earlier material from Chapter Two in order to be relatively self contained but its purpose is to generalize lay out the mathematics comment on some of the subtleties of the methodology and work through an example Basic Concepts and Definitions Figure 5 1 indicates how PAT operates analytically except with the Rankings method discussed below under Alternative Methods For each investment option PAT takes a series of inputs grayed items and characterizes how well the option performs by different criteria includ ing cost effectiveness The illustrative figure assumes 7 measures of which only the first and the mth are shown A given measure might be calculated from a combination of Level 2 and Level 3 data as shown for Measure 1 or might be specified directly as shown for Measure 7 The following paragraphs explain the PAT concepts and terminology Attributes of Investment Options The investment options are alternative programs of investment over time Each investment option has a number of attributes described below Input Costs Each PAT in
12. Davis Gompert and Kugler 1996 Davis Kugler and Hillestad 1997 e Strategic planning for the Missile Defense Agency MDA e Acquisition level planning for Prompt Global Strike Davis Shaver and Beck 2008 e Resource informed strategic assessments for the Joint Staff Davis Johnson Long and Gompert 2008 e Mission level analysis for the U S Air Force and DHS Davis Hillestad Long Dreyer and Dues forthcoming Our mid 1990s work was based on DynaRank a decision support system developed primarily by Richard Hillestad at RAND Hillestad and Davis 1998 Miller 2007 A par 2 The primary references for this report s concepts regarding strategic level decision support are Davis 2002a Davis Kulick and Egner 2005 and Davis Shaver and Beck 2008 The latter includes an application of PAT to acquisition issues Another recent publication Davis Johnson Long and Gompert 2008 includes an application to assessment of alternative global strategies Introduction 3 allel stream of RAND research at the time used the objectives to programs methodology An improved tool PAT MD was developed for MDA Dreyer and Davis 2005 along with an integrated capabilities model for missile defense CAM MD Willis Bonomo Davis and Hillestad 2006 RAND s work for MDA motivated the Under Secretary of Defense for Acqui sition Technology and Logistics to request development of a generic version of the approach and to
13. Methods and Reference Manual Figure 2 2 Illustrative Output Types Measures Overall Total Options A B C Effectiveness Cost 1 0 1 12 2 0 7 18 3 0 5 15 a ee Cumulative Costs We next introduce a number of basic concepts and terms Chapter Five provides a more complete and rigorous treatment Concepts and Terminology Multicriteria Scorecards Strategic planning typically requires evaluating options by multiple criteria These criteria may relate to different objectives operations circumstances time scale and so on They may include measures of risk and upside potential Although classic cost effectiveness methods emphasize combining the effectiveness scores for different criteria to obtain a single variable to be optimized modern policy analysis has long emphasized policy scorecards because decisionmakers need to see how the options fare by the different criteria The relative good ness of options may eventually be summarized in terms of a single index or utility but that simplification should follow more discriminate reasoning with multiple criteria The reason for this sequence is that the cross cutting thinking across criteria is precisely what strategic deci sionmakers are often most concerned about and most uniquely responsible for Such thinking is not mere mathematical problem solving Both risk essentially downside potential and upside potential should be considered in good decisionmaking Davis
14. Santa Monica Calif RAND Corporation DB 201 OSD 1997 As of September 2 2009 http www rand org pubs documented_briefings DB201 99 100 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Davis Paul K Jonathan Kulick and Michael Egner Implications of Modern Decision Science for Military Decision Support Systems Santa Monica Calif RAND Corporation MG 360 AF 2005 As of September 2 2009 http www rand org pubs monographs MG360 Davis Paul K Jimmie McEver and Barry Wilson Measuring Interdiction Capabilities in the Presence of Anti Access Strategies Exploratory Analysis to Inform Adaptive Strategies for the Persian Gulf Santa Monica Calif RAND Corporation MR 1471 AF 2002 As of September 2 2009 http www rand org pubs monograph_reports MR1471 Davis Paul K Russell D Shaver and Justin Beck Portfolio Analysis Methods for Assessing Capability Options Santa Monica Calif RAND MG 662 OSD 2008 As of September 2 2009 http www rand org pubs monographs MG662 Dreyer Paul and Paul K Davis The Portfolio Analysis Tool for Missile Defense PAT MD Methodology and Users Manual Santa Monica Calif RAND Corporation TR 262 MDA 2005 As of September 2 2009 http www rand org pubs technical_reports TR262 Dunn William N and Rita Mae Kelly Advances in Policy Studies Since 1950 Policy Studies Review Annual V 10 Edison N J Transaction Publishers 1991 Gates Ro
15. even though this will mean more data entry Errors often are the result of subtle misalignments or typos including extra spaces which can be erased with a fresh rebuilding After the new sheets are generated the user may use Copy and Paste to move data in the old workbook to the new one being careful to paste in the right place A common error is pasting into what appears to be the top left corner only to discover that scrolling had displaced the columns so that the pasting was actually onto interior columns The Show Hide button can be used to toggle on or off some in sheet examples and instructions After Template Builder is run the various input and output sheets will have a good deal of structure built in This does nor include option specific data unless they are retained from the previous version of PAT and it does mot include the setting of control variables at Levels 2 and 3 such as thresholds goals score at threshold and score at goal It also does not include specifying measure weights at any level In some cases PAT will build in default values e g values of 1 for all measure weights and a value of 1 for the score associated with raw values at or exceeding a measure s goal However those may not be what is intended The bottom line here is that even with Template Builder data entry requirements can be considerable and should be undertaken methodically and with proofreading because low level entry errors may go undetec
16. followed by the Ungroup command to move items within a larger block e g one menu item among a group of control panels e Use Excel s custom formats for charts to achieve some consistency Unfortunately such customization affects only some chart attributes so repetitive editing may still be nec essary from chart to chart Further all formatting is lost when charts are regenerated Unclick the Auto Update Chart checkbox to avoid losing chart formatting when switch ing between sheets e To annotate charts with text box notes e g labeling curves or specifying assumptions consider copying and pasting an entire set of notes into an extra worksheet so that they can be retrieved and reused Regenerate the chart copy and paste the previous notes and then move them around as appropriate e Do not hesitate to use custom worksheets to do normal Excel charting drawing upon PAT s input or output data as needed In some cases PAT s built in charts are not what the user is seeking and it is counterproductive to try to work around them l We modified PAT to provide compatibility with Excel 2007 but our experience with Excel 2007 is limited and other side effects of Microsoft s major changes may occur In particular we suggest that users avoid moving menus and perhaps other spreadsheet objects or at least save before attempting to do so And as mentioned earlier PAT will not work in Excel 2008 Mac Macintosh users should ret
17. perspectives can be defined only with respect to Level 2 data except through the method of extended perspectives discussed below Figure 3 5 Illustrative Perspectives Cases Level 1 Measure Level 2 Measure Weight of Level 2 Measure in Scoring Functions 0 to 1 High or Low Values Desired Threshold Value Goal Value Level 2 Measure Score for Threshold Value 0 to 1 Level 2 Measure Score for Goal Value 0 to 1 Level 1 Measure Level 2 Measure Weight of Level 2 Measure in Scoring Functions 0 to 1 High or Low Values Desired Threshold Value Goal Value Level 2 Measure Score for Threshold Value 0 to 1 Level 2 Measure Score for Goal Value 0 to 1 Level 1 Measure Level 2 Measure Weight of Level 2 Measure in Scoring Functions 0 to 1 High or Low Values Desired Threshold Value Goal Value Level 2 Measure Score for Threshold Value 0 to 1 Level 2 Measure Score for Goal Value 0 to 1 Level 1 Measure Level 2 Measure Weight of Level 2 Measure in Scoring Functions 0 to 1 High or Low Values Desired Threshold Value Goal Value Level 2 Measure Score for Threshold Value 0 to 1 Level 2 Measure Score for Goal Value 0 to 1 o 0 0 0 1 000 oojoo I ire 5 000 ooo 1i p 2 000 ooo ao Measure 2 Measure 2 Measu easure 2 2 Warning Measu igh High 0 0 iMeasure 2 Measure 2 Measu iMeasure 22 Warning Measu 1 0 igh High 0 0 1 0 0 0 1 0 Measure 2 Measur
18. treating the baseline as 0 We see that Option 2 is by far the most cost effective right column but is still not satisfactory as indicated by the yellow and red cells of the scorecard To do well however Option 5 is quite costly As noted above the values asserted for the aggregation of overall effectiveness depend not only on the assessments for each individual measure but also on how those assessments are combined Assumptions can be changed readily in either sensitivity analyses or more com prehensive exploratory analyses in which all key variables are varied simultaneously to explore what has been variously called the scenario space outcome space or possibility space If such wide ranging exploration has been accomplished first whether crudely or in detail the sce narios used to assess the options can be carefully defined analytically to stress the options in all of the ways regarded as important i e they may constitute an approximate spanning set Current day official planning scenarios do not typically have this virtue but they could have it in the future The number of test scenarios of course may be much larger than the number of scenarios in the illustrative examples in this report at least at the working level before sim plifications are introduced for communicating efficiently to leadership Cost Effectiveness Landscapes as a Function of Perspective Displays such as Figure S 3 include a great deal of informatio
19. 12 shows a condensed version of the scorecard portion of the Summary sheet for our simple example We have chosen to display only the answers plus the measure weights ignor ing the other features of the scorecard The red rectangle indicates that we will drill down for more information on Measure 2 Figure 4 11 Warning Flags in a Summary Sheet Measures Measure 1 Measure 2 Measure 3 Detail Detail Detail Investment Options gel Risk Some believe that M1 1 s value could be only 0 3 Option A Option B PAT Output Worksheets 39 Figure 4 12 Illustrative Summary Scorecard Level 1 Measures Measure 1 Measure 2 Measure 3 Detail Detail Detail Investment Options Option A Option B Level 2 Drilldown Sheet Clicking on the Detail button for any particular measure in the Summary scorecard brings up the Level 2 Drilldown sheet as Figure 4 13 illustrates for Measure 2 This sheet includes a scorecard showing the scores of all contributors to the measure s effectiveness and on the right side the calculated consequences which are the same as the column of scores or colors shown in the Level 1 scorecard for Measure 2 reading downward green yellow orange In more typical cases the scorecard may have from three to seven Level 2 measures and will therefore be more complex The default is that Level 2 drilldown is shown for the same settings as specified
20. 2010 2011 M Acquisition Cost 2010 2011 M O amp S Cost 2010 2011 M To specify a single year one should use just that year rather than a range i e 2010 rather than 2010 2010 2 The M is not included in the cost category name but is added separately The names being added cannot be simply copied and pasted from the Cost Data sheet PAT Output Worksheets 37 Figure 4 10 illustrates this example showing the cost columns of the Summary sheet The columns for the 2010 2015 interval would be generated automatically but the column for the period 2010 2011 has been added as mentioned above by typing in Total Cost 2010 2011 M PAT understands the word Total to mean the sum of the costs over the specified cost categories which in this case are R amp D acquisition and O amp S To show all categories of costs for two years rather than the six years the user could merely edit the column headers changing 2010 2015 to 2010 2011 Adding or Deleting Options Analysts using PAT will frequently enter data for more options than can reasonably be dis played However minds change and it is often necessary to add or delete an option in the Sum mary display This is done simply by inserting a row within the scorecard range and typing in the precise name of the option in Column A Cost Effectiveness The two rightmost columns in Figure 4 1 are labeled by PAT as Effectiveness and Cost Effectiveness The method used to calc
21. Abbreviations oie cn tested cain aea E EEE EEE EEEE EEE EEE EEEE E EENE EN xxiii CHAPTER ONE Tntrod ction ccd seen Ani a A E A EE E AAO E EORNA ASA 1 PR OOM inaccate sts vat Grinaiecing i E EE E E TE R GEE EE 1 Strategic Planning and the Balancing of Iayestiientsy j 4cescchdegasnahsacasonb dues enneagaenerdedunens 1 Dealing with Uncertainty and Disagreement iievvicca vedvvney visi vatavsdivwdeyedveatnvedivatavseivedyvedy 2 Portfolio Analysis Tools vdincseu cserndecesiaiescsianitansiacuihestndiacainanecbassmienenmsebaaseitaensanie 2 DeMons eon oA A Er eesti AAT eed de eect i ecg ae ie Mi lg enc 2 F nctionality of PA Teresin EAE E E E NE EEE E A AEE 3 Report Structure sirare A A EEE E renee AAE A AE EAN 4 CHAPTER TWO Overview of PAT aiscin onna ira ian a E a a E ivan R 5 Jap ts and Outputs se cas cdots ced ded eaire raai iaa AEEA ATEENA SETENE A 5 Concepts and Termino l yssen eroinin aiii aeia oTe aS ei E SA ia 6 Multictiteria Scorecard Sossi soene i E EE E E 6 Multiresolution Thinking sb eh tase aE EE EE a 7 Translating Raw Measures of Value into Scores opin secacieiess ch danesniies recete riririrrere reren rin 7 Aggregating ScOrESu irere o a E T te nia tia tng ar EEAO T EEEE EE 8 Extensibility Allowing Custom Aggregation Methods 0 0cceceeeeeeeeeeee eee eeees 10 Navigation and Architecture Inputs and Outputs cciscecsesdseciens daveisanisneiane ier ieeieeias tours 11 Architect reand Navies Aiait ita inp ncdardedewa
22. C Weighted score Since the weights of the measures are equal the effectiveness score is just the average of the two measure scores Although it reaches only one goal in measure M that submeasure is weighted twice as much as the other two hence it has the same weighted percentage of goals reached as Investment Option A which reaches two goals in measure M a Thresholds Method With the Thresholds method values for measures that meet or exceed the goal value are high lighted in green values that meet or exceed the threshold value but not the goal value are highlighted in yellow and values that fail to meet the threshold value are highlighted in red Table 5 8 In addition we show the calculation of the measure score and effectiveness score and color the summary score cell for each measure to correspond to the color scheme that appears on the Summary sheet in PAT for the Thresholds method We also show the effec tiveness score assuming the measures have equal weight for both measures in the rightmost column In a weighted score row a cell containing xxx means that the submeasure failed to reach the threshold value so the measure score for the entire measure is zero If an investment option reaches the goal the weighted submeasure score is 1 times the submeasure weight If the raw value is between the threshold and the goal say 5 because 5 is 60 percent of the way from the threshold 2 to the goal 7 the unweighted submeasure sc
23. Kulick and Egner 2005 but it is common even in schoolhouses for one or the other to be given short shrift which biases the analysis 2 The late Bruce Goeller helped pioneer work on such policy scorecards in the 1970s Goeller et al 1983 Dunn and Kelly 1991 pp 133 ff Overview of PAT 7 It is cognitively efficient to use policy scorecards that are color coded in the familiar way red means bad yellow means marginal and green means good although PAT has options for generating scorecards in other formats Interestingly such scorecards are sometimes criticized but for reasons that do not apply to our usage The primary problem is that officials are often briefed with scorecard based analysis that is one viewgraph deep with little if any discussion of what if anything underlies the color coding of results Our approach is different Multiresolution Thinking Four Levels of Detail Each investment option can be evaluated in PAT at up to four levels of detail overall effectiveness Level 1 Level 2 and Level 3 Figure 2 3 illustrates this schemati cally with an example that we use throughout this report Results at a given level are either specified directly or calculated from the next lower level e g results from Level 3 roll up or aggregate to results at Level 2 and those from Level 2 aggregate to results at Level 1 This allows drilldown zooming That is a result at one level can be explained by drilling dow
24. Level 2 measures when calculating the Level 1 scores those used for the Summary sheet A weight of zero means that the Level 2 measure is not considered Thus a Level 2 measure can be built in but then toggled on or off as appropriate High or Low Values Desired Either High or Low capitalization matters This row allows users to specify that for some measures e g probability of failure more is worse rather than better If Low is chosen g as with Measure 1 2 scoring uses a variant of Figure 2 2 This choice of directionalities applies only at the lowest level of data entry PAT re scales higher measures to be on a 0 to 1 scale with 1 being better Hint This dictates care in naming Lowest level entries for various types of risk might be defined so that low numbers are good Aggregations however because they will be rescaled by PAT should be named something like Confidence or Risk Mitigation to avoid semantic confusion Threshold Value The threshold values described in Chapter Two for the Thresholds Weakest Link and Weak Thresholds scoring methods Goal Value The goal value used in all scoring functions except Rankings Where high values are desired the goal value cannot be lower than the threshold value Where low values are desired the goal value cannot be higher than the threshold value 20 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Effectiveness and cost effectiveness
25. Paul Dreyer p cm Includes bibliographical references ISBN 978 0 8330 4887 5 pbk alk paper 1 Strategic planning Handbooks manuals etc 2 Investment analysis Handbooks manuals etc I Dreyer Paul 1973 II Title HD30 28 D3875 2009 658 4 012 dc22 2009040390 The RAND Corporation is a nonprofit research organization providing objective analysis and effective solutions that address the challenges facing the public and private sectors around the world RAND s publications do not necessarily reflect the opinions of its research clients and sponsors RAND is a registered trademark Copyright 2009 RAND Corporation Permission is given to duplicate this document for personal use only as long as it is unaltered and complete Copies may not be duplicated for commercial purposes Unauthorized posting of RAND documents to a non RAND Web site is prohibited RAND documents are protected under copyright law For information on reprint and linking permissions please visit the RAND permissions page http www rand org publications permissions html Published 2009 by the RAND Corporation 1776 Main Street P O Box 2138 Santa Monica CA 90407 2138 1200 South Hayes Street Arlington VA 22202 5050 4570 Fifth Avenue Suite 600 Pittsburgh PA 15213 2665 RAND URL http www rand org To order RAND documents or to obtain additional information contact Distribution Services Telephone 310 451 7002 Fax 310 451 6915 E
26. They should be addressed to Paul K Davis pdavis rand org or to the developer Paul Dreyer dreyer rand org The PAT program has been used extensively but has not been exhaustively tested The documentation was written using Excel 2003 Windows and 2004 Macintosh a few minor differences with Excel 2007 Windows are mentioned in footnotes This research was sponsored by the Office of the Secretary of Defense and conducted within the Acquisition and Technology Policy Center of the RAND National Defense Research Institute a federally funded research and development center sponsored by the Office of the Secretary of Defense the Joint Staff the Unified Combatant Commands the Department of the Navy the Marine Corps the defense agencies and the defense Intelligence Community For more information on RAND s Acquisition and Technology Policy Center contact the Director Philip Anton He can be reached by email at atpc director rand org by phone at 310 393 0411 extension 7798 or by mail at the RAND Corporation 1776 Main Street P O Box 2138 Santa Monica California 90407 2138 More information about RAND is available at www rand org Contents Preface o cpats dues E A T E E A E iii MUS odine a a A AE EA E E Pes AA A A tte begat teas Mec AEE EAA A EA ix Lables eeen ea EO E A AO E N N O nea Via xi SUMMALY eiii ra NE a A AAA E TA A EE O A E abies xiii Acknowledgme nts suser inr cae eeir eaen e EEEE EE E she dre le grade EEEE EE eg SE xxi
27. all measures Weighted sum of measures scores Minimum of measures scores Weighted sum of measures scores Comment Appropriate if component measures represent critical components of capability Appropriate if both component measures and measures are individually critical Appropriate if thresholding is valuable but not all component measures or measures are scores critical Goals Weighted fraction of the Weighted sum of component measure measures scores goals achieved by option Rankings Borda ranking Borda ranking aThis option was introduced in the DynaRank system Hillestad and Davis 1998 bA single winner election method in which voters rank candidates in order of preference Link method is similar but even more stringent It assesses the aggregate score to be the lowest of the contributing scores and assesses overall effectiveness as the lowest of the measure scores 4 The third method Weak Thresholds is less draconian Both measure level scores and overall effectiveness are simply weighted sums A contributing factor is scored zero if it does not reach its threshold value but the aggregate score is the sum of the factors scores rather than zero This method is suitable if the contributing measures are not individually critical but it seems important to impose thresholding These methods are effective heuristics that are well understood by decisionmakers A tough commander or manager for
28. calculations aggregate across measures and need to operate on acommon scale This is accomplished by setting two further parameters e Level 2 Measure Score for Threshold Value 0 to 1 Scores assigned to Level 2 measures if their thresholds are just reached for scoring methods that incorporate thresholds e Level 2 Measure Score for Goal Value 0 to 1 Scores assigned if goal values are reached or exceeded For scoring methods with thresholds the effectiveness score for a value that falls between the threshold and the goal value is interpolated linearly Figure 2 2 It is good practice to set this value to 1 for all measures Warning Comments Each Level 1 measure may reflect a special Level 2 measure called Warning The data in the Warning columns should be text If a given cell has a warning the Summary display will have a small flag in the top right corner of the corresponding cell Mousing over that cell will bring the warning message up This mechanism is a convenient way of flagging results that depend on reasonable but worrisome assumptions In a defense planning analysis such a flag might be Assumes at least one week s actionable strategic warning The Modify Summary Button After making changes in data at any level the user should click the Modify Summary button top left or go to the Summary sheet and select Update Summary to have the changes take effect Cautionary If any Level 2 measure for a particular Level 1 meas
29. capability Conversely small deviations in investments may have a disproportionate effect on the outputs of the system A budget that includes 90 percent of the cost to acquire a missile does not get the investor 90 percent of a missile For the most part small reductions in investment do not lead to proportionate reductions in system capability Marginal analysis tends to be more meaningful when the options that result from con stant deductions to each investment are really of equal value Consider a situation in which 16 million is spent to acquire three different types of missiles where each individual missile costs 1 million Assume further that the current investment plan purchases one missile of the first type five of the second type and ten of the third type Three equal cost investment options that could result from a 1 million cut in funding correspond to not purchasing one missile of each type In contrast consider a situation where 16 million is spent in the acquisi tion of three missiles one costing 1 million one costing 5 million and one costing 10 mil lion A cut of 1 million from any of the three missile purchases results in not getting any mis siles so this cut actually results in comparing three cases costing 6 million 11 million and 15 million for the acquisition of two of the three types of missiles This suggests a variant of marginal analysis in which the changes to the current set of investments represent t
30. column corresponds to the color of the corresponding cell on the Summary sheet If the Thresholds Weak Thresholds or Weakest Link scoring aggregation method is selected cells in the table are colored red yellow or green depending on whether the raw value 1 does not reach the threshold value 2 reaches the threshold value but does not reach the goal value or 40 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Figure 4 13 Level 2 Drilldown Sheet Scoring Method Display Recalculate Thresholds All rows Drilldown serstions Update Level 2 Use 5 color display Ss x Data Sheet Level 1 Measure Measure 2 Level 2 Measure Measure 2 1 Measure 2 2 Warning Weight of Level 2 Measure in scoring Functions 0 to 1 1 1 D High or Low Values Desired High High Threshold Value 0 D 0 Goal Value 1 1 0 Level 2 Measure Score for Threshold Value 0 to 1 0 0 D Level 2 Measure Score for Goal Value 0 to 1 Option A Option B Option C Color Code Level 1 and Level 2 Measure 0 0 to 0 2 or Score 0 8to 1 0 0 6 to 0 8 0 4 to 0 6 0 2 to 0 4 Failure F 3 reaches the goal value respectively The color of the cell corresponds to the measure score with the maximum score being 1 corresponding to green and the minimum score being zero corresponding to red For the Rankings scoring method Figure 4 16 each Level 2 measure is ranked individu ally with colors on a light blue to dark b
31. e eee eee eee eee eee ees 34 MRM Menu of Summary Sheet 01s casicucieandincucaiiunssiedeimincediucieuedbimnuntiass 34 Scoring Method Menu of Summary Sheet iin s ccd anscdeanguiianvidpedarndugedsaneeuteses 34 Current Perspective Menu of Summary Sheet Example Specific 0c cee 35 Discount Rate Menu Summary Sheeh io ciaccidacinneiardaneiaindaiahehudmdamiahbede 35 Cost Information in the Summary Sheet reestr ririri rrrrrre reren n 37 Warning Flags in a Summary Sheet onde iorie sadssiaruiantady ie AA aA A AEA 38 Illustrative Summary Scorecard Level 1c cacsciarsoncetcravmautrramenreumnqarewaberays 39 Level 2 Drilldown Shectsiractaieos nerna a E E N 40 Compressed Version of Level 2 Drilldown for Measure 2 00 000 cee ceeeeeee eens 41 evel Driilldown with Goals Method a ci5 14d0 i Wavareinededvaityeedansivicanstasdeesnatey 41 Level 2 Drilldown with Rankings Method 00 cccecceceececeeeeeeeeeeeeeeee eens 41 Level 3 Drilldown for Measure 2 2 foci cinrieccsmriariee rnc a ea 42 x RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual 4 18 4 19 4 20 4 21 4 22 4 23 4 24 4 25 5 1 5 2 6 1 All AL2 A 3 AA AAS A 6 A A 8 Level 3 Drilldown If Measure 2 2 Has a Threshold Value 0 0eceeeeeeee es 43 Illustrative Total Costs Versus Time Glatt ni iedssene sits crastadasandtsscaentadutnanecaniie 44 Illustrative Scatter Plote niseni wing bain wa
32. example may consider a unit to have failed inspection even if the unit did rather well in many respects The other two aggregation methods goals and rankings are useful in certain cases as discussed in Chapter Five Extensibility Allowing Custom Aggregation Methods In several of our applications of PAT we have found it necessary to aggregate results in ways that do not lend themselves well to the built in options This can be done straightforwardly using standard features of Excel and modest amounts of mathematics 4 This rule was introduced in Hillestad and Davis 1998 Overview of PAT 11 Hint one that will be meaningful only to someone who is ready to actually use PAT One approach when calculating an aggregation from a given level is 1 add a new measure at that level called calculated score 2 define option values for that measure with an equation referring to other data e g to the product of previous measures at the same level 3 set the weights of the other measures to be very small but not zero e g 0 0001 and the weight of the calculated score to be 1 4 use any of the three core scoring methods in Table 2 1 The result will be that the aggregate score will be the calculated value but the analyst will still see the values of the component factors when using PAT s drilldown feature as described in Chapter Three NOTE When extending PAT s functionality by entering equations users should ordina
33. in the fresh version of PAT As usual be careful in doing so Run Template Builder as above When copying and pasting data from the previous workbook recognize that the new structure and the old structure do not match Thus copying and pasting should be done carefully from block to block where it makes sense Other data may have to be filled in for the first time e g for a new option or a new measure As an alternative the user can copy and paste full data from a previous sheet and then edit by moving columns and rows around until they are in the right place also assuring 95 96 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual that control parameters are set correctly Although this can be done it is more error prone than the deliberate procedure suggested above The most important advice is probably to perform any data transfer slowly methodically and carefully rather than doing it quickly and trying to catch and correct errors afterward And of course have backups APPENDIX C Editing and Neatening It will often be necessary to edit PAT sheets either for clarity or to produce an output display suitable for use in a viewgraph or document Since PAT is implemented as an Excel spreadsheet file all of Excel s ordinary features apply However the following few hints may prove helpful e To move a button or control panel use Control Select to avoid triggering the action e Use Control Select
34. matters and do so using test scenarios with intelligent adversaries responding to projected U S capabilities and vulner abilities Ultimately perspectives differ in the value placed on various objectives judgments about how much is enough and the related matter of managing strategic risk The user can also define extended perspectives that make distinctions based on e g how the effectiveness of an option for a particular measure or submeasure is estimated xvi RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Figure S 2 Drilling Down for Explanation Measures Measure 1 Measure 2 Measure 3 Detail Detail Detail Investment Options Option A Option B Option C Summary F bevel 1 Level 1 Measure Measure 2 Level 2 Measure Measure 2 1 Measure 2 2 Warning Weight of Level 2 Measure in Scoring Functions 0 to 1 1 1 Option A Option B Option C Level 2 Weight of Level 3 Measure in Scoring Functions 0 to 1 Option A Option B Option C Level 3 Multiresolution Data Entry and Modeling A common problem in strategic decision support is that data whether empirical and objec tive the result of organizationally approved model calculations or the result of the structured judgment of experts can be overwhelming Data can be the enemy of agile analysis intended for sharp reductionist high level thinking PAT is designed to enable
35. modeling war gaming and operations The raw values may be expressed in different units and on different scales They may be objective or subjective A larger raw value 3 If Level 2 measures are specified directly they may be specified either as raw values on an arbitrary scale or as scores between 0 and 1 with higher being better 54 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual may be good or bad depending on how the quantity in question is defined Technical risk as measured by likelihood of failure during a mission is better if it is smaller Technical risk as measured by mean time before failure is better if it is larger By themselves such raw values do not convey a sense of sufficiency For that we need to introduce goals and thresholds Goals and Thresholds All of the methods used in PAT except the Rankings method involve goals thresholds or both Inputs to PAT include for each submeasure a raw value corresponding to the goal and a raw value corresponding to a threshold a minimum level for perceived utility Submeasure Scores PAT calculates the score of a submeasure from its raw value goal and threshold Except with the Rankings method discussed below the score is between 0 and 1 with 1 always being good Measure Scores The score of a measure is calculated from the scores of its submeasures That is a measure s score is an aggregation of its submeasure scores Except with the Rank
36. of the measures generated by PAT Overview of PAT 9 Figure 2 4 Mapping of Raw Values into Scores Score Score at or above goal Threshold score Threshold Goal Raw value of submeasure Diagrams such as Figure 2 3 indicate with arrows that different measures combine to generate the score of a higher level measure but they do not say how they combine It is fre quently assumed in commercial decision analysis tools and in introductory courses in decision analysis that factors combine via linear weighted sums This is so common that it has affected our vocabulary as when we refer to the relative weight of different input variables Linear weighted sums are often convenient and adequate but they can be quite mislead ing because in the real world the combining rules are nonlinear Do we want to be healthy or wealthy No amount of health can compensate for extreme poverty and no amount of wealth can compensate for extreme illness Overall utility is not a sum of scores for health and wealth but something more complicated Similarly in designing a system or an investment program the value of the whole may be essentially zero unless each of various critical compo nents is sufficiently good Because of such considerations PAT provides five built in methods for establishing scores and aggregating upward Ideally only one such method would be needed but theory and experience tell us that alternatives are needed even more al
37. only comment is to define Measure 1 as Effectiveness in Scenario 1 measured by outcome After entering data the user can click the Modify Summary button to have the data take effect l As with most modern tools the distinction between input and output sheets is blurred because some inputs can be changed in output sheets This undercuts architectural clarity and can be confusing but is quite useful in practice 2 Once entered these cannot be changed in Level 1 Data alone but can be changed using Find and Replace 17 18 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Figure 3 1 Level 1 Data ee Modify Summary Measure Comment Weight Effectiveness in Scenario 1 measured by Measure 1 outcome 1 00 Measure 2 1 00 Measure 3 1 00 fe aMail Although this is the official input sheet for Level 1 users will usually employ Template Builder to create the structured input and output sheets and they will usually specify the weights of Level 1 measures in the Summary sheet even though that is nominally an output sheet Level 2 Data The values of Level 1 measures i e summary measures are determined by data at Levels 2 and 3 Figure 3 2 shows the Level 2 Data sheet for our simple case with three measures each with two submeasures That is each Level 1 measure depends on two Level 2 measures Names of Measures and Options The Level 1 measure names are entered as column headers precis
38. other threshold methods Sik 0 if V ik Pa S ja G if V jk eV V V7 at i j k jk x T G Sai ree Ca Ta BV ESV je SV aay Aggregation to Find Measure Scores with Weakest Link Method With the Weakest Link Method the measure score is the minimum of the submeasure scores If any submeasure fails to reach its threshold the measure score will be 0 as in the Thresholds method but if all submeasures reach their thresholds the score will be different from that in the Thresholds method and typically smaller M min 1S Aggregation to Find Overall Effectiveness with the Weakest Link Method Aggre gation with the Weakest Link method is performed simply by taking the minimum measure score which is identical to the minimum submeasure score of all submeasures If any submea sure in any of the measures fails to reach its threshold effectiveness will be 0 Even if that does not happen effectiveness will typically be smaller in this method min m Rankings Method The Rankings method does not use goal or threshold values Instead for each submeasure the investment options are simply ranked from best to worst without regard to absolute perfor mance Instead of submeasure and measure scores we refer to submeasure and measure ranks for this method Submeasure Ranks Let R be the rank of investment option i for measure j and for submeasure k We define it as one more than the number of investment options that perform
39. subsets of the phases that can be implemented Next consider the following probabilities for engagement with a single attacker shown in Table 6 1 for four different scenarios homeland defense HD homeland defense with no boost phase access homeland defense with advanced countermeasures CMs and defense of deployed forces and defense of friends and allies DODF DOFA Assuming that the performances of the systems in each phase are independent of one another we can compute the probability that a single missile is intercepted for each of the eight investment options as well as for each of the four scenarios which can be thought of as mea sures If we set a threshold of 0 5 and a goal value of 0 8 for each scenario we get the results shown in Table 6 2 the colors that would appear on the Drilldown sheet under the default goal based with thresholds scoring method Treating the individual scenarios as measures or to be accurate treating each column as a measure consisting of a single submeasure we can calculate the effectiveness of each invest ment option for the Thresholds scoring method Assuming all scenarios are equally weighted with a submeasure score of 0 5 for reaching the threshold value and 1 for reaching the goal value the effectiveness score of each investment option is as shown in Table 6 3 Table 6 1 Notional Probabilities of Intercept for Illustrative Problem Mission Case Flight Phase HD No Boost Phase Advanced CMs
40. such as Parallels Desktop or VM Ware Fusion 4 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual 6 Links to capability analysis and other sources of data PAT links to even more detailed information such as that of an embedded or connected capabilities model data generated separately from a capabilities model empirical data or structured judgments 7 Marginal analysis Although PAT emphasizes multiobjective scorecards it also gener ates scores of overall effectiveness or cost effectiveness These can be used for marginal or chunky marginal analysis about how to spend or cut the next increment of funds 8 Ability to represent and contrast alternative perspectives PAT encourages analysts to deal explicitly with the serious differences of opinion and judgment that can be referred to as alternative perspectives Results of PAT analysis can then be shown as a function of strategic perspective There can be striking differences in implications for cost effectiveness assessments 9 Facilitated operations At a mechanical level PAT automates many tedious spread sheet operations so that users can generate and manipulate portfolio style scorecards and underlying detailed information quickly It also provides a variety of built in displays Report Structure The report is organized as follows Chapter Two introduces the principal concepts and terms in PAT and gives a schematic overview Chapters Three and Fo
41. the scoring method using the Scoring Method dropdown menu and up to four investment options Changing either will regenerate the spider chart as will clicking on the Generate Spider Chart button which should be used when the raw values or parameters on the data sheet are changed 48 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Figure 4 23 Multimeasure Spider Plot Scoring Measure 1 Measure 1 Method Thresholds gt Generate Spider Chart Measure 2 Measure 2 x Chart 1 option A x Measure 3 Measure 3 z Chart 2 Option B Measure 4 z Chart 3 option Measure 5 gt Chart 4 E Failed on at least one Level 2 i Measure Measure O Option A O Option B O Option C Measure 3 Measure 2 Selected Details Sheet Figure 4 24 generates a Selected Details display similar to the Drilldown sheets except that the Level 2 measures in the columns are specifically chosen for the purposes of the particular analysis for which PAT is being used and may come from different measures That is whereas the Drilldown sheets relate to a particular measure the Selected Details sheet allows the user to tailor a display with details selected from several measures As with the data sheets all scoring methods and the associated color schemes may be displayed on the Selected Details sheet by using the Scoring Method dropdown menu The dropdown menus allow selection of the avail able measures u
42. the weights as part of the titles so that we could remember what they represented Some other rules are necessary to use this approach effectively 1 After using the Current Perspective menu either to generate a new perspective or to change perspectives the user will be prompted by PAT Figure 3 7 and given the opportunity to enter a perspective name If the current settings weights etc have been reset for a new perspective enter that name If they have been reset to change the defi nition of an existing perspective type the old perspective s name and when prompted again choose Yes Otherwise hit OK without entering a name This query box appears routinely when closing PAT even if no changes in perspective have been made The usual response is OK 2 If errors are made and problems arise corrections to existing perspectives can usually be made by editing the Perspectives data sheet In some cases however there will be an error message about inconsistencies between the Level 2 Data sheet and the Perspectives sheet It is often easiest to delete everything on the Perspectives data sheet and recreate the perspectives from scratch That avoids tedious trouble shooting Figure 3 6 Using the Summary Sheet s Perspectives Menu Options Menu Please Select An Option Scoring Method Thresholds Current Perspective Default Default Baseline 1 1 1 Measure 1 Emphasis 2 1 5 Weakest Link 2 1 5 28 RAND s Portfolio
43. user may erase the name in the header and select Update Summary from the Options menu Adding a Numeric Column of Level 2 Information The user may wish to elevate the visibility of a particular Level 2 measure by including its numerical values in the Summary sheet using the next block of columns indicated in Figure 4 1 These are nominally Columns O to R the columns with Related Details buttons The procedure is as follows In Row 1 in any column in the reserved range type the name of the Level 1 measure followed by the name of the particular Level 2 measure separated by with no spaces An example would be Measure 2 Measure 2 1 Copying and pasting a name from the Level 2 Data sheet may help avoid typographical errors Altering Cost Related Columns The columns set aside for cost data nominally T through AB those with Cost Detail buttons can be used to display subsets of information in the Cost Data sheet Take a case in which PAT was set up initially to display R amp D acquisition and O amp S costs on the Summary sheet for the period 2010 2015 PAT will add a Total Cost column for that period as well But then suppose the user wants to show two year costs as well The additional columns could be added to the summary by simply typing in the correct names in Row 1 probably in columns X Y and Z or at least in columns with Cost Detail buttons In this case the user would type in the fol lowing names respectively R amp D Cost
44. warning flags as shown for Option 5 in Figure S 3 Well conceived work with PAT should convey a good sense of what is being assumed a traditional objective in analysis The goal in decision support however should be even higher It is one thing to summarize for the decisionmaker the assumptions and risks associated with an option The decisionmaker is better served however if he or she is presented also with options that mitigate or hedge against risks even risks that seem unimportant under pre vailing best estimate reasoning Wise decisionmaking is not about optimization for a set of assumptions it is about finding strategies that are not only likely to be acceptably effective under nominal assumptions but also flexible adaptive and robust Acknowledgments The authors appreciate the careful reviews and constructive suggestions of colleagues Manuel Carrillo Barry Wilson and Duncan Long as well as the many suggestions offered during the development of PAT by colleagues James Bonomo Henry Willis Russell Shaver and especially Richard Hillestad who developed a predecessor tool DynaRank that we used in designing PAT Abbreviations BMDS CM COMPOEX DHS DoD DODE DOFA FYDP GAO HD JICM JWARS MDA MRM O amp S PAT PSOM R amp D RDT amp E SEAS STORM ballistic missile defense system countermeasure Conflict Modeling Planning and Outcome Experimentation Department of Homeland Security Department of D
45. weighted As in Table 5 3 submeasure scores sum of measure scores Linear weighted sum of Linear weighted As in Table 5 3 submeasure scores sum of measure scores Zero if any submeasure Linear weighted As in Table 5 3 fails to reach threshold sum of measure otherwise linear scores weighted sum of submeasure scores Minimum of submeasure Minimum of As in Table 5 3 scores measure scores Weighted average of Linear weighted As in Table 5 4 ranks for measure sum of measure summary table linear scores weighted sum of Borda scores for effectiveness Raw Value M1 M1 M1 M2 M2 M1 1 M1 2 M1 3 M2 1 M2 2 0 25 0 25 0 5 0 25 0 25 High High High High High 2 2 2 2 2 7 7 7 7 7 0 0 0 0 0 Comment Simple and common but arguably simple minded May be appropriate if not all submeasures are critical May be appropriate if all submeasures are critical and have firm requirements May be appropriate if all measures and submeasures are critical and have firm requirements May be appropriate if one wishes to avoid discussion of goals and thresholds M2 M2 3 0 5 High 66 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Table 5 7 Illustrative Results for the Goals Method ma Mi 7 z N Level 2 measure High or Low values High High High i High desired Threshold value 2 2 2 7 Level 2 value for Threshold Level 2 value for Goal 1 Option A Weighted score Option B Weighted score Option
46. 08to10 O6to08 O4to06 O2toO4 0 0to0 2 Level 1 Measure Score 0 8to 1 0 06to0 8 0 4to0 6 02to04 0 0to0 2 or Failure or Failure F F calculatio Measures Measure 1_ Measure 2_ Measure 3 Measures Measure 1__ Measure 2__ Measure 3 Detail Detail Detail Detail Detail Detail Detail Detail Detail Detail Detail Detail Investment Options Investment Options Option A Option A Option B Option C Color Code MW Level 1 Measure Score 08to10 O6to08 O4to06 0 2to04 0 0to0 2 Level 1 Measure Score 0 8to1 0 0 6 to 0 8 0 2to04 0 0to0 2 or Failure or Failure F F calculatio calculatio 7 Delete Perspective This option brings up a query box in which the user can specify dele tion of any perspective other than the current one 8 Add Remove Failure Markers This is an option to show F in cells for which the score falls below a threshold Continuing with the other control panels of Figure 4 1 sorting is controlled by two menus one specifying the criterion for sorting i e the choice of column Figure 4 4 and one specifying whether the sorting will be in ascending or descending order Figure 4 5 The choices of the former menu change with usage but include all of the Summary sheet col umns including effectiveness and relative cost effectiveness the various cost columns and any Selected Measure columns that may have been added the columns with buttons ca
47. 2 are light green and green averaging to green in the Measure 2 2 Score column That is per haps reasonable But why then is the Level 2 score yellow We will walk through the calcula tions because they are somewhat subtle Note first from the rows above the scorecard that the Measures 2 2 1 and 2 2 2 have Level 3 thresholds of 5 and 5 Thus averaging the raw values in this case produces 1 2 8 5 5 10 5 5 or 0 8 as shown in the Measure 2 2 Score column That is the scoring starts at a raw value of 5 In this case however Level 2 also has a threshold The small print parenthetical expression below the Level 2 score tells us about this without our having to look in the Level 2 Drilldown sheet The syntax of 0 51 0 1 1 means see the note at the bottom of the display that the threshold is 0 51 the score at the Figure 4 18 Level 3 Drilldown If Measure 2 2 Has a Threshold Value Level 1 Measure Measure 2 Level 2 Measure Measure 2 2 Scoring Method Thresholds Measure Measure Level 3 Measure 2 2 1 2 2 2 weight of Level 3 Measure in Scoring Functions 0 to 1 High or Low Values Desired High High Threshold Value Goal Value Level 3 Measure Score for Threshold Value 0 to 1 Level 3 Measure Score for Goal Color Code Level 2 and Level 3 Measure 0 0 to 0 2 or Score 0 8 to 1 0 0 6 to 0 8 0 4 to 0 6 0 2 to 0 4 Failure F Numbers indicate Measure s Level 2 settings for threshold value 0 51 thresho
48. 5 og i H z 8 o o i i 18 w Su Sw i lt aS i I i 4 wo i i D 8 Q Se ke ESE f 2 Sages EDT O OLES moeze3 22852 HYooaogt SF i Beecugs EGEDE E H SoEsSon i Ww o mieie f Fu 20806 o2 ZHES 1 E Sc H e2oZ E o og2 Z i I 962 1 A i 0 i Pi 3s i epee o F n oO se i i i i gt i a i i Fi i 8i SLT i iy Sze er i wo E TD RS egoae i i o239 6 ji Ef SELDI i E i i Ns wo 2 j amp oo i 0 ol i D gt Lr i gt o2o q T6050 5 oJ i g2 i EEO 4 i oS a bas A al l oO OF 5 2 5S Sy ao a o i a 3 Io i D J 4 o amp i rt 1 EESE EA EA SA A E E S E eed EA Vd E Wea er E Er kel A Nn 2 a E nna a a Pe es Dp Fe oO v o T F Qo E S 58 E B 2 BB 3 eami 3 a ee z oO i 3 e 5 I 7 a q 2 l 2 gt a B mmj 2 2 ir aj oO 1 2 N s S lt 5 a i og m a Le sou R eae Sr E DE ee eee ER ieee en A eel E eee A overt Reed ER ee 4 ns La N o io of x olol le eleele Quickstart on Using PAT 85 e List the items for which you want to track costs separately Item 1 in the example This could be e g ships airplanes
49. 62 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual strictly better than investment option 7 on submeasure kof measure j Thus if two investment options have the same raw value they will have the same rank Aggregation to Find Measure Ranks with Rankings Method The aggregation from submeasure rankings to measure rankings is again a linear weighted sum We calculate the average rank R within measure j as The value R is used to set the color one of five shades of blue as shown in Table 5 4 on p 64 in the measure summary table depending on the quintile in which the average ranking resides Aggregation to Find Overall Effectiveness with Rankings Method This aggregation function is best explained with words and examples It is entirely different from the functions used in the other methods For this effectiveness aggregation each measure is assigned a score based on rankings Let us assume that K investment options are under consideration and that each submea sure has K K 1 2 points to distribute among the investment options If there are no ties for the values in submeasure j the submeasure score is given by R K R i j k If there are ties the points that would have gone to the investment options in those posi tions are combined and then equally distributed among the investment options For example with ten investment options the second and third place investment options would receive 8
50. ANEEN EE ETAN EEN EAEN EANET RANE 57 Weak Thresholds oid ole ape eC one ren ere Oe en ene en nen E En eiS 59 Thresholds Method aa e E E O E A T 60 Weakest Link IYER dna ateade haben ett heeded AE AS ENE K AEE 61 Rankings Vie MOA eis cinnsceracaviginsan rE E EEE T E E AARRE 61 Color Coding in Scorecards aysar iieo aa n Ei E E E A 63 Colors for Thresholds Weak Thresholds and Weakest Link Methods 0 000000 0 63 Colors for Rankings Method jsescssicrscnsvenreudnivassus caviar tireo ine ie ieia tai iiS 63 Examples of Scoring and Aggregation Using Different Methods 6 00 0cceeeeeeeeeee es 64 Goals Method crire taea niet aaa ae aoe ea eno Sane 65 Thresholds Method yc 3 spscc crise ssesrebetati acide hag eh Nad ele Nace hie Soe aaa ane geR NS 66 Weak Thresholds Wy ord 1016 ave eee ee ee ere ee eee eee eer rere EA 67 Ce cin Link Methoden eee pea a tetramer E neers e eerie eee reer E a re Te 67 Rankings Method ac eleva ss We eee cesta cie esa een nE n E Ni acetate code EEEE 69 CHAPTER SIX Marginal and Chunky Marginal Analysis 0 0 0 000 00 ccccecceeceeceeeeeee ete eee ete etereees 71 Tntrodtction sas cep bby heh neg te ihc etek PA de cake se hoot AAS 71 Chunky Marginal Analysis for a Ballistic Missile Defense Example 0 0 00cceeeeeeee es 72 CHAPTER SEVEN Concluding Observations 0 00 0 000 0c ccc sesiet errete t E eect ete E EEE E E eteetetererey 77 Purpose did Function Ol PAY
51. Analysis Tool PAT Theory Methods and Reference Manual Figure 3 7 Query About Saving or Creating a New Perspective Warning Save current perspective Do you want to save your current settings as a perspective IF so enter a name for the perspective in the box below Otherwise leave the entry blank Extended Perspectives An analyst getting into the spirit of exploring the consequences of alternative perspectives may wish for a change of perspectives that would have more consequences than are allowed by the built in features of PAT which apply only to changes at Level 2 relating to weights goals thresholds scoring scheme and aggregation method In particular it is sometimes logical to associate a perspective with changes in the assess ments of some or all options not just the weight of those measures For example a conservative perspective might evaluate risks to be much higher than an optimistic perspective would Or one perspective might logically weigh Level 3 data differently than another perspective would Although this is not enabled by PAT s built in machinery extended perspectives can be defined by someone skilled in using Excel This might involve for example appending some multipliers to the data elements that would be dependent on the extended perspective with an interface to those multipliers values on a custom sheet When changing an extended perspec tive the analyst should remember to change the multi
52. DODF DOFA Cost billions Boost B 0 7 0 0 0 5 0 7 9 Midcourse M 0 6 0 6 0 4 0 6 6 Terminal T 0 5 0 5 0 3 0 0 3 Table 6 2 Performance of Options Probabilities of Intercept by Mission Mission or Cost Option No Boost Phase Advanced CMs DODF DOFA Cost billions 74 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Table 6 3 Costs and Effectiveness Comparisons Equal Emphasis on all Scenarios Cost Option Phase CMs DOFA Sum 4 billions Le fone NOTE The unbracketed numbers are probabilities of successful intercept as in Table 6 2 The numbers within brackets are the effectiveness scores using the Thresholds method Another possible perspective would be to put a greater emphasis on dealing with a peer threat that can deny boost phase access and implement advanced countermeasures We can examine that case by weighting those two scenarios twice as much as the other two The effec tiveness of each investment option is then as shown in Table 6 4 The Scatter Plot sheet can display the effectiveness of each investment case for both per spectives with the cost of the investment option as the x axis Figure 6 1 The scatter plot view permits straightforward comparison of the investment options under both perspectives Reading the scatter plot however requires some instruction because it is unconventional Suppose that the budget must be cut from 18 billion to 9 billion At that bu
53. Menu of Input and Output Sheets x 050 e A otated ample 2b ied File Edit View Insert Format Tools Data Window GoTo Sheet Help Fae a REE ee ew a oe Wee 7 Input Sheets gt Level 2 Data A E a ae a ee el I MRM Level 2 Data Cost Data Output Sheets Tables gt Level 3 Data Output Sheets Graphics gt MRM Level 1 Data i Perspective Cases Measure Comments and Weights Template Builder gt In Excel 2007 the Go To Sheet menu is accessed under the Other Add Ins menu 12 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Figure 2 6 Tabs for Moving Among Sheets Tabs 4 Summary Level 2 Drildown Level 3 Drilldown Level 1 Data Level 2 Data Level 3 Data MRML lt Table 2 2 PAT Output and Input Sheets Output Sheets Input Sheets Tables e g Scorecards Level 1 Data Summary Level 2 Data Level 2 Drilldown Level 3 Data Level 3 Drilldown MRM Level 1 Data Selected Details MRM Level 2 Data Rankings Table Cost Data Graphics Perspectives Scatter Plot Template Builder Spider Charts Multimeasure Spider Charts Cost Charts Although convenient the Go To Sheet menu omits custom sheets that the user may have added for extra data specialized calculations or notes In contrast all of the sheets have corresponding tabs along the bottom The standard Excel way to navigate among sheets is by clicking those tabs Table 2 3 shows the default left to righ
54. Methods and Reference Manual Aggregation to Find Measure Scores with Weak Thresholds Method The score of measure j is again given by a linear weighted sum as in the Goals method but the submeasure scores are different nj LW Sis M Pd i DW Gi k l Aggregation to Find Overall Effectiveness with Weak Thresholds Method Overall effectiveness with the Weak Thresholds method is given by a linear weighted sum as previ ously but the measure scores are different Thresholds Method Submeasure Scores with Thresholds Method Submeasure scores are calculated with the same function as that used for the Weak Thresholds method S js 0 if Vs id Fa Sija Gi if Vi id 2 Vi V V7 S a T p t G pT a EVE SV SV ijk jk G T VeVi Aggregation to Find Measure Scores with Thresholds Method The difference in methods occurs here If any submeasure fails to reach its threshold the measure is assigned a score of 0 nj DY Sy M ifall submeasures reach their thresholds LW Gi k l M 0 otherwise Aggregation to Find Overall Effectiveness with Thresholds Method Effectiveness is calculated precisely as before but the measure scores are different Details of the Methodology 61 Weakest Link Method The Weakest Link Method is even more stringent in enforcing the requirements represented by thresholds Submeasure Scores with Weakest Link Method Submeasure scores are calculated with the same function as for the
55. Software 0 0 Operations Cost M Option 2010 2 011 Hardware 30 35 Software 10 15 jenuey ad uasajay pue spoyyay AJOaUL 1Yd JOOL sISAJEUY O1 0J140g S GNVY YZ PAT Input Worksheets 25 Simplifications It is often desirable for the analyst to take shortcuts in dealing with costs especially early in a project He may wish to think simply in terms of total costs over all categories and over all the years of interest Further when he conducts sensitivity analyses he may wish to change only one cost number per option PAT has no multiresolution data entry option for costing but the simplification is easy Either in the Cost Data sheet directly or via Template Builder the analyst can specify having only a single cost category such as Total Cost M and the only investment item may be Stuff Hint An additional worksheet can be created to enter detailed cost information for use when needed Later the information might be entered into Cost Data with a revised format The level of detail used then is very much up to the analyst and can be changed in the course of a study Perspectives The Basic Concept of Perspectives As discussed in Chapters One and Two an important feature of PAT is its ability to show how cost effectiveness landscapes vary as a function of strategic perspective A perspective is rep resented by the different sets of choices used in scoring and aggregation Figure 3 5 shows the P
56. Value focused thinking is a form of multiobjective decision analysis that organizes around an organization s objectives Keeney 1992 It has been used in a number of military applications see e g Parnell 2006 Details of the Methodology 57 PAT has five built in methods for aggregation of scores This greatly increases flexibility but it also increases complexity and can undercut the goal of having relatively simple logical and intuitive results Thus the analyst should decide which method or methods to use and then present only those The five methods and their short names are as follows 1 goal based Goals 2 goal based with weak thresholds Weak Thresholds 3 goal based with thresh olds Thresholds 4 goal based with weakest link Weakest Link and 5 rankings based Rankings These are described one by one below and are then summarized in Table 5 5 on p 65 For each method the description specifies how scores are generated from the raw values of submea sures and how higher level scores and effectiveness are calculated by aggregation We use a common notation as defined in Table 5 2 The notation and subsequent discus sion apply only to problems limited to Level 1 and Level 2 measures but can easily be general ized to the case that has Level 3 measures as well Goals Method The Goals method is the simplest to describe Every measure is composed of a collection of submeasures each of which has a goal value t
57. ain Excel 2004 even if they otherwise upgrade to Office 2008 or use a virtual machine such as Parallels Desktop or VMware Fusion Both solutions have been quite satisfactory in our experience 97 References Bracken Jerome Moshe Kress and Richard Rosenthal Warfare Modeling New York Wiley 1995 Cohen William Report of the Quadrennial Defense Review Washington D C Department of Defense 1997 ed New Challenges in Defense Planning Rethinking How Much is Enough Santa Monica Calif RAND Corporation MR 400 RC 1994 As of September 2 2009 http www rand org pubs monograph_reports MR400 Davis Paul K Analytic Architecture for Capabilities Based Planning Mission System Analysis and Transformation Santa Monica Calif RAND Corporation MR 1513 OSD 2002a As of September 2 2009 http www rand org pubs monograph_reports MR1513 Synthetic Cognitive Modeling of Adversaries for Effects Based Planning Proceedings of the SPIE Vol 4716 No 27 2002b pp 236 250 Davis Paul K and James H Bigelow Experiments in Multiresolution Modeling MRM Santa Monica Calif RAND Corporation MR 1004 DARPA 1998 As of September 2 2009 http www rand org pubs monograph_reports MR1004 Motivated Metamodels Synthesis of Cause Effect Reasoning and Statistical Metamodeling Santa Monica Calif RAND Corporation MR 1570 AF 2003 As of September 2 2009 http www rand org pubs monograph_reports MR1570
58. all cases more is better i e the farther out an option extends the better There are no values on the axes because the purpose of the chart is merely to com municate visually a sense of relative effectiveness by different measures Changing the measure to be charted or any of the investment options will automatically regenerate the spider chart as gt Spider charts are also called radar charts PAT Output Worksheets 47 Figure 4 22 Illustrative Spider Plot Measure to be charted Measure 3 z M Auto update chart Chart 1 Goal value Generate Spider Chart 2 Option a Chart Chart 3 option B x Chart 4 Option x Chart 5 Measure 3 1 OGoal Value OOption A O Option B O Option C Measure 3 3 Measure 3 2 will clicking on the Generate Spider Chart button which should be used when the raw values or parameters on the data sheets are changed Multimeasure Spider Charts Sheet Multimeasure spider charts Figure 4 23 provide a visual depiction across different Level 1 measures For each investment option displayed on the spider chart the values along each arm correspond to the average rank for the Rankings scoring method or the measure score for the other scoring methods For the Thresholds scoring method if any investment option for a sub measure fails the value for the corresponding measure falls inside a red polygon representing failed measures The user can select
59. and 7 points respectively If two investment options are tied for second place they would receive 7 8 2 7 5 points each This scoring method is used instead of a linear transfor mation of the rankings to avoid producing artificially large numbers in the case of ties To illustrate if each of ten investment options had the same value for a submeasure they would all tie for first If the ranking effectiveness score did not take ties into account each invest ment option would receive 9 points meaning that the total unweighted contribution to the final score summed across investment options by that submeasure would be 90 points By comparison for a submeasure where every investment option had a different value the total unweighted contribution to the final score by that submeasure would be 45 points i e 9 8 7 64 54 44 34 2 4 1 As with the other scoring methods each submeasure S has a weight W in the aggrega tion to the measure score Similarly the weights C qed the relative oa ibution of each measure to the effectiveness score for each investment option The measure score for an invest ment option 7 for measure j is Details of the Methodology 63 Because the maximum possible value of R j for each individual submeasure is K 1 the denominator in the above expression scales the effectiveness score to be between 0 and 1 The effectiveness score for each investment option over all measures denoted is Color Codi
60. and rebuild the sheets However changes can also be made as follows with the disadvantage that they will not be reflected in the Template Builder sheet Figure 4 8 Current Perspective Menu of Summary Sheet Example Specific Default Baseline 1 1 1 Measure 1 Emphasis 2 1 0 5 Weakest Link 2 1 0 5 Figure 4 9 Discount Rate Menu Summary Sheet 36 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Structuring Rows and Columns with Template Builder Looking back to Figure 4 1 and ignoring the control panels at the top and the mapping of colors at the bottom from left to right for each investment option there are four blocks of output data the dimensions of which are hard wired in PAT e Up to 12 columns in a scorecard showing results for Level 1 measures e Up to four columns displaying numeric data e g for selected measures from Level 2 e Up to nine columns displaying cost data e Up to two additional columns presenting the effectiveness and relative cost effectiveness of the investment options Adding or Deleting a Column The user can add any Level 1 measure that has been introduced in the data sheets to the Sum mary scorecard simply by typing the measure s precise name in Row 1 for any of the columns set aside for the scorecard nominally columns B through M the scorecard s range but in any case columns with Detail buttons To delete any column from the scorecard the
61. at some particular value of cost In this example Option 2 buys quite a lot for modest sums of money 10 million Improving results further however requires buying in to a next big increment of capability That is one must spend a good deal more 80 million to achieve a sizable jump in effectiveness Pushing effectiveness to its upper limits the goal of 1 0 is several times more expensive yet 200 million As shown in Quickstart on Using PAT 93 the edited version of the chart Figure A 8 it may be appropriate to connect the points for greater visual clarity In this case we assumed that there are no good options between the points shown so the dashed line constitutes what economists call the efficient frontier At any given level of cost x axis there is no option with greater effectiveness than the line and for any given level of effectiveness y axis there is no option with lower cost Figure A 8 Annotated Scatter Plot 1 0 8 2 ae O B Option number 0 6 u N gt 0 4 Efficient frontier 5 tf Ww 0 2 0 0 50 100 150 200 250 Total costs 2010 2030 M APPENDIX B Transferring Data from an Earlier Workbook Users working with PAT frequently want to move at least some data from a previous workbook into a new one Perhaps some errors have been made and results are now confusing Perhaps one has received an updated version of PAT that corrected some bugs Or perhaps one j
62. ate in diverse missions and circumstances None of these have the luxury of a single objective to be maximized Rather planners are confronted and sometimes confounded by multiple objectives few if any of which can be ignored Nonetheless choices must be made because resources are finite Consequently strategic planning often involves investing in a mix of capabilities and activities to address a mix of objectives It is therefore natural to use the terminology of port folio planning The portfolio itself may be characterized by the allocation across investment categories e g Army Navy Air Force tanks ships and planes or by the corresponding allo cation across objectives e g traditional versus irregular warfare In either case the idea is to balance the portfolio This does not mean spreading money evenly across categories because not all objectives are equal and because attending adequately to one may require much less effort than doing so for another Further given a large baseline of investment such as that enjoyed by DoD and DHS among others some ways of spending a marginal billion dollars provide far more leverage than others Spending or cutting that marginal billion in propor tion to the baseline patterns of expenditure is often irrational Early in 2009 Secretary of Defense Robert Gates made this point as he proposed a defense budget that will rebalance the Soon after the end of the Cold War it was useful
63. bert A Balanced Strategy Reprogramming the Pentagon for a New Age Foreign Affairs January February 2009 Goeller B F et al Policy Analysis of Water Management for the Netherlands Vol 1 Summary Report Santa Monica Calif RAND Corporation R 2500 1 NETH 1983 As of September 2 2009 http www rand org pubs reports R2500 1 Graham John Valuing the Future OMB s Refined Position University of Chicago Law Review Vol 74 Winter 2007 Hillestad Richard and Paul K Davis Resource Allocation for the New Defense Strategy The DynaRank Decision Support System Santa Monica Calif RAND Corporation MR 996 OSD 1998 As of September 2 2009 http www rand org pubs monograph_reports MR996 Hitch Charles J and Roland N McKean Economics of Defense in the Nuclear Age Cambridge Mass Harvard University Press 1965 Kaplan Robert S and David P Norton The Balanced Scorecard Translating Strategy Into Action Cambridge Mass Harvard Business School Press 1996 Keeney Ralph Value Focused Thinking A Path to Creative Decisionmaking Cambridge Mass Harvard University Press 1992 Keeney Ralph and Howard Raiffa Decisions with Multiple Objectives Preferences and Value Trade Offs Wiley 1976 Kirkwood C W Strategic Decision Making Multiobjective Decision Analysis with Spreadsheets Belmont Calif Duxbury Press 1997 Miller Drew Decision Support Systems A Way to Improve Cost Eff
64. best ignored However a checked box means that a measure with a score well below its threshold will get the same score 0 as it would have if it had just reached the threshold An unchecked box assigns a score of 1 in the former case 22 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual ware engineering Without the MRM option making even what seem like simple changes may require entering many low level data values That is both tedious and an invitation to error More important ultimately but more controversial among analysts who like to argue about such things is the fact that the MRM design encourages thinking top down and elimi nates the requirement to enter what may be spurious detail In many instances the informa tion available for the assessment of a particular measure is inherently of low resolution For example if expert consultants with years of managerial experience are asked to judge the risk of some alternative programs they may be able to do so quickly and well However if they are asked to break down their judgments listing components and subcomponents of risk and then estimating values and probabilities of those risks individually the quality of the assess ment may worsen rather than improve The reasons for this are many One reason is that many things can go wrong and experts often smell the potential for problems without being able to predict which particular problems will arise If they a
65. capabilities and activities to address a mix of objectives It is therefore natural to use the terminology of port folio planning The portfolio itself may be characterized by the allocation across investment categories e g Army Navy Air Force tanks ships and planes or by the corresponding allo cation across objectives e g traditional versus irregular warfare In either case the idea is to balance the portfolio This does not mean spreading money evenly across categories because not all objectives are equal and because attending adequately to one may require much less effort than doing so for another Further given a large baseline of investment such as is enjoyed by the Department of Defense DoD and the Department of Homeland Security DHS among others some ways of spending a marginal billion dollars provide far more leverage than others Spending or cutting that marginal billion in proportion to the baseline patterns of expenditure is often irrational Early in 2009 Secretary of Defense Robert Gates made this point as he proposed a defense budget that will rebalance the portfolio by shifting relatively small resources on the margin toward capabilities for irregular warfare and stabilization secu rity transition and reconstruction Dealing with Uncertainty and Disagreement Against this background portfolio analysis should assist decisionmakers to frame their think ing about balance to construct good multifaceted opt
66. center EEEE NEN EA ASEE EE A ERER S 39 Levels Drilldown Sheet ails ei sdiseacparscnsasloitey sedate aaa EERE S A 42 Cost Dara SHEET inosia AA EN EN E NAA NEE actin 44 Scatter Plot Sheet for Cost Effectiveness Latidscapes 2suemsinswrdariena ree netaas 45 Spider Charts Sheet sssi sogeror osiers ie iias t iea Eea n EE AE E AEE E EEE e E E EA 46 Multimeasure Spider Charts SNS Cb esm ereisiresisi eneoti ene ninin eienn aS nS AEn a NEA 47 Selected Derails Seetzen isone e eos ae EE TA A N E 48 Rankings Table Sheets erasoetan inaaianei ket ch ance A GUA nat 48 CHAPTER FIVE Details of the Methodology a2 is sacasixssacasadsonsusscnsncassdssa ans oxsoaensacvancusevnsecnsecvounesinados 51 Basie Concepts and Definitions iirc citvetnnetdviseg Cote cnetdvnden aiduiimatdvbaeg oivenetdvidegerieive 51 Attributes of Investment Optons cnseacsiccsenssscasenthecnsenrs piesa nts veadentnedadeads eessnaseensnaenied 51 Contents vii Measures and Submeasures Level 1 2 and 3 Measures Raw Values and Scores 53 Relative Cost Ei eCriveiess cuit endure nnn 54 Methods and FUNCTIONS yeh oia au sinuisnactepsiesvseeeasien iio a A A EE A AAE a 54 Summary of Let elt oa ana iE EE AEE EEA TEE EEEE EE T IEEE ERE 55 Alternative Methods gs inrnugansnedriardalrminmahiadhiiernoriovehanthsiiinedaribandbdahaninnuebanihtd sanaaiedd 55 The Need for Alternative Methods soscssensosissetssosiiscs ia 55 Goals Method meissrocsisss to eh cundieun itise nan SENEE A
67. ces Further he knows that overall warfighting effectiveness in any particular scenario depends on the balance among the types of forces More aircraft cannot compensate for lack of infantry in some irregular warfare scenarios and more mechanized forces cannot compensate for ceding air superiority to the adversary or for losing the ability to support and sustain operations through sea and air logistical chains Again representing such system issues implies using nonlinearities in the mathematics To reflect common system related issues something other than simple linear sums is needed There are a number of possibilities ranging from using a multiplicative relationship instead of an additive one to methods that involve enforcing threshold requirements for each of the critical components In PAT terminology this corresponds to enforcing threshold require ments for each submeasure that characterizes an investment option 6 The classic introductory book on decision analysis Raiffa 1968 is quite readable A later text treats multiobjective deci sion analysis Keeney and Raiffa 1976 Although it emphasizes combining sums into a single utility the book discusses alternatives to simple linear weighted sums 7 This consideration has led to policy analysis emphasis on scorecards An early RAND application to a Netherlands water management problem Goeller et al 1983 was particularly influential in causing scorecards to be adopted 8
68. creen comes up the user must take the additional step of selecting the desired Level 3 measure from the Level 3 Drilldown Options menu at the top left not shown The result is shown in Figure 4 17 for our continuing simple example In this chart however we have chosen using the display menus to show the numerical raw value figures that determine the colors They might be suppressed in a presentation but the analyst some times finds them useful Note that the final column shows scores of green yellow and red reading downward which agrees with the column for Measure 2 2 in Figure 4 13 Figure 4 17 Level 3 Drilldown for Measure 2 2 Level 1 Measure Measure 2 Level 2 Measure Measure 2 2 Scoring Method Thresholds Measure Measure Level 3 Measure 2 2 1 2 22 Weight of Level 3 Measure in Scoring Functions 0 to 1 Option A Option B Option C PAT Output Worksheets 43 The reader may wonder why such drilldown charts have two apparently identical columns on the right entitled Measure 2 2 Score and Level 2 Score To explain suppose that we change the assumptions of our simple problem slightly by specifying a threshold value for Level 2 Measure 2 2 In that case results would change and the Level 3 explanation would look like Figure 4 18 If we scan the colors at the left of the row for Option A ignoring the numbers for a moment it may seem that they don t average correctly For example Measures 2 2 1 and 2 2
69. d be added for risks of various types at each level of analysis Acquisition of Weapon Systems Broad mission classes within each suitable test sce narios within each suitable technical measures e g probability of being able to penetrate air defenses International Business Operating divisions measures of performance in each operating division diagnostic measures to explain performance Planning of Staff for a Knowledge Industry Business Business areas within each core expertise leadership quality number diversity and affordability of staff mix within say the measure of quality various diagnostics such as experience highest degree or equivalent specialty experience in the business area and past performance Table A 1 Format for Entering Measure Names in Template Builder Mewes f a O a ee monies mere OO O eaa veaa S ee s wees O S e 2 For some actual examples from past studies see Davis Kugler and Hillestad 1997 Davis Shaver and Beck 2008 and Davis et al 2008 86 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual None of the above is to be taken literally The examples are merely starting points If you click on the Show Hide Example button an example will pop up with entries very similar to those used for the example in the main text Click the button again to hide the example Reviewing Your Template Builder Sheet Review your sheet carefully to make sur
70. data to be entered at alternative levels of resolution Top down thinking may begin with rather aggregated assess ments which may be quite sufficient for some purposes Subsequently more detailed assess ments can be adopted in which case the aggregated level results stem from those details Alternatively a bottom up approach can be taken initially but more aggregated inputs can be used for quickly addressing What if questions In practice greater detail is only some Summary xvii times justified and can even be misleading A particular option s technological risk for exam ple may best be judged at a high level because that risk is the result of a myriad of problematic issues not all of which are even identified Illustrative Outputs An Illustrative PAT Summary Sheet Figure S 3 shows an illustrative Summary sheet in which portions that may ordinarily be sep arated horizontally are juxtaposed The left side of the sheet shows that options are being assessed by their effectiveness in two scenarios A and B and by something called Other Mea sures which might in a defense planning context relate to something like shaping the inter national environment The costs shown are total costs for a 20 year period The effectiveness column shows an overall effectiveness across measures Relative cost effectiveness compares the cost effectiveness ratio of an option with that of the most cost effective option except for
71. dget level x axis we see that the best option by the first perspective squares is M T elimination of boost phase That is the topmost square has the color associated with the M T option in the legend box at the right When PAT is being used live the square can be identified by merely mousing over it and seeing its name pop up Table 6 4 Costs and Effectiveness Comparisons Extra Emphasis on Peer Threat epin mo Mine ents Sor Sista eon Cost Option Phase CMs DOFA Sum 6 billions eee ee NOTE The unbracketed numbers are probabilities of successful intercept as in Table 6 2 The numbers within brackets are the effectiveness scores using the Thresholds method Marginal and Chunky Marginal Analysis 75 Figure 6 1 Cost Effectiveness Comparisons for Two Perspectives t a m m 08 e a Se A O 06 4 g a e O S E 3 None o u m m oad T only E M only e E B only E m M T a 02 B T E B M o E B M Te m om 0 3 6 9 12 15 18 Cost B But what if we are more concerned about the peer threat to the homeland Here the circles indicate the effectiveness of each investment option and in this case the same invest ment option is best That certainly might not have been the case as can be seen for example by reading the chart for budget cuts of 3 billion 6 billion and 9 billion and then compar ing the results to what follows In this contrived exam
72. ds of invest ment cases and can display some or all of those cases on its output sheets The output sheet that is perhaps most useful in assisting with chunky marginal analysis is the Scatter Plot where the user can select the x axis from a collection of cost metrics over vari ous time periods constrained to R amp D investment or deployment investment only etc and the y axis from two different evaluation metrics Investment options are plotted on the display as points of different colors and different shapes if more than one measure is displayed on the y axis This is particularly useful in determining which investment option is best and the cor responding programs that should be cut if the budget is reduced from the base case Consider a notional example of a base case that consists of investments in three options called engagement sequence groups at MDA one boost phase one midcourse phase and one terminal phase based in the United States There are three obvious steps away from this base case the cancellation of programs specific to each phase of the defense in general defense systems from different phases may share tracking systems so a cut of a particular phase of the Marginal and Chunky Marginal Analysis 73 defense does not necessarily eliminate all programs associated with that phase Because the number of steps is so small there is no reason to not consider the eight investment options cor responding to all the
73. e generating a recalculation of results by using the corresponding menu Figure 4 8 The entries in the menu are based on the perspec tives the user has defined so far The names in Figure 4 8 illustrate using shorthand names to remind the user of what a particular perspective means In Baseline for example the weights of the three measures of our continuing example are all 1 whereas they are different in the next two perspectives The last perspective changes both the emphasis via weights and the scoring aggregation scheme The final control menu Figure 4 9 allows the user to specify the discount rate which is assumed to be between 0 and 0 1 corresponding to 0 to 10 percent If Cost Data sheet items are in constant inflation adjusted dollars this menu item should be seen as the real discount rate the earning power of money above the inflation rate If Cost Data sheet items are in cur rent dollars the menu item should be seen as the sum of the inflation rate and the discount rate to be assumed in Summary sheet calculations We recommend that users specify the basic structure of their sheets by using Template Builder as discussed at the end of Chapter Three Most of the Summary sheet s columns will then be generated with appropriate headers and values when PAT is first run by selecting Update Summary from the Options menu If the user wishes to add or delete columns or options it is usually best to do that in Template Builder
74. e 2 Measu easure 2 2 Warning Measu 1 0 igh High 0 0 1 0 0 0 1 0 iMeasure 2 Measure 2 Measu easure 2 2 Warning Measu 1 0 igh High 0 0 1 0 0 0 1 0 jenuey a2u 1 4 4 pue spoyya AJOaUL 1Yd JOOL SISAJEUY O1 OJ140g S GNVUY 9Z PAT Input Worksheets 27 Easier Ways to Create and Store Perspectives In practice the easiest way to create perspectives may be by working with the measure weights and the Current Perspective menu in the Summary sheet The procedure also described in Chapter Four is as follows e In the Summary sheet change Level 1 measure weights to those appropriate for the new perspective If desired also change the scoring method used e g Weakest Link instead of Thresholds If desired this is more unusual go to the relevant Level 2 Data sheet the normal one or the MRM sheet and change any of the control parameters that control scoring and aggregation click Modify Summary and then return to the Summary sheet e Go to the Current Perspective menu top left and select Generate New Perspective e When prompted fill in the name of the new perspective and hit return e PAT will automatically copy and paste appropriate data into a new block of the Perspec tives data sheet Figure 3 6 illustrates the Perspectives menu from the Summary sheet It indicates the pres ence of four perspectives Default Baseline Measure 1 Emphasis and Weakest Link just as shown in Figure 3 5 In this example we included
75. e for real discount rates of both 0 03 and 0 07 to bound the calculations For discussion of how the Government Accountability Office GAO came to suggest this see Graham 2007 Such issues were addressed in a recent RAND study of the resource implications of alternative U S global military strategies Davis et al 2008 Details of the Methodology 53 where DR real discount rate E payment promised at end of ith year in real inflation protected dollars The sign is positive or negative depending on whether one is receiving or paying and on the syntax of discussion If the yearly expenditures are expressed in current i e then year dollars without correction for inflation the formula is pr S L E 1 7 TES 1 1 DR where D nominal discount rate E payment promised at end of ith year in real inflation protected dollars DR D lI The approximation is good except for very high rates of inflation From a simple Taylor s expansion the first order correction would be a factor of J i e 0 97 for 3 percent inflation Measures and Submeasures Level 1 2 and 3 Measures Raw Values and Scores The options are characterized at the Summary level by criteria called measures or Level 1 measures each of which has one or more submeasures at Levels 2 or 3 or both We refer to the measures in shorthand as Level 1 Level 2 and Level 3 measures If we refer to submeasures we include both Level 2 and Lev
76. e g on a 0 to 1 scale or visually with a correspond ing discrete set of colors e g red orange yellow light green and green Often however the initial measures are on heterogeneous scales One measure may be in terms of probability 0 to 1 another may count something such as the number of army divisions perhaps 0 to 100 or estimate the expected lifetime of a weapon system in years Such initial measures are expressed in terms of raw values PAT maps the raw values into scores by comparing them to thresholds and goals as shown in Figure 2 4 Because we seek ultimately to describe options in discretized terms the scores have lower and upper bounds No matter how poor or good a raw value is the score will never be lower than 0 or higher than 1 In some instances a higher raw value is worse than a lower value e g higher risk is considered bad In such cases we use a simple variant of Figure 2 4 discussed in Chapter Five Aggregating Scores Level 3 raw values are combined to generate Level 2 scores Level 2 raw values as well as computed Level 3 scores are combined to generate Level 1 scores At some point the Level 1 scores are combined to generate a composite or overall score called effectiveness Cost effectiveness can be calculated or effectiveness can be shown as a function of cost a better practice Figure 2 3 illustrated the relationships schematically showing relative cost effectiveness as the most aggregate
77. e that there are no inappropriate blanks and that syntax is exactly as in the examples This is where most errors occur in building PAT work sheets For example the word Threshold may not appear in the last row listing the measures and submeasures because you added one and forgot to fill in that column If all looks well click Build Sheets top left corner You will be prompted about whether you want to save data If you are starting fresh the answer is No but if you are rerunning Template Builder to reflect some changes in the middle of a project you may want to save the data In fact you will be saving only the data associated with options and measures that still have the same names in the new structure but that may be quite a lot Once Template Builder has stopped running go to the Summary sheet Select Update Summary from the Options menu and the Summary will be generated You should then see the desired structure of headers Check carefully some error may have occurred in which case you may not see all the option names or all the top level measures Click on the Detail buttons to see whether the Level 2 structures are correct Then click on the Level 3 Drilldown tab to see whether the appropriate Level 3 measures if any are available from the menu box at the top left Level 3 Drilldown Options Data Entry Chapter Three presents completed examples of all the data entry sheets so we will be brief here Let us assume that you wa
78. e the weights of the measures are equal We consider three investment options which we call Investment Options A B and C The raw values for each investment option for each submeasure are given in Table 5 6 This and subsequent tables in this chapter are not PAT displays but rather were constructed for the discussion Table 5 4 Color Coding in the Rankings Method Average rank in first quintile Average rank in second quintile Average rank in third quintile Average rank in fourth quintile Average rank in fifth quintile Table 5 5 Summary of Methods Method Submeasure Scores Goals 0 or 1 depending on whether goal is reached Weak As in Figure 5 2 Thresholds Thresholds As in Figure 5 2 Weakest As in Figure 5 2 Link Rankings Modified Borda count Table 5 6 Illustration of Scoring Methods Submeasure Level 2 measure Weight High or Low values desired Threshold value Goal value Level 2 value for Threshold Level 2 value for Goal Goals Method Table 5 7 presents illustrative results for the Goals method Values that meet or exceed the goal value are highlighted in green The table also shows the calculation of the submeasure and measure scores and the measure score cell for each measure is colored to correspond to the color scheme that appears on the Summary sheet in PAT Details of the Methodology 65 Overall Coloring Measure Scores Effectiveness Method Linear weighted sum of Linear
79. ecisionmaker chal lenges a particular assessment out of interest or to test the depth of staff work Alternatively it 1 The primary concepts are described elsewhere Davis 2002a Davis Kulick and Egner 2005 and Davis Shaver and Beck 2008 The latter applies RAND s Portfolio Analysis Tool PAT to acquisition issues Another publication Davis Johnson Long and Gompert 2008 uses PAT in an assessment of alternative global strategies Applications are ongoing with the U S Air Force and DHS Red orange yellow light green and green denote very poor poor marginal good and very good respectively The underlying numerical scores may also be shown but they are usually distracting and misleadingly suggestive of precision Summary xv Figure S 1 Schematic View of PAT Summary Sheet Control Panels rs oe C Ld Numerical Information a ea Effectiveness Measures of Effectiveness MOEs Selected Cost Cost Options A B Cc D E Information Information Effectiveness Mapping of colors into numbers can simply be part of staff work ensuring that analysis is clear well grounded comprehensible and accompanied by an audit trail That is undertaking a project with the expectation of using PAT can help structure the work along the way Alternative Combining Rules Aggregation Rules High level assessments depend on more detailed assessments but how do they dep
80. ectiveness in the Department of Defense Armed Forces Comptroller Magazine Vol 52 No 3 2007 National Academy of Sciences Post Cold War Conflict Deterrence Washington D C National Academies Press 1996 National Research Council Defense Modeling Simulation and Analysis Meeting the Challenge Washington D C National Academies Press 2006 Natrajan Anand and Paul F Reynolds Resolving Concurrent Interactions Distributed Interactive Simulation and Real Time Applications DIS RT Proceedings 2001 pp 85 92 Parnell Greg Value Focused Thinking Using Multiple Objective Decision Analysis in Methods for Conducting Military Operational Analysis Best Practices in Use Throughout the Department of Defense Alexandria Va Military Operations Research Society 2006 References 101 Raiffa Howard Decision Analysis Introductory Lectures on Choices Under Uncertainty Reading Mass Addison Wesley 1968 Ratliff Thomas A Comparison of Dodgson s Method and the Borda Count Economic Theory Vol 20 No 2 2002 pp 357 3721 Reynolds Paul F Anand Natrajan and Sudhir Srinivasan MRE A Flexible Approach to Multi Resolution Modeling Proceedings of the Eleventh Workshop on Parallel and Distributed Simulation Lockenhaus Austria 1997 Willis Henry H James Bonomo Paul K Davis and Richard Hillestad Capabilities Analysis Model for Missile Defense CAM MD User s Guide Santa Monica Cal
81. efense defense of deployed forces defense of friends and allies Future Years Defense Plan Government Accountability Office homeland defense Joint Integrated Contingency Model Joint Warfare System Missile Defense Agency multiresolution modeling operations and support Portfolio Analysis Tool Peace Support Operations Model research and development research development testing and evaluation Synthetic Environments for Analysis and Simulation Synthetic Theater Operations Research Model xxiii CHAPTER ONE Introduction Background Strategic Planning and the Balancing of Investments The motivation for developing RAND s Portfolio Analysis Tool PAT was the importance of balancing investments across multiple objectives in strategic planning This balance is impor tant for planning in such distinct domains as the Department of Defense DoD the Depart ment of Homeland Security DHS international business enterprises and personal finance Today s defense planners for example have objectives relating to force capabilities for future traditional and irregular warfare and for operations other than war The objectives apply sepa rately for different geographical regions and time periods Acquisition planners have objectives of providing future weapon system capabilities in each of many mission areas again for dif ferent operational circumstances and time periods Trainers have objectives such as preparing troops to oper
82. el 3 measures or just Level 3 measures depending on context A given input to PAT is usually made at either Level 2 or Level 3 If a particular input is made at Level 3 then Level 2 information is calculated As is the case throughout mathematics and computer science the same label e g a missile s single shot kill probability is used to refer to the abstract concept that a measure or submeasure represents and also the value ascribed to that measure The measures and submeasures may relate to various types of capabilities or risk They are akin to metrics but it is important to distinguish between the intended measure s concept and the metric that is used to represent it For example one may wish to assess a force structure s capability for short warning cases That is an abstraction whereas the evaluation for a par ticular planning scenario intended to represent short warning cases is a metric Such an evalua tion depends not only on the scenario but also on measures of outcome the models employed and the detailed inputs to those models Raw Values of Submeasures We refer to inputs in terms of raw values gt For a weapon system such data might come from system specifications and an assumption that the specifica tions will be met from test data from models or from expert judgment For high level strate gic assessments of force structure the inputs might be subjective but based on a strong analytic background of
83. ell brings up the comment which might be a cryptic definition or nuance Figure 4 11 If a cell in the scorecard has a similar flag the flag is a warning Mousing over it will cause a dropdown to appear with a brief description of what is being warned about see Figure 4 11 Finally mousing over the name of a measure brings up any comment from the Level 1 Data sheet that may describe the measure it will also show the measure s weight in effective ness calculations Measure Weights The weights of the various Level 1 measures used in calculating effectiveness can be shown or hidden as a group by selecting Show Hide Weights from the Options menu If the weights are shown just below the Detail buttons they can be changed directly in which case PAT re calculates effectiveness and relative cost effectiveness immediately without prompting PAT also modifies the weights of the measures on the Level 1 Data sheet Buttons The Summary sheet includes various buttons Clicking a Detail button will bring up the Level 2 Drilldown sheet for the particular measure column Clicking a Related Details button for any of the numeric data columns will also bring up the relevant Level 2 Drilldown sheet Clicking the Cost Detail button opens the Cost Charts sheet There is no button directing access to Level 3 information One accesses Level 3 Drilldown as discussed in the subsection on that topic below An Illustrative Summary Level Scorecard Figure 4
84. ely as shown one row for the Level 1 measure s name one row for each of the Level 2 measures names including items called Warning The names of the investment options are entered as row headers All such names must be consistent across workbook sheets For example the Level 1 measure names must agree with those in the Level 1 Data sheet Hint Editing names which is commonly important to improve intuitive clarity is best done by using Excel s Find and Replace function applying it to the workbook rather than to the current sheet rather than attempting to type consistently across sheets Data on Option Effectiveness The lower part of Figure 3 2 below the yellow divider is filled in with Level 2 data for each investment option and measure Most of the data can be directly inputted at this level but Measure 2 2 s italicized values indicate that they are calculated from Level 3 information If the analyst types over the italicized numbers the calculated numbers will be regenerated in the next PAT run overwriting the corrections Thus any changes to Measure 2 2 s data must be made at Level 3 Any entry in the Warning column for a particular option should be text as in the example shown for Measure 1 Such warnings cause little flags to appear in cells of the Summary scorecard They appear in pop ups if the user mouses over the flags see Chapter Four The data on the options may come from a capabilities model or other so
85. end on them When information is sent upward as in Figure 2 mathematical rules must be speci fied for doing so Should lower level assessments simply be averaged Should linear weighted sums be used What if the problem itself is not linear Options for systems or strategies may have zero effectiveness in the real world if any of various critical components fail A better weapon buys nothing if it cannot be delivered to the target A stabilization strategy may fail if any of its political military or economic components is severely lacking despite the quality of the other components The natural measures in PAT analysis often include some that are individually critical and some that are important but perhaps substitutable Recognizing such complexity we designed PAT to accommodate a wide range of combining rules Some are built in and can be chosen from a menu others may be analyst defined Alternative Perspectives As discussed above the concept of strategic perspective is important in framing issues and evaluating options In our work with PAT a strategic perspective translates into a coherent set of assumptions about the way objectives are weighed and assessments of capability for those objectives calculated and combined in aggregating from lower levels One perspective for example might emphasize relatively near term capabilities and activities along with currently relevant test scenarios whereas another might emphasize longer term
86. endabiadoneiwhedendediadwheenhedanss 9 Menu of Input and Output Sheets ci2sinciiveatnug ao a AE A A e 11 Tabs for Moving A mong She tsisc enosmerno ro eene tiia EEEa E EEEE 12 Schematic of Summary Sheet rispe sieeve heinehlet a Wig wie aee gles aa Eea EEE ie Er Ea GS 13 Schematic of Drilldown Z00mM sss sessiciisrsssiiisrri strast a drana rna UNEREN NE EENES NE EARO RI 14 Sample Output Displays from PAT i cucaincs i caencupicesvbaeentaeasversrais nana eneanees 15 Level DEE E E 18 Level 2 Data Shestani dake vse E AE E E T S 19 Level 3 Data Shecta cous cree re a nr e ara E E EAE T E EET ETAT 21 G st Data Sheet sei to ansaan A E E A E A E N 24 Mlustrative Perspectives Cases saris sas dimaatad ars satanaimantndass hacamniwiltom ary iaanntenninds 26 Using the Summary Sheets Perspectives Menu i6 45s sasispsiapscadhiehsaagieweiauadabees 27 Query About Saving or Creating a New Perspective oi ciiiescccaseisedsineesionsswraedonne 28 Template Builder tor a Simple Example yj ic tccisapedapleder aedeaimniagheleniaiipndraheinned 29 Illustrative Summary She ty ic fiseisi dan avarcndnciiesdaiea de iundsutlanduin adie ventsuvanantnavadien 32 Options Menu or Summary Sheet rciris isiotu taiii iva inbiria 33 Alternative Color Schemes for Scorecards 2is1c seis siionsgersveninssdivctstsdiesiwestvestaves 33 Sorting Categories Menu of Summary Sheet Example Specific 0 0000ee 34 Sorting Method Menu of Summary Sheet 00 00sec cece cee
87. ere various numbers came from This can be done to a significant extent within PAT by for a particular measure drilling down or zooming we use the terms interchangeably to sheets describing matters at Level 2 and Level 3 Figure 2 8 shows this schematically suppressing all aspects of the sheets except the scorecards at Level 1 Level 2 and Level 3 In the example the user drills down on Measure 2 of the Summary chart discovering that it is based on two Level 2 measures If he drills down Figure 2 8 Schematic of Drilldown Zoom Measure 1 Measure 2 Measure 3 Detail Detail Detail Option C Summary Llevel1 9 oo Level 1 Measure Measure 2 Level 2 Measure Measure 2 1 Measure 2 2 Warning Weight of Level 2 Measure in Scoring Functions 0 to 1 Al Level2 Level 3 Overview of PAT 15 further on the second of the two Level 2 measures he finds that Measure 2 is based on calcu lations using three Level 3 measures In most cases this drilling down provides an adequate visual explanation of results However such is not always the case and it may be necessary to study more analytically detailed results from capability models and the equations used to generate aggregations PAT also generates numerous graphics which can be useful in the course of analysis and in the presentation of results Figure 2 9 illustrates several types of built in graphics a line chart a s
88. erspectives sheet for our continuing example It illustrates the format for entering alternative perspectives starting with a blank sheet or one that already includes a default perspective Each perspective is a block of rows Each block has a header row with the name of the perspective the rest of the block has precisely the same structure as the top portion of a Level 2 Data sheet That is it includes the part that specifies the weighting factors thresholds goals and so on for each Level 2 measure The blocks contain no information about the options The blocks corresponding to perspectives are juxtaposed i e there is no space between them The weights of the Level 2 measures appear in the header line In the example the weights appear in every relevant column but it is permissible to have them appear only once per group The user can create a full set of perspectives directly on a previously blank Perspectives sheet by adhering to the syntax of the example It is relatively easy to make data entry mistakes however Once changes are made in a given perspective they will take effect only if one of them is chosen anew in the Summary sheet as the current perspective If a change is made to the cur rent perspective in the Perspectives sheet the user should select a different current perspective in the Summary sheet s menu and then change again the perspective in question That will refresh the current perspective appropriately Currently
89. es for investment Level 2 or Level 3 Data i Vil aes gal ijn option i and measure j s 5 5 Scores of the n submeasures submeasure scores Calculated AGW MP2 MT jn for investment option i and measure j M pM gt M p Scores of the m measures measure scores for Calculated i i tom investment option E Overall effectiveness of investment option i Calculated Threshold raw values for each submeasure of measure j Corresponding goal raw values for each submeasure of measure j Submeasure scores when submeasure raw values Level 2 or Level 3 Data Level 2 or Level 3 Data Level 2 or Level 3 Data have reached thresholds under measure j Scores of submeasures that have reached their goals Level 2 or Level 3 Data under measure j Weights of the submeasures in computing measure Level 2 or Level 3 Data scores WIWE WE j J2 jn Weights of the measures in computing effectiveness Level 1 Data VO Cai Aggregation to Find Overall Effectiveness with Goals Method The function used to set the overall effectiveness under investment option 7 is also a weighted sum over measures LCM 5 C j l E L The measure and effectiveness scores all have values between 0 and 1 The weights need not be between 0 and 1 since PAT performs normalizations but it is arguably good practice to enter weights in that range that add up to 1 so that the significance of a given weight s value will be more readily understood wi
90. f diverse other information such as structured expert judgment PAT can accommodate very different management purposes including highlighting prob lems to be confronted emphasizing accomplishments rather than residual shortcomings and making the relatively soft and forgiving assessments that are common in broad balance of power studies Seeking Flexibility Adaptiveness and Robustness Some who hear of PAT for the first time may think of using it to optimize resource allocation mathematically They may see the mechanism for calculating cost effectiveness and assume that the objective is or should be to maximize that quantity That would be a misreading of our intentions and indeed of the philosophy underlying our approach The most important outputs of PAT are 1 the portfolio style scorecards in which alternative investment options are assessed simultaneously by a number of very different measures and costs and 2 the next layer of scorecard detail to which the viewer is able to drill down to understand the basis of the color coded summary assessments and to change higher level assumptions or priorities that affect those assessments Further aggregation to a single number as in cost effectiveness calculations should be deferred to a kind of refinement stage a stage in which one is tidying and thinking about communicating the results of decisions We recommend this because the cost effectiveness 1 A capabilities model is a
91. fec tiveness and relative cost effectiveness Those show up only after relevant options and data have been entered The Detail buttons allow you to drill down to a second level of detail one that explains the summary level results for the column in question Template Builder Template Builder s Structure Go next to the Template Builder sheet at the far right of the tabs or via Go To Sheet Inputs This is where you enter the information that dictates the overall structure of your portfolio analysis Figure A 2 shows Template Builder s structure The various blocks in dashed edge red rectangles are independent That is do not assume that two items relate to each other just because they are in adjacent columns Items in a given block relate to each other This is what you have to do for each block the red text items are placeholders e Fill in the names of your options one line per option Baseline is the placeholder example e Indicate the range of years that you wish to consider 2010 2030 in the example e Define the units of currency Thousands in the example although that surely doesn t apply to government e Enter the names of Level 1 Level 2 and Level 3 measures along with some control parameters for the latter This is a large and relatively complicated block 84 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual 4 i TETE t i oezo i 3 5l i ager i i gt
92. fectiveness Comparisons Equal Emphasis on all Scenarios 74 Costs and Effectiveness Comparisons Extra Emphasis on Peer Threat 74 Format for Entering Measure Names in Template Builder 0 00 cceeeeeee 85 Level 2 Data for Illustrative Exerciser ocscocscurniccereronee 88 Level 3 Dati tor Mustrative Exercise 5 c9scdandsdiacehaiwhednedadviwhiseiataneibiadnidnebadandy 89 ost Data for Illia cian ve EXerCissroionsre ia yin T Aa E OAE 89 xi Summary Challenges of Strategic Planning Strategic planning often seeks to balance investments across numerous objectives Defense planners for example have objectives relating to force capabilities for future traditional and irregular warfare and for operations other than war The objectives apply separately for differ ent geographical regions and time periods Acquisition planners have objectives of providing future weapon system capabilities in each of many mission areas again for different opera tional circumstances and time periods Trainers have objectives such as preparing troops to operate in diverse missions and circumstances None of these planners have the luxury of a single objective to be maximized Rather they are confronted and sometimes confounded by multiple objectives few if any of which can be ignored Nonetheless choices must be made because resources are finite Consequently strategic planning often involves investing in a mix of
93. for DoD to think in terms of balancing across the objectives of shaping the international environment e g through alliances and forward deployment of U S forces having capabilities appropri ate for deterring or fighting wars in the near to mid term and building future capabilities to deal with emerging threats and opportunities what came to be called military transformation An early analysis proposing such portfolio balancing was Davis Gompert and Kugler 1996 DoD s actual strategy was called Shape Respond and Prepare Now Cohen 1997 it specifically addressed all three objectives despite the erroneous claim by outside observers that it was merely to build forces for two major regional conflicts of the sort considered plausible in that period 2 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual portfolio by shifting relatively small resources on the margin toward capabilities for irregular warfare and stabilization security transition and reconstruction Dealing with Uncertainty and Disagreement Against this background portfolio analysis should assist decisionmakers in framing their think ing about balance constructing good multifaceted options for consideration and making sub sequent choices Such analysis must include solid data and accurate calculations but it will also include subjective inputs and analysis under deep uncertainty Further it must deal with the reality of major disagreements a
94. h vaelio aay asn wai AA AAE ET EERE R 45 Scatter Plot for the Simple Problem Using Two Scoring Methods 2 46 Mlustrative Spider WWE si cae cavucy du ax eaae a E E 47 Multimeas re Spider Plot aiavcasune via atuauivaaaeigeveaeangeean cen E EO 48 Selected Details SMC cisthaenrcastrnnanaicwny acm oudpaah E e tanga aaisthoten aida OA E EES 49 Rankings Table Sheet for Effectiveness or Relative Cost Effectiveness 05 49 Schematicot PATS Alewlations cnncarneticuiradh nalaceinaan EEG 52 Score Versus Raw Value for Goals and Thresholds Methods 0 0000e0 0s 59 Cost Effectiveness Comparisons for Two Perspectives isi scsacsscivinesvenssadsauedecontas 75 Overview of Summary Sheets eser a E aenea eaa EEE EEE AR EEN a Es 82 Illustrative Template Builder Sheet rrieretecerotni isrener aiei iE 84 Tug Bar for Viewing Separated Portions of an Excel Spreadsheet 0 0005 90 Summary Sheet Excerpt for Exercise Problem 0 04 ssacssassuswsodaessiearevensesiacdes 90 Level 2 Drilldown for Exercise Probletty 050 02s vniwvsivecehdesveveneevanguustiastewscsaaeees 91 Level 3 Drilldown for Exercise Problems rissie dart zseietewnsxtebenudncevercerieeanianees 91 Scatter Plot of Effectiveness Versus Cost for Exercise Problem 00000005 92 Annotated Scatter Plat sca savas snivanalee rnio aan E E A E E E 93 Tables 2 1 2 2 2 3 5 1 D2 5 3 5 4 D2 5 6 5 7 5 8 5 9
95. hat any of a number of operations with PAT will cause recalculation in which case your overwrites will be overwritten If you really want to think at Level 2 you should be using the optional Level 2 MRM approach Note Enter 0 and 1 as the values for Threshold Value and Goal for the columns of values calculated from Level 3 since PAT has generated numbers assuming this result When you are done entering data click Modify Summary top left You will be taken to the Summary sheet where you should see the colorful scorecard display Costs Next you need to enter costs for the options Do this in the Cost Data sheet It will have a structure corresponding to the number of years that you specified in Template Builder You need to enter costs by year by option for each category of investment and for each type of investment item which may simply be none as in the example To save trouble at this point you can enter all the costs in the cells for 2010 Or you could go back to Template Builder and start over again with only a single category Total and a single item Stuff The default is that the costs are in millions Once you re done click Modify Summary Cost Effectiveness You will be taken back to the Summary sheet If you scroll far enough to the right you will see columns with the costs an effectiveness measure and relative cost eftectiveness The effective ness is merely a linear weighted sum over the vari
96. hat each investment option is trying to achieve Submeasure Scores with Goals Method A submeasure s score is 0 or G depending on whether or not the raw value has reached the goal That is for any investment option 7 and any measure j if the scale is increasing so that goals correspond to high values then the Ath submeasure s score is given by Sija 0 if Vs ja Sik Gin if V jk 2 Fi G a If goodness increases with decreasing raw values the equations change accordingly Aggregation to Find Measure Scores with Goals Method The function used to calcu late the score of a measure is just a weighted sum of the submeasure scores LW Si M i j n LWG ja k 1 If all submeasures are equally weighted the measure score is the fraction of the mea sure s submeasures that reached their goals The resulting measure scores are between 0 and 1 because of the normalization accomplished by the denominator In practice with PAT the values of G will usually be set to 1 with the W values establishing the relative weights of the submeasures but for the sake of completeness we include G in our equations 58 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Table 5 2 Notation for Defining Scoring Methods Symbol Meaning Source i j k Indexes for investment option measure and n a submeasure respectively m Number of measures n a n Number of submeasures of measure j n a Raw values of the submeasur
97. hat the analyst defers even develop ing a Level 3 description until after considerable work has been done at Level 2 to sharpen appreciation of where additional detail is actually useful This is analogous to what many good designers architects and analysts do routinely proceed top down It is the analyst s responsibility to maintain consistency among the several data sheets Translating Raw Measures of Value into Scores Any scorecard method and any of the classic decision analysis methods such as those using utility functions requires that the various measures of options goodness be on a common 3 The theory of MRM has evolved over the past 20 years Davis and Huber 1992 Davis and Hillestad 1993 Bracken Kress and Rosenthal 1995 Reynolds Natrajan and Srinivasan 1997 Natrajan and Reynolds 2001 Davis and Bigelow 1998 Davis and Bigelow 2003 Yilmaz and Oren 2004 A number of detailed applications to defense problems have also been published Davis Bigelow and McEver 2001 Davis 2002b National Academy of Sciences 1996 Davis McEver and Wilson 2002 8 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Figure 2 3 Four Levels of Detail Relative cost effectiveness versus cost Relative effectiveness Effectiveness N Level 1 Measure A Measure B Level 2 Measure A 1 Measure A 2 Measure B 1 Measure B 2 Level 3 Measure A 2 1 Measure A 2 2 scale This may be done quantitatively
98. he removal or addition of whole purchases As with marginal analy sis these purchases are removed individually from the current investment with each possible 71 72 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual removal defining a new investment option The costs of the resulting investment cases provide another measure that can be used as a basis for comparison The important question is often not Where do we spend the next dollar but rather How do we invest this extra 50 mil lion Pessimists or realists will suggest that the question asked more often is How do we handle this 50 million budget cut This chunky marginal analysis method will be described in more detail below as will the application of PAT to assist with this analysis Chunky Marginal Analysis for a Ballistic Missile Defense Example We consider as a base case a fixed collection of ballistic missile defense system BMDS invest ments over time in both R amp D and acquisitions We generate a collection of investment options by considering variations from the base case These steps are large scale additions to or sub tractions from the base case Some examples would be the cancellation of R amp D on a particu lar program or a reduced or increased acquisition of a radar system or interceptor The steps should not be so small that it is impossible to discern the base case from the BMDS associated with the new investmen
99. if RAND Corporation TR 218 MDA 2006 not available to the general public Yilmaz L and T Oren Prospective Issues in Simulation Model Composability Basic Concepts to Advance Theory Methodology and Technology The Modeling and Simulation Information Analysis Center MSIAC e Journal 2004 Zeigler Bernard Herbert Praenhofer and Tag Gon Kim Theory of Modeling and Simulation 2nd ed Integrating Discrete Event and Continuous Complex Systems San Diego Calif Wiley 2000
100. il Total Cost 2010 2015 Relative Cost M Effectiveness Effectiveness Detail Detail Detail Detail 2561 0 5 1 2177 0 53 1 25 852 0 25 1 5 13 At the top of the sheet are some controls At the bottom of the sheet is a translation from color coding into numbers In the main portion of the sheet rows represent investment options Each option is scored by different criteria or measures represented by the columns The first block of assessments is the color coded scorecard of Level 1 measure values A B C Moving rightward blocks contain numeric values such as of selected measures that the analyst wishes to highlight cost data and effectiveness and cost effectiveness values Ordinarily the user will have only a portion of the Summary sheet visible e g the scorecard portion The primary challenges in working with such high level depictions are assuring that they frame the problem well highlighting the right considerations and assuring that the depictions reflect assessments based on solid analysis with a good audit trail The analysis may be based on 14 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual information from capability models structured subjective judgment sources expert judgment based on detailed studies or other sources Users should have the ability to understand why a given cell of the Summary sheet is red rather than green bad rather than good or wh
101. in the Summary e g Thresholds However the user can vary the scoring method and weights in this sheet in order to test some variations He should use the buttons at the top right to make the changes take effect and he should be cautious about using different scoring systems in the Summary sheet and this sheet Depending on which options are selected from the two Display menus only some of the information in the sheet is shown Users will not always want to see the information above the scorecard For production purposes the color explanation chart at the very bottom is also not very interesting Figure 4 14 shows a screenshot with many of the upper rows suppressed using the Display menu and with the color explanation chart left out The rectangle indicates that we will drill down on Measure 2 2 The italic letters in the boxed column indicate that this measure is calculated from Level 3 At the price of disrupting flow from Figure 4 14 to the Level 3 Drilldown sheet let us first note that Level 2 Drilldown displays look somewhat different when other scoring aggregation methods are used If the Goals scoring method is selected cells will be green or red depend ing on whether or not the raw value for each Level 2 measure reaches the goal Figure 4 15 If it does so the rightmost column gives the weighted percentage of goals achieved over all of the Level 2 measures for the particular Level 1 measure The color of the cell in the right most
102. ings method measure level scores or simply scores are defined as being between 0 and 1 Overall Effectiveness Scores Once PAT has calculated scores for the measures charac terizing the investment options it can also generate the options scores for effectiveness which is shorthand for composite or overall effectiveness Relative Cost Effectiveness An option s cost effectiveness is the ratio of its effectiveness and cost but different costs can be used for the denominator A PAT user can select any of the inputted cost categories e g R amp D acquisition or O amp S or their total for the inputted time span e g 2010 2029 PAT then calculates an intermediate cost effectiveness for each option not shown in Figure 5 1 uses the largest value as the base and compares all options cost effectiveness to that The result then is the relative cost effectiveness for each option Methods and Functions The progression summarized in Figure 5 1 requires numerous functions FF F as indi cated in Figure 5 1 These accomplish the following transformations e From submeasure raw values to next level scores from Level 3 to Level 2 and from Level 2 to Level 1 e From submeasure scores to measure level scores potentially at both Levels 2 and 3 e From costs of various types to a single cost used in cost effectiveness calculations e From Level 1 scores to effectiveness scores e From effectiveness scores and cost to cos
103. ion or of inflation plus the real discount rate as needed in present value calculations Figure 3 4 illustrates the format of the Cost Data sheet For compactness this example has only a three year time frame and two types of investment items A more realistic example would require scores of columns and rows To specify cost structure and data starting with a blank Cost Data sheet the user should specify one block of columns for each cost category Each block should have the same number of columns with the same range of years One block of rows should be entered for each option Each block should have the same set of rows representing different cost items over the same time period Each block of columns and rows should be separated from the next by one column or row respectively The analyst is responsible for entering appropriate data such as current year also called then year or real inflation corrected costs For present value calculations of cost effectiveness a real discount factor can be specified here or in the Summary sheet That will override data in the data sheet when making calculations for the Summary sheet That is if the Cost Data sheet has constant dollar entries but the Summary sheet specifies a discount rate of 3 percent the dollar values listed in the Summary sheet and used for cost effectiveness calcu lations will be in present value terms assuming that the value of money is 3 percent on top of inflation An Easier A
104. ional Academy study National Research Council 2006 3 As PAT is continuously improved more automatic updating is being introduced Concluding Observations 79 tions and effectiveness calculations If PAT used only linear weighted sums the methodology would be simpler but less satisfactory for system analysis Although we cannot itemize here all the considerations that an analyst should have in mind a few are particularly worth mentioning if only as a partial checklist e Measures and submeasures should provide an adequately complete assessment e Ideally measures would be independent and the submeasures of each measure would be independent When that is not appropriate i e when correlations exist weighting factors should be chosen so as to avoid results being overly sensitive to a single underlying issue 4 e The choice of scoring and aggregation methods merits particular thought If all of a mea sure s submeasures are individually critical then the Threshold method may be called for If all of the measures are also individually critical then the Weakest Link Method may be appropriate If these conditions do not apply however and one is more interested in seeing progress than in flagging problems then the Goals method may be appropriate The Rankings method can be helpful when for one reason or another it is inappropriate to discuss goals and thresholds e In any case it is essential to plan for systematic exploration
105. ions for consideration and to make xiii xiv RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual subsequent choices Such analysis must include solid data and accurate calculations but it also includes subjective inputs and analysis under deep uncertainty Further it must deal with the reality of major disagreements among senior leaders disagreements that we can partially capture under the rubric of strategic perspective A strategic perspective corresponds analyti cally to a way of weighing various objectives and priorities and assessing options adequacy in meeting them Much of this is about managing strategic risk The essence of strategic decision making is often either choosing a perspective or crafting options that will be valuable across important perspectives Analysis can help by making perspectives related issues explicit and in some cases by suggesting nuanced alternatives that are seen as having cross perspective value RAND s Portfolio Analysis Tool RAND s Portfolio Analysis Tool PAT was designed to facilitate strategic portfolio analysis dealing with both uncertainty and differences of perspective It reflects lessons learned with earlier tools and has evolved considerably as the result of various RAND studies since 2005 Comparing Options by Various Measures and by Cost PAT generates high level summary depictions for discussing issues of balance It uses a spreadsheet based format with options
106. ions of Worst Case Option B 1 B 2 B 3 1 Base 0 1 0 1 0 1 2 0 1 0 3 0 3 3 0 1 0 3 0 3 4 0 3 0 5 0 3 5 0 5 0 5 0 3 Table A 4 Cost Data for Illustrative Exercise Option Cost millions Baseline Option 1 0 2 10 3 80 4 160 5 200 Now suppose that as a first approximation you think of overall effectiveness as an aver age of that for Scenarios A and B and as an average of results for best estimate best case and worst case variants Indeed even in thinking about the worst case outcome for Scenario B which has troubling variants you take an average With this background the exercise is as follows e Starting with Template Builder set up PAT appropriately and enter data recognizing that the data provided here are not in the proper format for PAT That is you will have to translate these data into PAT s terms e Specify weights thresholds and goals appropriately e When everything is working look at the various output sheets to check for any egregious mistakes e Recalling that different types of information are separated horizontally on the Summary sheet see Figure A 1 use standard Excel functionality to arrange the sheet so that you can see the scorecard and immediately adjacent to it the columns for cost and effective ness This will require hiding some columns and creating a two pane view To hide col umns select them and choose Column Hide from the Excel Format menu To create the two pane view use the
107. lation of effectiveness and cost effectiveness Change Color Scheme As shown in Figure 4 3 PAT provides four different color schemes for the scoreboards the standard five color style a style with letters added to indicate the colors a style with alternative colors and a gray scale style The user can rotate through the set of styles by repetitively selecting Change Color Scheme Adding letters top right is valuable for people who are color blind Gray scale is sometimes seen as less dramatic Update Menus In some instances the user should update menus For example if the content of the Perspectives Cases sheet has been deleted to define fresh perspectives the menu will continue to show the old ones until it is refreshed Generate New Perspective This option will copy current settings to the Perspectives sheet with whatever name is specified in the query box that arises after the option is chosen Update Perspective This option replaces data in the Perspectives sheet with data being used in the currently operative perspective whether set in the Summary sheet or elsewhere Further tuning of colors and patterns is possible using Excel s built in capabilities but doing so is tedious and in our experience unrewarding because of Excel s limited palette To obtain the colors intended it may be necessary to recalibrate one s printer or computer 31 Figure 4 1 Illustrative Summary Sheet Options Menu Sorting Catego
108. ld score 0 goal value 1 and goal score 1 respectively 44 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual threshold is 0 the goal is 1 and the score at the goal is 1 The value of 0 8 to be evaluated the Measure 2 2 score is 59 percent of the way between 0 51 and 1 Thus the Level 2 score is 0 59 as shown which is just within the range associated with yellow PAT s flexibility in allowing the user to attach goals and thresholds is powerful but using goals and thresholds requires caution and attention to detail especially if they are used at both Levels 2 and 3 4 One price is that a visual explanation may not be enough in drilldowns it may be necessary to look at the details as with Figure 4 13 Cost Data Sheet Costs are an input to PAT as discussed in Chapter Three see Figure 3 4 but PAT also gener ates output charts If for example the user specifies cost streams for categories R amp D acquisi tion and O amp S by option by year the Cost Data sheet will show plots of yearly or cumulative cost by option and category or total costs across categories Figure 4 19 illustrates the latter It expresses costs in current dollars also called current year or then year dollars but the user will often wish to use constant dollars also called real dollars or present value calculations as discussed in Chapter Three Figure 4 19 Illustrative Total Costs Versus Time Chart 3 000
109. lled Related Details 34 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Figure 4 4 Sorting Categories Menu of Summary Sheet Example Specific Measure 1 Measure 2 Measure 3 Total Cost 2010 2015 M R amp D Cost 2010 2015 M Acquisition Cost 2010 2015 M O amp S Cost 2010 2015 M Effectiveness Relative Cost Effectiveness Figure 4 5 Sorting Method Menu of Summary Sheet Largest value first Smallest value first As discussed in Chapter Three the user can at any given time choose to work with any of several versions of his data sheet This is the MRM feature for which the choices are Figure 4 6 entering data at Levels 2 and 3 referred to as Use Level 3 Data entering data at Level 2 or entering data at Level 1 The user controls the scoring method actually the method of aggregation which some times has substantial effects The choices available from the related menu Figure 4 7 are those discussed in Chapter Three The Thresholds method is the default choice If no change is made to default thresholds and goals the familiar method of linear weighted sums is used Figure 4 6 MRM Menu of Summary Sheet Use Level 3 Data Use MRM Level 2 Data Use MRM Level 1 Data Figure 4 7 Scoring Method Menu of Summary Sheet Goals Thresholds Rankings Weakest Link Weak Thresholds PAT Output Worksheets 35 Current perspective can be changed at any tim
110. lue scale showing how the investment options rank relative to one another The rightmost column shows a weighted average ranking The score and color of the cell will be carried up to the Summary sheet for that investment option and measure Figure 4 14 Compressed Version of Level 2 Drilldown for Measure 2 PAT Output Worksheets 41 Level 2 Measure Measure 2 1 Ont Figure 4 15 Level 2 Drilldown with Goals Method Level 2 Measure Measure 2 1 Measure 2 2 Warning Figure 4 16 Level 2 Drilldown with Rankings Method 1 1 rr Option B OH s 0E E Option C 0 50 0 00 42 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual The raw values and the scoring function parameters may be edited in the Drilldown sheets even though they are nominally output sheets Once the data and parameters have been changed the user should click on the Update Level 2 Data button which will update the data sheets and will also recreate the Summary and Drilldown sheets with the new information Level 3 Drilldown Sheet Returning now to the stream of examples using the Thresholds scoring method and pick ing up with the drilldown shown in Figure 4 13 the Level 3 Drilldown sheet is shown in Figure 4 17 To access it requires a small change of procedure The user must first click on the Level 3 Drilldown tab or go to that sheet using the Go To Sheet menu at the top of the Excel window When the next s
111. mail order rand org Preface This report documents RAND s portfolio analysis tool PAT which was developed at the RAND Corporation for the Department of Defense but should be useful in other strategic planning organizations in government and the private sector as well The report documents theory and methodology it also serves as a combination reference manual and user guide In a sense it is a second edition because PAT builds on an earlier application specific tool PAT MD which was developed for the U S Missile Defense Agency s Program Integration Office MDA PI Dreyer and Davis 2005 Since PAT MD was developed however we have enhanced it substantially and have used it in a number of very different projects for the Office of the Secretary of Defense the Joint Staff the U S Air Force and the Department of Home land Security Because of these enhancements and expressions of interest by potential users outside of RAND we undertook a start to finish revision of the documentation The approach to analysis enabled by PAT is oriented toward supporting high level deci sionmakers The reasoning behind the approach is described in this report most of which however is technical documentation that will be of interest primarily to analysts and those who manage analysis The report assumes that the reader is at least moderately familiar with Microsoft Excel Since PAT is an evolving tool questions and comments are especially welcome
112. mong senior leaders disagreements that we can partially capture under the rubric of strategic perspective A strategic perspective corresponds analyti cally to a way of weighing various objectives and priorities and assessing options adequacy in meeting them Much of this perspective is about managing strategic risk The essence of strategic decisionmaking is often either choosing a perspective or crafting options that will be valuable across important perspectives Analysis can help by making perspectives related issues explicit and in some cases by suggesting nuanced alternatives that are seen as having cross perspective value Portfolio Analysis Tools Definitions We use the term portfolio analysis tool to mean a tool for comparing investment options according to a number of quantitative and qualitative criteria including costs upside poten tial and downside potential risk In strategic planning such a tool can generate holistic top down depictions of alternatives and their possible implications perhaps over many years into the future Such a tool can assist in balancing programs either with a start fresh approach or in marginal analysis i e assessing where to add or subtract the marginal dollar RAND s approach to strategic portfolio analysis has evolved over the past dozen years It has been applied at several very different levels of analysis e Force planning for the Office of the Secretary of Defense in the mid 1990s
113. n The scorecard information is especially important because decisionmakers need to understand how the options fare by dif ferent criteria e g in the two scenarios Balancing across such measures is fundamentally in the province of decisionmakers and they are ill served if they are presented only with super ageregated information such as a nonintuitive single metric pretentiously called effectiveness So also the drilldown feature is in our view essential This said after decisionmakers have oriented themselves adequately and discovered or asserted their preferences and the alternative strategic perspectives they wish to take seriously it can be quite useful to generate visuals that exploit the simpler metric of overall effectiveness This can be useful for communication and identifying marginal changes to improve cost effectiveness Figure S 4 illustrates this with what we call a cost effectiveness landscape It plots the overall effectiveness and cost of each option and does so for each of two strategic per spectives A and B If enough well chosen options have been considered such a chart can com municate important information on what is gained as a function of expenditure The results for xviii RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Figure S 3 Illustrative Composite Summary Sheet Total Cost Effect in Other 2010 2030 Relative Cost Measures ScenarioA Scenario B Measu
114. n to deeper levels The focus of our work is at Level 1 Level 2 and Level 3 At these levels PAT s displays assess the options by a number of criteria called measures At Level 1 for example an illustra tive measure might be effectiveness in Scenario A another might be risks i e downside poten tial the potential for doing worse than is shown in the effectiveness columns More extensive examples are presented later Multiresolution Data Entry The straightforward way to build such a multilevel system of outputs is bottom up which typically means developing detailed data sheets Those how ever can be tedious to build and to use especially if uncertainty analysis is needed Further the complexity and detail of such sheets can result in errors as analysts vary assumptions We designed PAT to have multiresolution data entry This means that PAT maintains several data sheets At any given time PAT will run whichever data sheet is specified The choices are called Level 3 which actually means that data can be entered at Level 2 or Level 3 MRM Level 2 and MRM Level 1 The labeling stems from the relationship to MRM We shall make this more concrete later with an example For agile exploration of changed assumptions the analyst can use the MRM Level 2 Data sheet because it has many fewer items to specify Later after understanding the issues better he may revert to the more detailed work It may also be t
115. neubhrdnesddiedoneed acnreibneneanehasesedadennendaeiaeas 11 lop ts and OU le icii E A E EE EEE EEE O ESE EAEAN 12 CHAPTER THREE PAT Input Worksheets Aners E T E EE E T T a 17 Template Builder sriep eneon eE a EE EENE EE N NE E 17 Tevel Weta sacs orcanevuuthssecteenesnes ie ak ET a aE E EEEE SEE DOE ESEE USN ERINES 17 vi RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Feye Data eee ee ee eee oe ene ee een eee are ee ee ere ee ee reer ee 18 Names on Measures and OPNS isori an E E EE E AETA O 18 Data on Option Effectiveness sins daa cha ccavenstensa axpuciavedsalnesanepiethunds EPES EEEE E EEEE EErErETEEn 18 Parameters Specifying the Nature of Scoring and Aggregation sisicicscsrerrresrsrrrrrrsren 19 Warnine Comments kesene aaa a e n hind ine a nepe ts 20 The Modify Summary Batten isaccsciuncunnenurnenananenin S ANSE 20 Level 3 Data miisssisiso erore ro Kiviai ETNE IEEE EENE RETTE EENE EIKES EN SE OEE 20 MRM Level 1 and Level a dst eto rw cer ost eaux ie eae Delotta ohn 21 Gost D ta eresi isere ead EE EEE EEEO TEE EOT TES EE EEE IE EEE A OEE O 23 Establishing Cost Structure eniseniaiena pake e E A ENEA 23 An Easier Approach to Structuring Cost Data scssccndncdviaacisannineedotan vivaenad need ireren 23 G stomized Cost Calculations sireisas iia a E E AE AA 23 SS AE NA dec a r e a E E E EE E EGT 25 PerspectiveSsscrse iiaee inns Meaicusdeeiidseided ceendded S Eee AE Ea a EA eed Ena 25 The Basic Concept of Pers
116. ng in Scorecards On scorecards PAT represents the scores of measures or submeasures by colors or by a com bination of colors and numerical values if desired The conventions used for the color coding are of two types one for the Thresholds Weak Thresholds and Weakest Link methods and one for the Rankings method Colors for Thresholds Weak Thresholds and Weakest Link Methods For all the non Rankings methods used by PAT the scores of the investment options can be mapped into the colors of familiar stoplight charts where red is worst and orange yellow light green and green are successively better A five color system is used for our measure summary table because over the decades five has proven to be a comfortable number that makes sufh cient distinctions but avoids cognitive overload The mapping for the measure scores is shown in Table 5 3 In addition for the Thresholds and Weakest Link methods if any submeasure fails to reach a threshold the cell in the measure summary may optionally have an F shown in the upper right hand corner The convention in PAT is that the score leans upward at boundar ies so that a score of 0 800000 is dark green whereas a score of 0 799999 is light green Colors for Rankings Method Color coding on the measure summary table for the Rankings method is different and even uses different colors to avoid conveying the impression of good and bad associated with the stoplight charts for the Goals and Th
117. nk 1 4 2 0 8 f J ost 3 A 5 B lt a 2 l 20 6 4 0 6 o i E z f 204 t c 049 c i o v Z 2 0 2 4 0 25 v hr ay i 0 a 1 0 0 1 000 2 000 3 000 Total costs 2010 2015 M Changing the evaluation metric or any of the investment options will automatically regenerate the Scatter Plot as will clicking on the Generate Scatterplot button which should be used when changes are made in the data for the various options given in the Levels 2 and 3 databases The scatter plot will not update for such changes automatically in part because it is convenient to have the chart remain constant while one looks at other sheets Another reason for constancy is that when Excel regenerates charts it restores default formatting which may undo a good deal of work If the user does want automatic regeneration however he needs only to click the appropriate box Spider Charts Sheet The Spider Charts gt sheet Figure 4 22 allows the user to select a measure and up to four investment options The goal and threshold values for the Level 2 measures may be inputted instead of two of the investment options The Level 2 measures of the selected Level 3 mea sure are shown as arms of the spider chart The values in the spider chart are scaled relative to the investment option or goal or threshold value selected as Chart 1 Also PAT inverts the scales if necessary so that in
118. nking permissions please see RAND Permissions This product is part of the RAND Corporation technical report series Reports may include research findings on a specific topic that is limited in scope present discus sions of the methodology employed in research provide literature reviews survey instruments modeling exercises guidelines for practitioners and research profes sionals and supporting documentation or deliver preliminary findings All RAND reports undergo rigorous peer review to ensure that they meet high standards for re search quality and objectivity TECHNICAL REPORT RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Paul K Davis e Paul Dreyer Prepared for the Office of the Secretary of Defense Approved for public release distribution unlimited ra NATIONAL DEFENSE RESEARCH INSTITUTE The research described in this report was prepared for the Office of the Secretary of Defense OSD The research was conducted in the RAND National Defense Research Institute a federally funded research and development center sponsored by the OSD the Joint Staff the Unified Combatant Commands the Department of the Navy the Marine Corps the defense agencies and the defense Intelligence Community under Contract W74V8H 06 C 0002 Library of Congress Cataloging in Publication Data Davis Paul K 1943 RAND s portfolio analysis tool PAT theory methods and reference manual Paul K Davis
119. nt to use Level 2 and Level 3 data rather than the MRM options in which you enter data only at Level 2 or Level 1 Go to the Level 3 Data sheet via the tab or the Go To Sheet menu See Chapter Three for a filled out example Enter weights for the different Level 3 measures in the fourth row These might be e g 4 and 2 in which case PAT will normalize them as necessary Instead you could have entered 2 3 and 1 3 For the example however you can use 1 and 1 Rows five through eight tell PAT how to handle the data Since we want high values we might set the values at 0 and 10 for rows five and six meaning that the data entered will be between 0 and 10 inclusive However you will then want 0 and 1 in rows nine and ten so that PAT will rescale to that range If you had specified that good was low you would reverse the values in rows five and six Had you chosen something other than Thresholds the proce dure might be slightly different Now enter data in the 0 to 10 range for your options When you are done click on Modify Level 2 Data top left Select Yes about updating You will now be taken to the Level 2 Data sheet where you should fill in data in the same way some data in italics will already be filled in the result of the Level 3 calculation Do not type in these cells obviously or you will be overwriting the calculations You might want to Quickstart on Using PAT 87 do that for some reason but remember t
120. o B Level 2 Measure Worst Case Outcome Risk Scenario B 3 Scenario B 1 Scenario B 2 Level 3 Measure Baselne Option 1 Option 2 Option 3 Option 4 Option 5 92 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Figure A 7 Scatter Plot of Effectiveness Versus Cost for Exercise Problem 1 0 8 Yn ke O lt 0 6 e 7 n f Cc 20 4 8 Pa W 0 2 0 0 50 100 150 200 250 Total costs 2010 2030 M These figures are worth pondering because they make a point about the relative useful ness of different information Figure A 4 is a good condensation of information for the analyst It shows a multiobjective or multicriteria scorecard comparing the options and it also shows cost and aggregate effectiveness In our experience focusing too early on the aggregate effec tiveness is a bad idea Who after all really understands what an effectiveness of 0 5 rather than 0 4 means The scorecards in Figures A 4 through A 6 provide much more information than a single effectiveness number information needed for decisionmaking Effectiveness is useful primarily for summary charts after the decisionmaking has occurred perhaps for telling the story of effectiveness versus cost as shown in Figure A 7 The landscape of effectiveness versus cost is quite understandable but much more informa tive than say the cost effectiveness ratio
121. o show relative weights and whether to change coloring scheme You can select the change in coloring scheme to see what happens repeat four times to get back to where you started e Scoring Method allows you to select a scoring method as discussed in the text Thresh olds is the default method That is equivalent to linear weighted sums if the thresholds and goals are set to 0 and 1 respectively An important alternative is the Weakest Link method i e when a score should be no better than the weakest of its contributors The Current Perspective menu allows you to choose from among various perspectives that you have previously defined or to make new ones A perspective is characterized by its weighting schemes and other parameters There are none at the outset e Sorting Category and Sorting Method allow you to select a column and then sort rows based on some criterion This is discussed in Dreyer and Davis 2005 although some details have changed e Multi Resolution Modeling Level gives you a choice of which of three databases to use The option of entering information at a higher level is part of a multiresolution model ing philosophy Use Level 3 Data actually means enter data at Level 3 and Level 2 Use Quickstart on Using PAT 83 MRM Level 2 Data means enter data only at Level 2 Use MRM Level 1 Data means enter data only at Level 1 essentially specifying answers as you might when experiment ing with story line and displays If
122. of how assumptions on the above matters affect both results and perceptions and to tune assumptions so as to pro vide a set of baseline results that are as robust as possible For example it serves decision makers poorly when color coded conclusions change markedly if some low level assump tions are changed slightly e g moving a goal from 0 89 to 0 9 should not change results dramatically e A consequence of the above admonition is that goals and thresholds need to be seen as heuristics not as absolutes to be accepted mindlessly Finally we note that many of these issues are generic There is a considerable literature dealing with multiattribute measures and objectives that discusses approaches to weighting these measures obtaining utilities from individuals or groups and the use of other aggregation rules gt Next Steps Over time PAT will be improved and enhanced building on the experience of applica tions Suggestions will be appreciated In addition to correcting errors and improving user friendliness we are currently thinking aboutat least the following possibilities for enhancement e Permitting different scoring and aggregation methods to be used for different measures or for calculation of cost effectiveness rather than measure scores 4 For example an option could be made to look better by piling on a number of measures each of which is driven by something accomplished well by the option Similarly an option may a
123. ol for capabilities analysis That became PAT the subject of this report Functionality of PAT PAT is not a model in the usual sense rather it is a cross platform spreadsheet tool built in Microsoft Excel 4 that facilitates planning by presenting information in a way that is useful to senior leaders However using PAT encourages a structured way of thinking that generates a conceptual model for the problem being analyzed Further PAT can use a variety of separate or embedded models as sources of input data PAT is an empty vessel but one with many useful features 1 Summary scorecards PAT generates stoplight charts simple color scorecard summa ries of how options rate on a number of juxtaposed criteria such as measures of capabili ties risks upside potential and costs These criteria may be quantitative or qualitative objective or subjective 2 Drilldown zooming PAT generates its summaries from more detailed consider ations which can be viewed by drilling down to a level that provides assumptions a terse logic and a measure of rigor even for qualitative assessments Two levels of drill down are available 3 Multiresolution modeling MRM and data entry PAT allows the analyst to enter data at a lowest level of detail or at one of two more aggregated levels Entering data at the more aggregated levels reduces the amount of data entry greatly and is consistent with the time honored analytic approach of working top down sta
124. omenon in which as cost increases larger increases are neces sary before there are significant increases in effectiveness Figure 4 20 shows a scatter plot with six options In this case the behavior is classic Effectiveness rises with cost but at a diminishing rate And at any given cost level there are better and poorer performers Figure 4 21 shows the Scatter Plot sheet for the simple example used in this report which has only three options It also shows the controls for the scatter plot For this display we chose to evaluate the options for both the Thresholds and Weakest Link methods In this case the relative goodness of the options is unchanged but that will not always be so Figure 4 20 Illustrative Scatter Plot Effectiveness 0 2 000 4 000 6 000 8 000 Total costs 2010 2015 M NOTE The points are individual options Their names are revealed by mousing over them or by changing the relevant chart option Costs in this figure are given in current then year dollars 46 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Figure 4 21 Scatter Plot for the Simple Problem Using Two Scoring Methods Left vertical axis _JAuto update chart Cost Metric Total costs 2010 2015 M ha Place cursor over points to see option name H Right vertical axis Evaluation Metric 1 Effectiveness Thresholds x Generate Scatterplot Evaluation Metric 2 Effectiveness Weakest Li
125. or a Simple Example First Year of Timeframe 2010 Last Year of Timeframe 2015 Trosark 000s 7 Scoring Method Enter once for each Level 2 Build Sheets Show Hide Example Measure with Level 3 Measures Goals Level 2 Measures enter Thresholds Level 2 measure name Weak High or Low once for each set of Level 3 Thresholds values Investment Investment Investment Options Level 1 Measures Level 3 measures Measure Weakest Link Desired Items Categories Option A Measure 1 Measure 1 1 Thresholds High Hardware R amp D Option B Measure 1 2 Thresholds Low Software Acquisition Option C Measure 2 Measure 2 1 Thresholds High Operations Measure 2 2 Measure 2 2 1 Thresholds High Measure 2 2 2 Thresholds High Measure 3 Measure 3 1 Thresholds High Measure 3 2 Thresholds High 4 The Scoring Method and High or Low Values Desired columns can be filled in straight y forwardly using the following choices capitalization matters Scoring Method Thresholds Weakest Link Weak Thresholds Rankings Goals High or Low Values Desired High Low If the user changes these settings while using PAT by going into the Level 2 Data or Level 3 Data sheet for example Template Builder s data will be rendered obsolete That may be good or bad Template Builder s data could be regarded as a default to which one could return or one might be worried that rerunning Template Builder for other reasons would introduce errors Caution is necessary
126. ore would be 0 6 which is then multiplied by the submeasure weight to get the weighted submeasure score In this case Investment Option A did not fail to reach any of the threshold values so it has the highest effectiveness score even though Investment Option C meets four out of six goals Details of the Methodology 67 Table 5 8 Illustrative Results for the Thresholds Method Weight 0 25 0 25 0 5 0 25 0 25 0 5 High or Low values High High High High High High desired Threshold value 2 2 2 2 2 2 Goal value 7 7 a 7 7 7 Level 2 value for 0 0 0 0 0 0 Threshold Level 2 value for Goal 1 1 1 1 1 1 Option A Weighted score Option B Weighted score Option C Weighted score Weak Thresholds Method For the Weak Thresholds method values that meet or exceed the goal value are highlighted in green those that meet or exceed the threshold value but not the goal value are highlighted in yellow and those that fail to meet the threshold value are highlighted in red Table 5 9 We also show the measure score and effectiveness score and color the cells appropriately We show the effectiveness score assuming the measures have equal weight for both measures in the rightmost column Even though Investment Option C fails for submeasure M reaching the goal value on four of six submeasures pushes its effectiveness above that of the other two op
127. ous measures using weights that you speci fied in the Level 2 data To see those weights toggle the corresponding control in the Options menu You can actually change the weights here as well although their proper place for change is in the Level 2 Data sheet Scatter Plot PAT has many displays one of which is the scatter plot To access it go to the Scatter Plot tab It will probably be blank If it is click on Generate Scatterplot You will now see each option represented by a dot at a point in the scatter plot corresponding to its effectiveness and cost Such cost effectiveness landscapes are good for understanding cost and effectiveness together They are much better than working only with the ratio of effectiveness cost However remem ber that the effectiveness calculation depends on the weights given to the measures and in this simple example on the assumption of linear weighted sums An Exercise To conclude this Quick Start material we provide an exercise for you In this exercise you want to use PAT to compare a baseline option Option 1 with four alternatives 2 3 4 and 5 Suppose you want to assess them in terms of their effectiveness 88 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual in two scenarios A and B and a catch all category of Other in national security work this might include something like Shaping the Environment In characterizing effectiveness you disting
128. pectives iiss sinise neii eienenn aie knis ee nin Roi ia Ee 25 Easier Ways to Create and Store Peispecti veya cc irinin i a a EA EE E E h 27 Eztended Perspectives serrer AG neie he Gad n A cae EE RA 28 Template B ilder o ce nis cua si scnesna n E E E E E T E OEE 28 CHAPTER FOUR PAT Output Worcs iets eco silis osis acs siseracned deena eieedanweddvednueadudenwed ivan EErEE E dvennae veneer 31 Summary OE aren Saanich Gunrtasn sand Sant EEE EEA EOIR EE EE EEOSE 31 Structureof the Summary Sheetcssscsnussos i a a E a aa 31 Structuring Rows and Columns with Template Builder iin cccosasasisasarsssaniedsrestavervenne ees 36 Adding at Deleting a Golminty ccccacsanisnrtsaccaanestmennntsanaen a EA E 36 Adding a Numeric Columnof Level 2 Informations crunorerianr ee e 36 Altering Cost Related sol Wiis srusrasriisoisn aes Aerian atea ARA EON STAN 36 Adding ot Deleting OPNS essorer ironie tine ses ieont a ETET ONE EE TNES 37 Cost Effectiveness sid neccardatanciinannieheundandibddeneemnus rudadaenneadeuehanaiuaaneheaninehuananniees 37 Comments Flags and Warnings sseisisisisisis isteti itirir r rero tE rE tE PE rErEE EEEIEE ririri 38 Measure WY IIS apa let init head Dl e E AAE edhe edge IA EAE AAE ANRA 38 Buttons gate ith ete alate saa esd epee ha pe ete asad eddy eta in hg ite ee aad tented ape een 38 An Illustrative Summary Level Scoreestd oii sx diurausaavdrawnstearakiseNateiscniunesnuinedennewnss 38 Level 2 Wal eget Sheet rets sieisen ainoa dc coe
129. pider chart and a scatter plot Examples of each type of chart will be discussed in later chapters Because a PAT file is an Excel spreadsheet the user can also create and custom ize charts using Excel s standard features With this quick overview let us now turn to a more detailed discussion Chapters Three and Four describe input and output sheets at a reference manual level of detail Chapter Five then provides detail on methodology presenting equations and definitions for the most general cases treated by PAT Chapter Six illustrates marginal analysis Readers who prefer to move directly into doing rather than reading may wish to use the QuickStart example in Appendix A Figure 2 9 Sample Output Displays from PAT Line chart of costs vs time Spider chart for values of Measure 3 s components Measure 3 1 Option A Option B Option C Measure 3 4 Measure 3 2 2015 Measure 3 3 Scatter plot of effectiveness vs cost 0 8 o e o 2 0 6 t c v 2 e get F oo W 0 2 0 f 0 2 000 4 000 6 000 8 000 Total costs 2010 2015 M CHAPTER THREE PAT Input Worksheets This chapter describes each PAT input worksheet and shows either a screen capture or a sche matic of each Names of the worksheets are in italics throughout the report except in headings Template Builder Users will ordinarily start their work with PAT by filling out the info
130. ple the terminal phase midcourse phase and boost phase options should be canceled for the budget cuts of 3 billion 6 billion and 9 billion respectively to maintain the greatest effectiveness for the equal weighting perspective Although the boost phase option costs as much as retaining the midcourse phase and terminal phase options the capability is much lower This is particularly amplified under the peer emphasis perspective the squares when the effectiveness score of the midcourse only and the boost phase only options are nearly equivalent For budget cuts between 3 billion and 9 billion cancellation of the boost phase option has greater effectiveness than cancellation of the less expensive mid course phase option under the peer emphasis perspective Thus depending on the perspective chosen different priorities are placed on the retention of different phases of the defense CHAPTER SEVEN Concluding Observations Purpose and Function of PAT As described from the outset PAT is an empty vessel tool not a model Its purpose is to help frame manipulate analyze and present results of multifaceted information to decisionmak ers particularly those concerned with strategic level planning For defense this would include what are sometimes called force planning mission level capabilities planning and cross capability area planning PAT is designed to work in parallel with appropriate capabilities models and to make use o
131. pliers values A somewhat safer approach would be to build a macro that would be used each time PAT ran and depending on the extended perspective chosen would set the parameter values Template Builder We have mentioned Template Builder repeatedly throughout this chapter Figure 3 8 illustrates it for the very simple example used in most of this report The example is much smaller and less busy than a real application will be The inputs are in red Although it is conceptually simple some rules must be followed to fill it in correctly 1 Filling in the Timeframe top left is simple as is picking the cost units from the menu item 2 The block of items in the first column Investment Options must have the options spelled exactly as they are to appear on displays This list is unrelated to anything else on the sheet That is Option A has nothing to do with Measure 1 even though it is adjacent to it 3 The Level 1 Level 2 and Level 3 measures are entered in the syntax shown in the example which means that their relative positioning matters Thus Measure 1 1 and Measure 1 2 appear in cells adjacent to the cell showing Measure 1 with the first of them being right beside it Similarly for the one example of a Level 3 measure Measure 2 2 1 the name must go immediately to the right of Measure 2 2 One of the most common mistakes in using PAT is getting these alignments wrong PAT Input Worksheets 29 Figure 3 8 Template Builder f
132. ppear worse than it probably should if the measures chosen reflect a pure worst case perspective gt Some of these approaches appear under discussions of multiattribute utility theory Keeney and Raifa 1976 Kirkwood 1997 value focused thinking Keeney 1992 Parnell 2006 and balanced scorecard methods used in business Kaplan and Norton 1996 The original DynaRank documentation also includes some discussion of these approaches Hillestad and Davis 1998 80 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual e Providing the ability to generate the measure of cost used in cost effectiveness as a linear weighted sum of the various costs provided as inputs e Developing a richer and more structured mechanism for exploratory analysis probably building on the alternative perspectives mechanism This could include limited mecha nisms for search e g finding the combinations of key parameters that would cause a particular option to be assessed well or poorly APPENDIX A Quickstart on Using PAT This appendix is written for those who want to learn by doing at least initially It assumes that the reader has a copy of the relevant Excel file for PAT The format is informal The example is the one used in the main text Opening PAT When you open PAT you may be asked whether to enable macros If you are asked the answer is Yes If the computer refuses to open the file for security reasons you will need to
133. pproach to Structuring Cost Data It can be tedious to set up the data structure in the Cost Data sheet so the recommended approach is to specify the structure in Template Builder which will then fill out the sheet except for the costs of the options themselves Simplification is also desirable in many cases as shown below Customized Cost Calculations It will sometimes be desirable to customize cost data or cost calculations in ways not antici pated by PAT s built in options or to juxtapose calculations for different assumptions e g different rates of inflation or discount rate or different horizons short of infinity for calculat ing present values That can be accomplished in a separate worksheet that may draw from the Cost Data sheet Davis et al 2008 Results can be displayed in the Summary sheet using its custom cost columns or in the worksheet itself Figure 3 4 Cost Data Sheet R amp D Cost M Option A Hardware Software R amp D Cost M Option B Hardware Software R amp D Cost M Option C Hardware Software Modify Summary Discount Rate Rel to First Year o Be Acquisition Cost M Option A 2 010 2 011 Hardware 0 0 Software 0 0 Acquisition Cost M Option B 2010 2 011 Hardware 0 0 Software 0 0 Acquisition Cost M Option C 2010 2 011 Hardware 0 0 Software 0 0 Operations Cost M Option A 2 010 2011 Hardware 0 0 Software 0 0 Operations Cost M Option B 2010 2011 Hardware 0 0
134. previous Level 2 information from the Level 2 Drilldown sheet into the MRM Level 2 Data sheet invokes MRM Level 2 data with the Multi Resolution Modeling menu and proceeds When he is done he may revert to the detailed analysis He may find however that some Level 3 to Level 2 calculations were spurious because of hopelessly ambiguous data or uncertain phenomena Dispensing with such spurious and pretentious detail would then be desirable In other cases the detail is essential Using MRM Level 1 data is unusual but can be useful for experimenting with story lines and displays Using it will specify the colors that appear in the Summary scorecard Some of the items in MRM data sheets can be changed in output sheets but the user should then click appropriate buttons to trigger updating It is best to make the changes in the input sheet and click the appropriate Modify button And as mentioned earlier it is arguably dangerous to enter inputs in output sheets because of the potential for confusion PAT Input Worksheets 23 Cost Data Establishing Cost Structure PAT allows the user considerable latitude in representing cost information about various options The choices include e Time frame e g six year 20 year or forever costs e Categories of cost e g R amp D acquisition O amp S e Items the classes of investment items such as hardware and software or ships aircraft and tanks e Discount rate representation of inflat
135. re forced to itemize they may omit some of the possibilities Another theoretical problem is that the errors made in low level high resolution subjective estimates do not propagate upward nicely An aggregate level assessment may be more accurate than the result of calculations based on many low level assessments For brevity we do not show examples of the MRM Level 1 and MRM Level 2 Data sheets here The former is trivial one merely inputs the answers that are to be displayed in the Summary display The latter looks like and operates like the Level 2 Data sheet Hint Two usage scenarios are worth mentioning In the first the analyst is thinking top down and doesn t want to specify details Like a designer or system engineer he leaves markers or stubs by entering the names of Level 2 and Level 3 measures but instead of filling in the detail he creates a separate MRM Level 2 Data sheet and fills that out He does initial work at that level invoking that data sheet in the Summary sheet s Multi Resolution Modeling menu When the time comes he reverts to using the data sheet with both Level 2 and Level 3 data and fills in details as needed In the second scenario the analyst begins bottom up working with Level 3 detail At some point he finds it necessary to be more agile and to think at a higher level perhaps when exploring uncertainty or responding to aggregate level What if questions He copies his
136. relatively aggregate level depiction of capability that has attributes such as comprehensibility and parameterization that permit exploratory analysis under uncertainty Such models differ from e g the high resolution high fidelity simulation models used for training and mission rehearsal In some cases they can be formula models or simple programs suitable for a single analyst to use on his personal computer In other cases e g the more agile and com prehensible campaign models used by DoD for defense planning they are somewhat more complicated Thunder STORM and JICM are capability models of this type JWARS is not nor are models such as Brawler Within the emerging realm of models for irregular warfare the often mentioned models e g PSOM SEAS COMPOEX are much more complicated although they have modules that are analogous to capability models 77 78 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual calculations depend sensitively on the assumptions and priorities that go into them which are precisely what decisionmakers are paid to think about and decide on The decisionmakers are responsible for worrying about say the balance of a portfolio across missions the extent of risk to be taken and the ways in which risk can be managed Therefore they need to reason at the portfolio level not at the level where they are merely comparing cost effectiveness numbers PAT provides a number of wa
137. res 000s Effectiveness Effectiveness Cost Detail Detail Detail D Detail Investment Options 1 1 Baseline Option 1 0 0 1 0 Option 2 10 0 37 1 Option 3 80 0 48 0 16 Option 4 160 0 51 0 09 Option 5 200 0 85 0 12 Perspective A upper dark blue diamonds indicate that Option 2 is significantly better than Option 1 for little cost and that incremental improvements occur with expenditures Options 3 and 4 Another big improvement occurs with the much more expensive Option 5 In contrast in Perspective B the options short of Option 5 have much less value Perhaps Per spective B is based on more stressful missions and more conservative assumptions Upon look ing at such charts one might choose to talk in terms of minimum and stretch goals shown by the horizontal lines The latter would be feasible only if a larger budget were forthcoming Figure S 4 Cost Effectiveness Landscapes for Two Perspectives Effectiveness Perspectives Number 0 50 100 5 Stretch Goal a Minimum Goal 150 200 250 Costs M Summary xix Risk Management Risk management is very important in strategic decisionmaking so we note that PAT can depict various types of risk in a number of ways e g with explicit top level measures with lower level measures relating to more stressful versions of a test case by showing consequences of more and less conservative combining rules or with
138. reset the level of protection e Close and reopen Excel e Go to the top of window menu Tools Macro Security e Set security level to medium You will now be permitted to continue Navigation and Manipulation To begin PAT will open a Summary sheet Figure A 1 with some placeholder data which you will replace Although you will seldom work with the entire Summary sheet at a given time you should be aware that as shown in the figure it has four different vertical blocks of information The leftmost block is the scorecard that characterizes options by different mea sures of effectiveness measures The second block is a set of columns that allow the user to show selected data by option from the second level of detail The third block contains selected output cost information The final rightmost block contains calculations of the options effec tiveness and cost effectiveness when aggregated across all of the measures Ordinarily the user will focus on the scorecard portion of the Summary sheet 1 This instruction is for users of Excel in Microsoft Office 2003 for Windows XP You may not encounter any such warning if you are using Excel 2004 for the Macintosh The procedure for adjusting the security settings is slightly different in Excel 2007 because of changes in the interface 81 82 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Figure A 1 Overview of Summary Sheet Option
139. resholds methods 64 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Table 5 3 Mapping Measure Scores into Colors 0 2 lt score lt 0 4 0 4 lt score lt 0 6 0 6 lt score lt 0 8 As shown in Table 5 4 the colors go from light blue to dark blue denoting the quintile in which the weighted average rank of the investment option lies For example if there were ten investment options an investment option would receive the lightest blue color if the weighted average rank was two or lower and would receive the darkest blue color if it was eight or greater The same coloring method based on quintiles is used on the Drilldown sheets where the rankings in each submeasure set the color for the cell Table 5 5 summarizes the various methods concisely Examples of Scoring and Aggregation Using Different Methods To provide examples for each scoring method we look at two measures M and M each consisting of three submeasures All submeasures take values between 0 and 10 inclusive with 2 being the threshold value and 7 being the goal value in each case For both measures submeasure M will have weights twice those of the other two submeasures all submeasures receive a score of 0 for not reaching the threshold value By setting the weights of the three submeasures to 0 25 0 25 and 0 5 we get a sum of 1 so there is no need to divide the sum of the submeasure scores by the sum of the weights We also assum
140. rily use absolute references with signs so that the formulas will be automatically adjusted if they change PAT s structure by e g adding a column or row Checking is important Users should also avoid or at least be very cautious about using array formulas within PAT if further changes in structure are likely Navigation and Architecture Inputs and Outputs We now provide an overview of how one navigates within PAT the inputs that must be pro vided and the outputs it generates Subsequent chapters will describe inputs and outputs in much more detail Architecture and Navigation Since PAT is implemented in Excel it uses a spreadsheet paradigm for entering data and gen erating familiar kinds of tables and charts In basic terminology a PAT file is an Excel work book that contains multiple worksheets In this report we use the terms worksheet and sheet interchangeably The user navigates among worksheets in three ways 1 by using the Go To Sheet menu item at the top of the program s window 2 by clicking tabs along the bottom of the program s window the standard method in Excel or 3 by clicking buttons that appear in some sheets again a standard method in Excel The first option is often the easiest Figure 2 5 shows where the menu is found on any PAT sheet Figure 2 6 shows tabs at the bottom of the Excel window and Table 2 2 shows the menu of built in worksheets Figure 2 5
141. rmation in a tool called Template Builder Here they will specify the structure of most sheets such as the names of the Level 1 Level 2 and Level 3 measures the names of the options the time period cost catego ries and expenditure items and control variables Template Builder then generates various other sheets appropriately at which time the user specifies the option dependent data and fine tunes some of the controls It is akin to filling out an outline and having the computer generate an entire book with chapter and section structure but no content Despite the centrality of Tem plate Builder for structuring PAT in practice we defer its discussion until later in this chapter because serious users should first understand PAT s underlying architecture This will improve intuition for what can and cannot be done and will help them diagnose problems that may arise because of erroneous data entries Level 1 Data Figure 3 1 shows the Level 1 Data sheet for a simple case with only three measures These data are reflected on the Summary sheet The data to be inputted are 1 the names of the measures to be displayed 2 optional comments and 3 the relative weights of the measures when calculating overall option effectiveness As discussed in Chapter Four if a user mouses over i e passes the cursor over a measure s name in the Summary sheet a pop up will show any comments e g a cryptic definition and weights In this case the
142. rting at a high level then adding enriching detail where warranted The data themselves may be generated by a multiresolution model or family of models 4 Sensitivity analysis and exploratory analysis PAT allows the analyst to quickly recognize key assumptions and to change them interactively This may be done parameter by parameter or more broadly These analyses are greatly facilitated by the MRM feature 5 Alternative aggregation methods PAT allows the analyst to quickly change how summary depictions are generated i e how they are aggregated from details Choices include for example simple linear weighted sums some nonlinear weakest link meth ods linear weighted sums with threshold constraints and rank ordering The analyst can also use customized aggregation rules i e PAT is extensible We have found that to be important in practice 3 A mid 1990s RAND tool the objectives to programs methodology used utility functions and spreadsheet methods to arrive at a higher level of aggregation unpublished work by Manuel Carrillo and Preston Niblack 1996 4 This documentation is intended for Excel 2003 Windows and Excel 2007 Macintosh except for a few differences in Excel 2007 Windows noted in footnotes with which we have had relatively little experience PAT will not work with Excel 2008 Macintosh because it does not support Visual Basic macros However Macintosh users can use PAT readily within a virtual machine
143. ry Multi Resolution Modeling Level x List of Sorting Categories Please Select An Option z use Level 3 Data El Scoring Method Sorting Method Cost Effectiveness Cost Metric Thresholds z Largest Value First 7 Total costs 2010 2015 m ca Current Perspective Default Discount Rate Rel to First Year Color Code 0 8to 1 0 0 6 to 0 8 Level 1 Measure Score 0 4 to 0 6 or Failure calculation Measure Total Cost 2 Measure 2010 2015 Relative Cost Measures Measure 1 Measure 2_ Measure 3 2 1 M Effectiveness Effectiveness Detail Detail Detail Petice Ges i Details Detail Investment Options Option A S a 0 65 1 Option B a ae 0 6 1 09 Option C ooo 0 3 1 39 jenuey aduasajay pue spoyy a AOAYL 1Yd JOOL siskjeuyy O1 0f140d S GNVUY ZE PAT Output Worksheets 33 Figure 4 2 Options Menu of Summary Sheet Update Summary Sheet Show Hide Weights Change Color Scheme Update Menus Generate New Perspective Update Perspective Delete Perspective Add Remove Failure Markers Figure 4 3 Alternative Color Schemes for Scorecards Measures Measure 3 Measures Measure 1_ Measure 2 Measure 3 Detail Detail Detail Detail Detail Detail Detail Detail Detail Detail Detail Detail Investment Options Investment Options Option A Option B Option C WV MW Level 1 Measure Score
144. s Menu Sorting Category Multi Resolution Modeling Level Please Select An Option k List of Sorting Categories Use Level 3 Data Scoring Method Sorting Method Cost Effectiveness Cost Metric Thresholds Largest value First Total costs 2010 2015 m Current Perspective Discount Rate Rel to First Year Default o Measure Total Cost 2 Measure 2010 2015 Relative Cost Measures Measure 1 Measure 2 Measure 3 23 M Effectiveness Effectiveness Detail Detail Detail Detail Detail Detail seme As Investment Options M 10 2561 0 65 1 6 2177 06 1 09 5 852 03 1 39 Color Code Level 1 Measure Score 0 8to1 0 O6to08 04to0 6 0 2to04 0 0to0 2 No data or Failure incl in F summary calculation If you are looking at the Summary sheet on a computer note along the bottom of the window that the Summary tab is highlighted You can switch among sheets by clicking other tabs Since there are many such tabs it is often easier to use the Go To Sheet menu item at the top of your screen Custom sheets will not be listed e g sheets for notes or for specialized cost calculations not done by PAT or a miniature capabilities model that can generate data entries for PAT s primary sheets On the Summary sheet mousing over dragging the cursor over the various menus arrows reveals the following e The Options menu contains miscellaneous controls such as whether t
145. shown in rows and various measures of option good ness in columns Figure S 1 indicates the structure of PAT s Summary sheet schematically On the left is a five color scorecard toward the right are optional columns for specific numerical outputs of interest in a summary and still farther to the right are column groups for different depictions of cost and for overall effectiveness across measures and cost effectiveness At the top left are various control panels that allow the user to change the way underlying calcula tions are made as well as a number of other items At the lower left is a color bar that relates the colors to underlying effectiveness scores in a range from 0 to 1 Drilling Down PAT makes it possible to drill down zoom to understand the basis of high level character izations and to put a spotlight on troublesome problems The mechanism for what we call the drilldown function is shown schematically in Figure 2 which indicates that the results shown for the Measure 2 column come from a lower level Level 2 calculation that considers two subordinate measures M2 1 and M2 2 Similarly the results shown for M2 2 come from an underlying calculation that considers Level 3 measures A user looking at the Summary sheet can click on a column to bring up the Level 2 scorecard or go deeper to see the Level 3 scorecard of interest if such detail is provided Such drilldown can be part of interactive high level presentations if a d
146. sing the format Measure Submeasure Rankings Table Sheet Figure 4 25 shows the Rankings Table sheet in which the investment options are ranked by effectiveness or cost effectiveness for each perspective defined on the Perspectives sheet The scoring method and cost metrics used can be selected from menus In each cell the rank is shown in large bold text and the relative cost effectiveness appears in parentheses under the PAT Output Worksheets 49 Figure 4 24 Selected Details Sheet List of all Category Subc List of all Category Subc Measure 3 Measure 3 1 Scoring Method Thresholds Level 1 Measure Measure 2 Measure 2 Measure 3 Level 2 Measure Measure 2 1 Measure 2 2 Measure 3 1 Weight of Level 2 Measure in Scoring Functions 0 to 1 i 1 1 High or Low Values Desired High High High Threshold Value 0 0 0 Goal Value 10 1 10 Level 2 Measure Score for Threshold Value 0 to 1 Level 2 Measure Score for Goal Value 0 to 1 Option A Option B Option C Figure 4 25 Rankings Table Sheet for Effectiveness or Relative Cost Effectiveness Scoring Method Thresholds Rank on cost effectiveness instead of effectiveness poat erable Total costs 2010 2015 M gt Measure 1 M Auto update chart RANKING Baseline Emphasis Weakest Relative Cost Effectiveness Default 1 1 1 2 1 5 Link 2 1 5 3 3 2 2 Option A 0 33 0 74 0 88 0 88 2 2 3 3 Option B 0 39 0 89 0 87 0 87
147. sure Goals Thresholds Weak Thresholds Weakest Link Thresholds Thresholds High Option B At the top left of the sheet is a button called Modify Level 2 Data Sheet used to recalcu late the values of Level 2 information generated from Level 3 The tiny box with a checkmark should remain checked Some Level 3 data can be changed in output sheets but the user should be sure to click appropriate buttons such as Modify Level 2 Data Sheet so that the changes will take effect Arguably this functionality of changing inputs in an output sheet should be avoided because of the potential for confusion MRM Level 1 and Level 2 Data The MRM Level 1 Data and MRM Level 2 Data sheets allow the user to have a low detail mechanism for inputting data so that fewer changes are necessary when doing sensitivity analysis or more general exploratory analysis This can be quite useful but the user should remember that these data sheets are distinct from the nominal data provided in the Level 2 Data and Level 3 Data sheets The analyst must maintain such consistency as is needed This MRM functionality is valuable to the analyst who wishes to work at a low level of resolution until everything is making sense at which point more detail and more care in specifying data may be appropriate This is standard in good top down analysis and in soft 3 This option affects certain displays if a measure s raw value is below its threshold It is
148. t at level 1 via level 2 data Cost for cost effectiveness F Cost Data Measure 1 1 score A 1 Fo 1 Measure 1 1 s raw value Input at level 2 Measure 1 2 1 s easure 1 2 2 s ra raw value Input at level 3 a 1 1 mapping ni1 mapping aggregation F function NOTE Grayed items are inputs Output Cost After constructing PAT consistent cost data typically in inflation adjusted dollars for his options the analyst may use PAT or custom worksheets within PAT to gener ate cost information in many different forms Some of this activity will require merely manip ulating the core data in different ways but some will involve calculation For example the analyst may wish to generate present value costs for various options If so he must specify a real discount rate or a set of rates to use for bounding the problem The formula used for the present value of a set of payments to be received over 7 years is z E py y ATR 2 A promise to pay someone 1 million in ten years is less painful than paying 1 million now because one could invest the 1 million for ten years pay the debt at that point and keep the returns The economic calculations are well understood as discussed in many sources including Wikipedia Hitch and McKean 1965 but the real discount rate is controversial and dependent on the particular problem considered It is arguably good practice to calculat
149. t effectiveness The different methods in PAT to which we have alluded use different functions as discussed in the next section 4 A more rigorous term is abstraction A measure abstracts from or captures the relevant essence of the submeasures It may be a simple average or it may be a more context sensitive projection Historically aggregation meant the result of collecting e g one might aggregate the strengths of nine battalions to estimate the strength of a division It is coming to have a more general meaning akin to abstraction Zeigler Praenhofer and Kim 2000 Details of the Methodology 55 Summary of Definitions Table 5 1 summarizes the terms used in PAT and their meanings and includes an example of each Alternative Methods The Need for Alternative Methods Each of the steps in Figure 5 1 requires specifying the appropriate mathematical function In a given application of PAT this is accomplished when the analyst chooses a scoring method Table 5 1 A Glossary of PAT Terminology Term Meaning in PAT Examples Abstraction A generalization derived from more Engagement effectiveness as derived from radar detailed or concrete cases perhaps for missile and kill vehicle effectiveness a particular context of use measures Effectiveness of an Army brigade for different are abstractions of submeasures classes of combat relative to a standard brigade Aggregation Abstraction e Ten year cost
150. t option but they should not be so large that one could be decomposed into a collection of smaller meaningful steps Thus a step should not consist of the cancella tion of three unrelated programs rather three new options should be defined by the cancel lation of each individual program In addition a step should be maximal in the sense that any program that is made unnecessary or necessary by the cancellation or addition of one program should also be cut or included along with that program For example cutting the development of a radar platform should also cut the development of any battle management suite associated with that radar platform unless the suite can also be used for other radar sys tems under development Once these steps are defined each investment option consists of the base case and one or more of these steps As the number of steps increases the number of possible investment options increases exponentially It may be best to restrict analysis to options that are at most a fixed number of steps away from the base case With 7 possible steps there are roughly 7 2 options that are at most two steps away from the base case and about 77 6 options that are at most three steps away from the base case The number of investment options under consider ation should also be tempered by the ability to determine the costs associated with each as well as the ability to analyze the BMDS that results from each PAT can store thousan
151. t order of tabs but users may have other tabs corresponding to application specific worksheets and will in any case find it convenient to reorder the tabs by moving them around Thus the order in Table 2 3 may not apply A Notes sheet is optional something we suggest that the user add as a custom sheet for keeping track of changes and subtleties and for maintaining con figuration control i e assuring that collaborators are using the same version of PAT In a spreadsheet program such as PAT architecture is largely indicated by the choice of worksheets and the relationships among them e g Level 2 Drilldown provides more detail on some aspects of the results shown in the Summary sheet Inputs and Outputs With this background let us review schematically PAT s key outputs This will provide a sense of what PAT does before we get into the details Figure 2 7 is the schematic diagram of our topmost display the Summary sheet Figure 2 7 Schematic of Summary Sheet Overview of PAT Table 2 3 Default Ordered Listing of Tabs for PAT Sheets Welcome Screen Summary Level 2 Drilldown Level 3 Drilldown Scatter Plot Spider Charts Cost Charts Rankings Table Level 1 Data Level 2 Data Level 3 Data MRM Level 1 Data MRM Level 2 Data Selected Details Cost Data Perspectives Template Builder Dropdown Measures Measure 1 Measure 2 Measure 3 Investment Options Option A Option B Option C Deta
152. ted In the vast majority of cases in which the authors have encountered errors in PAT over the past year or two the problems originated in data entry and were not problems with PAT itself A few additional problems had to do with misunderstand ings Few involved bugs although bugs undoubtedly remain especially for the least exercised aspects of PAT functionality CHAPTER FOUR PAT Output Worksheets Summary Sheet Structure of the Summary Sheet The Summary sheet is the main output of PAT We showed its schematic form in Figure 2 7 but Figure 4 1 shows a screenshot of the actual Summary sheet Numerous columns have been hidden see gray bars so that the portions could be juxtaposed Ordinarily users work with only a portion of the sheet at a time but it is convenient to have much information on the top level sheet The Summary sheers control panels provide a great deal of flexibility All employ drop down menus the first of which is the Options menu see Figure 4 2 The Options menu is especially important for setting a number of user preferences Some items are self evident some are less so 1 Update Summary Sheet The user should update the displays after changing data This is the command to select when the user is in the Summary some equivalent buttons exist in other sheets Show Hide Weights Having the weights of the various Level 1 measures shown can be toggled on or off The weights are used in the calcu
153. ternatives than the five built in methods Thus we have also allowed for extensibility as discussed later The five built in methods are defined in Table 2 1 using the concepts of thresholds goals and nonlinearity Chapter Five provides more details on all five methods Of the five the first three are the core methods referred to as the Thresholds Weakest Link and Weak Threshold methods The default method Thresholds characterizes a given aggregate measure s score as zero if any of its submeasures scores are below analyst specified thresholds That is a Level 1 measure is zero by this method if any of its Level 2 submeasures are below the threshold simi larly a Level 2 measure is zero by this method if any of its Level 3 submeasures are below the threshold This enforces the concept that a system fails if any of its critical components fail The method is appropriate if the submeasures happen to be individually critical The Weakest 10 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Table 2 1 Core Built In Aggregation Methods Method Thresholds Weakest Link Weak Thresholds Component Measure Scores See Figure 2 3 See Figure 2 3 See Figure 2 3 Aggregate Measure Scores 0 if any raw value does not reach threshold otherwise a weighted sum of component measure scores Minimum of component measure scores Weighted sum of component measure Overall Effectiveness across
154. thout the user having to look at the other weights and do a mental calculation Details of the Methodology 59 Weak Thresholds Method Submeasure Scores with Weak Thresholds Method With the Weak Thresholds method each submeasure has a threshold and a goal If the threshold is not reached the submeasure score is 0 If the submeasure reaches or exceeds its goal the submeasure score is its score for the goal value In between the score is determined by a linear relationship That is in cases where goodness increases with raw value S js 0 i V ja Fi i j k G if V id 2 ae V ar aT toe Ca Ta ee a Via K z j j If goodness decreases with raw value the equations must be adjusted Figure 5 2 shows graphically the resulting scoring method for cases in which more is better solid line and cases for which more is worse dashed lines The order of goal and threshold is reversed for the two cases for the more is worse case dashed lines the threshold comes second and the goal first as one reads along the x axis see the parenthetical values Figure 5 2 Score Versus Raw Value for Goals and Thresholds Methods Score Score l Score at or SERN o at a above goal above goa gt l N N l Lessis N l better l l l l l l N l N l b I N N l Threshold score 4 Threshold score l l l l jRaw value of submeasure Threshold Goal Goal Threshold 60 RAND s Portfolio Analysis Tool PAT Theory
155. tions Weakest Link Method For the Weakest Link values that meet or exceed the goal value are highlighted in green those that meet or exceed the threshold value but not the goal value are highlighted in yellow and those that fail to meet the threshold value are highlighted in red Table 5 10 We also show the unweighted submeasure score the measure score which is the minimum of the submeasure scores for each measure and the effectiveness score which is the minimum of the measure scores for each investment option The weights of the submeasures do not apply here We color the effectiveness score cell for each measure to correspond to the color scheme used in PAT for this method Because Investment Option A is the only option that did not fail on any submea sure it has the highest effectiveness score under the Weakest Link method 68 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Table 5 9 Illustrative Results for the Weak Thresholds Method Weight 0 25 0 25 0 5 0 25 0 25 0 5 High or Low values High High High High High High desired Threshold value 2 2 2 2 2 2 Goal value 7 7 7 7 7 7 Level 2 value for 0 0 0 0 0 0 Threshold Level 2 value for Goal 1 1 1 1 1 1 Option A Weighted score Option B Weighted score Option C Weighted score Table 5 10 Illustrative Results for the Weakest Link Method
156. tions are monotonic non decreasing they are not convex however as can be seen in Figure 5 2 Thus improving performance of an investment option in some submeasure cannot decrease the effectiveness score The aggregations to an overall effectiveness score have the same property More to the point our scoring methods do not lead to counterintuitive conclusions except in obscure cases that have no significance 56 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Most work in mathematical decision analysis uses the method of linear weighted sums con tributing to a single measure of utility That method is taught in many schoolbooks and embedded without comment in much decision support software It is often quite useful but in strategic planning capabilities based planning and in much of systems analysis and policy analysis more generally it is flawed for several reasons e Decisionmakers need to know about some of the apples and oranges separately aggre gation into a single utility suppresses too much information and depends too heavily on underlying assumptions and preferences that are properly in the decisionmaker s province rather than the analyst s e As decisionmakers sometimes observe Who knows what it means for an option to score 0 74 rather than 0 77 Such aggregate indexes often have no intuitive significance beyond more is better e Similarly decisionmakers need to kno
157. tures in the budget categories of research and development R amp D acquisition and operations and support O amp S Invest ment options may differ in what is to be developed and how fast in what will be deployed operationally and so on Or they may differ because of alternative technical approaches or because of alternative strategies As shown in Figure 2 1 other inputs to PAT include capabilities risks and costs for each investment option as well as control parameters which determine the form of the outputs the assumptions and methods used for evaluation and aggregation and so forth They can strongly affect how the various options stack up in summary displays As indicated schematically in Figure 2 2 PAT s outputs include color coded scoreboards which compare options by different objectives or measures Columns A B and C with red indicating poor and green indicating good tabular outputs on overall effectiveness and cost and standard charts such as charts of cost versus time Many more types of output display are available or can be readily constructed we describe those more fully in subsequent chapters Figure 2 1 PAT as a Black Box Investment options Capability assessments Upside potential Risk assessments Scorecards Cost data Graphic displays Control paramaters for Effectiveness comparisons aggregations formats Effectiveness versus cost comparisons 6 RAND s Portfolio Analysis Tool PAT Theory
158. ues on a Drilldown sheet it is necessary to click on the Modify Data Entries button to make those changes propagate through the rest of the tool Similarly unless the Auto Update Chart checkbox is checked on each sheet the Scatter Plot sheet and the Spider Chart sheet do not automatically update to reflect changes that have been made to the data sheets it is necessary to click on the Generate Scatter Plot and Generate Spider Chart buttons on the applicable sheets to see how the changes in the data have affected the outputs PAT allows considerable flexibility in what can be changed without having to regenerate the portfolio view as described above Weights of measures and submeasures threshold and goal values scoring methods and data values can be changed easily The only constraint on the numbers of investment options measures and submeasures etc that can be used is the ability of a spreadsheet to hold all of the input data That should not be constraining in practice The Importance of the Measures and Methods As with all scorecard methods considerable care must be taken in the development of the mea sure submeasure structure for any analysis performed with PAT Similar care should be taken with the selection of the scoring method and the parameters and weights for the scoring func 2 This emphasis in RAND work has been articulated in a number of RAND monographs Davis 1994 Davis 2002a It was also highlighted in a recent Nat
159. uish between results using best estimate assumptions best case assumptions a measure of upside potential and worst case assumptions a measure of risk In evaluating the worst case effectiveness for Scenario B you feel that it is necessary to consider several bad variants of Scenario B which stress capabilities in different ways They are called variants B 1 B 2 and B 3 Assuming that all these measures have been appropriately defined and put on the same scale from 0 to 1 you might have the following data sets where larger numbers are always better for the measures and where you need not worry about subtleties such as thresh olds and ceilings The data in Tables A 2 through A 4 may be based on studies that used organization approved models and data sets for the scenarios Some translation must have been made between outputs of models and the scores shown for the options but we will not concern ourselves about that here Table A 2 Level 2 Data for Illustrative Exercise Option Best Estimate Case Best Case Worst Case Scenario A 1 Base 0 1 0 1 0 1 2 0 5 0 6 0 3 3 0 55 0 7 0 5 4 0 57 0 9 0 5 5 1 1 0 5 Scenario B 1 Bae o OF Calculated 2 0 41 0 55 Calculated 3 0 45 0 7 Calculated 4 0 6 0 82 Calculated 5 0 75 1 Calculated Other Measures Level 2 Score 1 Base 0 1 2 0 5 3 0 5 4 0 5 5 1 0 Quickstart on Using PAT 89 Table A 3 Level 3 Data for Illustrative Exercise Outcomes for Different Definit
160. ulate effectiveness is selected from the Cost Effective ness Scoring Method dropdown menu the cost metric used in the cost effectiveness cal culation is selected from the Cost Effectiveness Cost Metric dropdown menu To make present value calculations the user can apply a discount rate to the cost numbers using the Discount Rate dropdown menu This should be understood as the real discount rate if the costs in the Cost Data sheet are already corrected for inflation and as the sum of inflation and real inflation rate otherwise The cost effectiveness values are scaled so that the most cost effective investment option has a value of 1 and all other investment options are compared to it A no changes baseline if present will always have a cost effectiveness of 0 rather than allowing it to be infinity because it involves zero cost Figure 4 10 Cost Information in the Summary Sheet R amp D Cost Acquisition 2010 2015 Cost 2010 Operations Total Cost 2010 Cost 2010 2015 M 2011 M Cost Cost Detail Detail Cost Detail Detail 3 This menu sometimes has duplicate versions of the same costs with slight differences in names 38 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Comments Flags and Warnings The cell showing the name of an investment option may have a comment indicated by a red triangle in the upper right corner Mousing over that c
161. ur then describe PAT s input and outputs in user manual detail Chapter Five discusses selected theory and methods in more detail especially aggregation methods and methods for marginal and chunky marginal analysis Chapter Six wraps up with suggestions and cautions for users and with thoughts about future work Appendix A is a QuickStart exercise for those who like to learn by doing Appendixes B and C describe some practical hints for those who are actually using PAT gt An embedded model might be implemented in a particular worksheet of the PAT workbook A connected model might import data from a program written in a different language such as Analytica a product of Lumina Decision Systems Inc www lumina com We used such a connection approach in our work for MDA Willis et al 2006 A capability model built in Analytica CAM MD could be exported to PAT MD CHAPTER TWO Overview of PAT Inputs and Outputs PAT takes a series of inputs and generates outputs in the form of portfolio style tables and vari ous charts and graphics Figure 2 1 That is viewed as a black box it primarily generates displays to describe implications of input information in a structured way Many of the inputs such as the investment options to be compared are what one might expect A given investment option specifies expenditures in each budget category for each year covered by the analysis This could include e g separate expendi
162. urces including subjective judgment or detailed studies PAT Input Worksheets 19 Parameters Specifying the Nature of Scoring and Aggregation The rows at the top of Figure 3 2 above the yellow divider specify the control parameters needed to aggregate from Level 2 to Level 1 How and whether a given control parameter is actually used depends on the scoring and aggregation methods chosen which will be discussed later These parameter values can be left blank for Warning columns PAT ignores any values that are entered Figure 3 2 Level 2 Data Sheet Hodiy i Summary i i i i i Weig SUE on of Level 2 Measure in Scoring Functions Oto 1 1 00 1 00 0 00 1 00 1 00 0 00 1 00 1 00 Low High High High High 0 00 10 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 Go 10 00 0 00 0 00 1 00 1 00 0 00 10 00 10 00 ooo Level 2 Measure Score for Threshold Value 0 to 1 0 00 0 00 0 00 0 00 0 00 0 00 0 00 0 00 o oo 0 Level 2 Measure Score for Goal Value 0 to 1 OptionA a 10 00 2 00 onlyl 3 1 00 0 80 1 00 2 00 8 00 400 0 60 0 50 400 7 00 5 00 8 00 0 50 0 00 3 00 3 00 ome believe that M1 1 s value could be The scoring and aggregation functions depend on parameters such as goal values Four rows in the Level 2 Data sheet set those parameters e Weight of Level 2 Measure in Scoring Functions 0 to 1 The relative weights of the
163. ure is calculated from Level 3 data any Level 2 measures inputted directly should also be given raw values from 0 to 1 PAT reports the calculation from Level 3 using the 0 to 1 range and visual explanations such as the Level 2 Drilldown sheet become non intuitive if the various Level 2 measures are on different scales Hint Entering data with decimal points can be more tedious and prone to error than entering whole integer numbers The user can enter the option specific data with whole integers and then scale them down by 1 entering 0 1 in a cell outside the array 2 cutting 3 selecting the range of cells to be scaled and 4 selecting Edit Paste Special Multiply from the Excel menu If that menu option does not appear use Tools Add Ins to activate some optional features of Excel Be sure not to leave any stray items in the sheet e g the 0 1 Level 3 Data Figure 3 3 shows the Level 3 Data sheet for our simple example in which only Measure 2 has Level 3 data Measures 2 2 1 and 2 2 2 The sheet is very similar to that for Level 2 Although small in this example it is often quite large in applications it may have tens of columns PAT Input Worksheets 21 Figure 3 3 Level 3 Data Sheet Failure scores zero not 1 Modify Level 2 Data Sheet Level 1 Measure Measure 2 Level 2 Measure Measure 2 2 Level 3 Measure Measure 2 2 1 iMeasure 2 2 Scoring Method Enter once for each Level 2 Mea
164. ust wants to start a number of things fresh The recommended procedure for copying into a new copy of PAT a copy of Clean PAT is as follows Assuming that the old Template Builder is correct copy and paste data from it first That is copy and paste the rows for the investment options and their measures rows 3 Then enter or manually set the few remaining items such as the range of years intended and the cost units from the related menu Alternatively you can copy and paste the entire sheet Run Template Builder in the new sheet i e click on Build Sheets When prompted say No to the retention of prior data Copy and paste data carefully from the other data sheets notably some or all of Perspec tives Cost Data Selected Details MRM Level 1 Data Level 2 Data and Level 3 Data Be sure to paste into precisely the correct cells Go to Summary and choose Update Summary from the Options menu Check the new Summary sheet If you run into trouble it may be easier to start over than to find the errors and move blocks of data around until everything works The usual prob lems result from pasting into the wrong location In other instances a user may want to copy data from one workbook to another after having set up a new structure in Template Builder The procedure for setting up a new workbook with a partially new structure carrying over as much data as possible is as follows Set up the new Template Builder
165. vestment option must have an associated investment stream specified by projected cost for each year cost category e g R amp D acquisition and O amp S and expenditure item e g ships aircraft of interest for the analysis Such data can be rather complex but can be simplified to e g by using a single category Total a single type of item General or a short time period The required cost streams are inputs expressed in current i e then year dollars or dollars corrected for inflation The initial data available to the PAT analyst will often be a combination of detailed cost streams and fragments of other more aggregated information such as the pro jected 20 year or Future Years Defense Plan FYDP cost of a weapon system or type of force unit The PAT user then must draw on such heterogeneous information to estimate the cost streams required for PAT as best he can He may choose to make simplifying approximations such as spreading aggregate costs equally over the entire time period l As a matter of implementation to specify a Level 1 measure directly in PAT one defines the measure as a function of only one Level 2 measure It is that Level 2 measure that is directly inputted 51 52 RAND s Portfolio Analysis Tool PAT Theory Methods and Reference Manual Figure 5 1 Schematic of PAT s Calculations Relative cost effectiveness versus cost Effectiveness composite score Measure n s score Inpu
166. w how a given option addresses each of their sepa rate high level objectives e Aggregation rules sometimes need to be nonlinear because of system effects as described in the following examples First Example Ballistic Missile Defense In reviewing his program the Director of MDA needs to understand separately the current and projected capabilities for different missions homeland defense defense of allies and defense of U S forces deployed abroad Further he needs to understand how well the defense system would do against both small and large attacks and against attacks with and without various countermeasures There is no single way to roll all such information meaningfully into a single measure Further at a more technical level the effectiveness of a defense system for a particular mission and a particular attack depends in a nonlinear way on the effectiveness of the system s components e g sensors interceptors and kill vehicles If any one of those fails the system fails regardless of how well the other components perform Representing such system issues implies using nonlinearities in the mathematics Davis 2002a Second Example Joint and Combined Forces The Secretary of Defense might wish that overall force structure effectiveness could be reduced to a single number but he needs to see separately measures of the capability of air ground and sea forces for a variety of geographic regions and operational circumstan
167. you want to do a first rough cut analysis start with Level 2 e Cost Effectiveness Cost Metric is self evident referring to the cost that is to be used when evaluating relative cost effectiveness by dividing effectiveness by a cost The options on the menu depend on what you specify for costing options in the Template Builder or the Cost Data sheet both of which are described later e Discount Rate gives you the option of doing present value calculations by selecting the real discount rate assuming that the options costs are already in constant dollars If they are in current dollars you would use a discount rate equal to inflation plus the real dis count rate The illustrative Summary sheet s first column should be read as Investment Options The word Measures pertains to the subsequent columns in row 1 The phrase Level 1 is present tem porarily and is later replaced by the name of the first Level 1 measure Subsequent columns that contain Detail buttons are reserved for potential measures of the options goodness or utility As suggested by Figure A 1 if you scroll to the right you will encounter a block of columns reserved for cost information Which cost columns appear here depends on information entered elsewhere You might have columns for say R amp D acquisi tion and O amp S as well as total costs in a particular period etc Continue scrolling and you will find columns to the right that are reserved for net ef
168. ys civeiinsvavencutinispcrsenvahdnieadtahtasanth e e iae iaaea 77 Seeking Flexibility Adaptiveness and Robustness a5 iasacacndxnacgusletunancisloexnnause Guararadentnse tun 77 PAT as Sat Wy eet ee cht cee e E E E A cot E 78 The Importance of the Measures and Method s y enscsiecsiaesi cs devnacivoensdesnienaccsnans pice devnes 78 Next StepSiris riori nor iE EE O EEA EEE TAOTE EIEE E T 79 APPENDIX A Quickstart on Using PAT ics iniccacsniensivdvaniasivetsverteetnwensne ts rentiedsvsieuetavensietennsnetaves 81 B Transferring Data from an Earlier Workbook rererere 95 C Editing and Neatening jgsc543 c403bacrasiep seca ianei e aiei e E secon 97 References EE E E OA E 99 Figures 5 1 S 2 S 3 S 4 2 1 22s 2 3 2 4 203 2 6 2 7 2 8 29 3 1 3 2 3 3 3 4 3 5 3 6 3 7 3 8 4 1 4 2 4 3 4 4 4 5 4 6 4 7 4 8 49 4 10 4 11 4 12 4 13 4 14 4 15 4 16 4 17 Schematic View of PAT Summary Sheet irerssisrreriosimri a T a tunes xV Drilling Down for Esp Ariat sos au snsrvatoveanster anise ronio noaa outa ts xvi Illustrative Composite Summary Sheet iscisxivisccnctsaninntansknantaiiansxesanns xviii Cost Effectiveness Landscapes for Two Perspectives 21 cianctiadteneranecnnexgoadaene ness xviii DT sash Blick Boxviner T a EE TE TE OE 5 Iu stratiye Output Types iroso ingia sinaia a A A A a tats 6 Four Leyels of Detail penie e EEEE RE EEEE AAR 8 Mapping of Raw Values into Sqonesj ij255 lt ipncrbneinnerehed
169. ys to assess alternatives and some useful albeit limited mechanisms for exploring the consequences of alternative assumptions and priorities but that is very different from optimization The purpose is to find strategies that enjoy FARness flexibility adaptiveness and robustness within plausible budgets PAT as Software PAT is not industrial strength software rather it is a tool for relatively sophisticated analysts who worry more about functionality than about cosmetics It has been tested in a number of applications but it undoubtedly has residual problems Users are encouraged to contact PAT s developer Paul Dreyer at dreyer rand org if they discover mistakes or have technical ques tions about PAT PAT checks for most mistakes that users make for example entering an investment option or measure in the Summary table that does not exist on a data sheet However PAT has not been exhaustively tested or gorilla proofed nor has it been refined in the manner of commercial software Neither the authors nor RAND offers any guarantees or warranties on its use We encourage users to keep a clean copy of the PAT template available in the unlikely event that something occurs to make the software unusable The proper use of PAT also requires discipline We have attempted to simplify much of the initial setup of a portfolio view in the Template Builder sheet but other operations require consistency For example if one edits the val

RAND`s Portfolio Analysis Tool

Contents

Download Pdf Manuals

Related Search

Related Contents