92Mar_Krotow - Calhoun: The NPS
Contents
1. Calhoun Institutional Archive of the Naval Postgraduate School Calhoun The NPS Institutional Archive DSpace Repository Theses and Dissertations 1 Thesis and Dissertation Collection all items 1992 03 The impact of cognitive feedback on the performance of intelligence analysts Krotow Geraldine S Monterey California Naval Postgraduate School http hdl handle net 10945 30606 Downloaded from NPS Archive Calhoun goals of open government and government transparency All information contained A KN OX herein has been approved for release by the NPS Public Affairs Officer LIBRARY Dudley Knox Library Naval Postgraduate School 411 Dyer Road 1 University Circle Monterey California USA 93943 atthe DUDLEY Calhoun is a project of the Dudley Knox Library at NPS furthering the precepts and http www nps edu library NAVAL POSTGRADUATE SCHOOL Monterey California THESIS THE IMPACT OF COGNITIVE FEEDBACK ON THE PERFORMANCE OF INTELLIGENCE ANALYSTS by Geraldine S Krotow March 1992 Thesis Advisor Kishore Sengupta A Approved for public release distribution is unlimited Zp Thesis K86767 Unclassified SECURITY CLASSIFICATION OF THIS PAGE REPORT DOCUMENTATION PAGE Re on la ipa pega eed CLASSIFICATION 1b RESTRICTIVE MARKINGS a oo ee AN tl 2a a Co CASON ATOR CLASSIFICATION AUTHORITY 3 DISTRIBUTION AVAILABILITY OF REPORT Fea ea abc lc gS e for public release distribution2. Research on decision making has revealed the role of feedback in affecting the decisions of individuals Hogarth 1987 Feedback affects the choice that a decision maker will make and can be a crucial element in the decision making process as it affects both the task environment and the decision maker s perception of the environment Hogarth 1987 Feedback can be categorized into several different types including cognitive and outcome feedback As described by Jacoby et al 1984 outcome feedback is information that describes the accuracy or correctness of the response cognitive feedback represents information regarding the how and why that underlies this accuracy A type of feedback which has shown beneficial results is cognitive feedback Balzer et al 1989 Cognitive feedback may have particular utility in a complex probabilistic environment where pertinent information provided to the decision maker in a timely fashion can lead to improved decisions An example of such an environment is military intelligence analysis The examination of the effects of cognitive feedback on intelligence analysts may lend validation to current beliefs about cognitive feedback and raise additional questions which when answered could provide a basis for future intelligence decision support systems B EXPERIMENTAL TASK To best capture the effects of cognitive feedback on intelligence analysis a task specifically designed to
3. Robert V Christine A Montgomery and John R Thompson Human Processes in Intelligence Analysis Phase I Overview U S Army Research Institute for the Behavioral and Social Sciences Alexandria VA Operating Systems Inc 1979 Lehner Paul E James R McIntyre Leonard Adelman Kermit Gates Peter Luster Matthew Probus and Michael L McDonnell Combining Decision Analysis and Artificial Intelligence Techniques An Intelligent Aid for Estimating Enemy Courses of Action U S Army Research Institute Alexandria VA PAR Technology Corporation 1985 Lewis Leslie and Melinda Copeland Human Performance Requirements in C3I Sustems and their Implications in System Design Annual Conference of the Military Testing Association 24th San Antonio TX Air Force Human Resources Lab 1982 Libby Rober Accounting and Human Information Processing Theory and Applications Englewood Cliffs NJ Prentice Hall Inc 1981 Luckie Peter T Dennis E Smith and Grace H Wright Investigation of a Bayesian Approach to a Specific Intelligence Problem Science Park PA H R B Singer Inc 1968 Patten Samuel M An Inductive Taxonomy of Combat Intelligence Data Syracuse University Research Corporation U S Army Research Institute for the Behavioral and Social Sciences 1974 Probus Matthew A and Michael L Donnel DINAA The Dynamic Intelligence Assessment Aid Alexandria VA U S Army Research Institute for the Behavioral
4. The four variables last contact ship type speed weather and Cuban Naval activity proximity to the suspect vessel were all presented in the message set The analyst was presented with forty message sets in total each formatted 26 identical to the sample message set in the Appendix but with the values of the cues or variables randomly varied 2 Geographic Representation The task was designed to best represent the tasks performed by analysts on a daily basis in the Fleet The task asked the analyst for his her best estimate of the suspect vessel s position based upon the information presented in the respective message set In a real situation the analyst would be asked to provide an estimated position in degrees latitude and longitude To provide a more straightforward representation of the task and also to place necessary limits on the pertinent geographic area of the task a grid square was designed to cover the geographic area of the experiment The analyst was asked to locate the suspect vessel in an area in the Caribbean north of Colombia and Venezuela The seogtaphba area of the task is depicted in the Appendix The grid square was bounded by the following four coordinates 12N 72W 18N 72W 12N 78W 18N 78W The grid square was divided into thirty six smaller squares and each square was approximately sixty by sixty nautical miles Each square was assigned a two digit unique number This allowed for greater ease in provid
5. the analysts found the task to be difficult and felt that it moderately resembled a task they had previously performed in the Fleet Subjects found the task to be clear and took the experiment very seriously Analysts who received cognitive feedback found it to be moderately helpful and all subjects felt that the outcome feedback actual ship s position was moderately helpful 39 V CONCLUSIONS A SUMMARY The purpose of this study was to evaluate the efficacy of cognitive feedback in improving intelligence analysts performance The statistical results of the experiment indicate that analysts who received cognitive feedback did indeed perform better than their colleagues who received only outcome feedback B CONTRIBUTION Evidence thus indicatis that cognitive feedback positively impacts performance The significance of this study is that it involved U S Naval Intelligence and Cryptologic personnel and indicates that cognitive feedback could be beneficial to more types of intelligence analysis For example intelligence analysts tracking a suspect vessel would be able to receive information on the significance of the cues in the particular environment in which they are working environmental weights Analysts could have at a glance a summary of the emphasis that they have been placing on various cues in a particular problem decision weights Analysts could also access information relating how consistent their choic
6. 9 does not resembles resemble at all very closely 4 Have you performed a similar task in the past Y N a If YES how long ago in years months 5 Now that you have completed the task can you think of any other factor other variables etc that may have influenced you in making your decision 6 How clear were the instructions regarding the task 1 2 3 4 5 6 7 8 9 Not clear Very at all clear 2 7 How would you present the task differently l 58 ak 8 How seriously did you take this task generally 0 1 2 3 4 5 6 7 8 9 10 not very seriously seriously 9 To what extent was information about the ship s actual position helpful in improving your own decision 1 2 3 4 5 6 7 8 9 Not at all Very helpful helpful 10 How easy was this system to use 1 2 3 4 5 6 7 8 9 Not at all Very easy easy 59 MAP OF CARIBBEAN National Geographic 1988 ty Brownsvi site ULF O Fi Isla Isabela Puerto Villamil S erizo Puerto Bag a ISLANDS ARCHIPI LAGO DE COL N Ecuador i LIST OF REFERENCES Adelman Leonard and Michael L Donnell of Decisions and Designs Inc and Ruth H Phelps of Army Research Institute Intelligence Preparation of the Battlefield Critique and Recommendations U S Army Research Institute for the Behavioral and Social Sciences 1984 Balzer William K Michael E Doherty and Raymond M O Connor 1989 Effects of Cognitive Feedback on Performance
7. Psychological Bulletin v 106 pp 41 433 1989 Barclay Scott and L Scott Randall Interactive Decision Analysis Aids for Intelligence Analysts Griffis Air Force Base NY Decisions and Designs Inc 1975 Brehmer B The Role of Judgement in Small Group Conflict and Decision Making Progress and Applied Social Psychology v 2 1984 Cohen Marvin Kathryn B Laskey and Martin A Tolcott A Personalized and Prescriptive Decision Aid for Choice From a Database of Options Technical Report Reston VA Decision Science Consortium 1987 Dudycha Linda Weathers and James C Naylor 1966 Characteristics of the Juman Inference Process in Complex Choice Behavior Situations Organizational Behavior and Human Performance v 1 pp 110 128 1966 Garrett Wilburn E editor The World Map National Geographic Magazine v 174 no 6 p 910A 1988 Hogarth Robin Judgement and Choice Chichester New York Brisbane Toronto John Wiley and Sons 1987 Jacoby Jacob David Mazursky Tracy Troutman and Alfred Kuss 1984 When Feedback is Ignored Disutility of Outcome Feedback Journal of Applied Psychology v 69 pp 531 545 1984 Katter Robert V Christine A Montgomery and John R Thompson Cognitive Processes in Intelligence Analysis A Descriptive Model and Review of the Literature U S Army Research Institute for the Behavioral and Social Sciences Alexandria VA Operating Systems Inc 1979 61 Katter
8. School Monterey CA 1991 Wargame Prepared for U S Naval Postgraduate School Improved Support to C3I Requirements Decisions J and J Enterprises Inc Poway CA 1991 64 INITIAL DISTRIBUTION LIST No Defense Technical Information Center Cameron Station Alexandria Virginia 22304 6145 Library Code 52 Naval Postgraduate School Monterey California 93943 5000 Administrative Sciences Department Naval Postgraduate School Attn Prof Tung Bui Code AS Bd Monterey California 93942 5000 Administrative Sciences Department Naval Postgraduate School Attn Prof Kishore Sengupta Code AS Se Monterey California 93942 5000 i Joint C3 and Space Systems Department Naval Postgraduate School Attn Prof Carl R Jones Monterey California 93942 5000 LT Geraldine S Krotow 27600 Dobbel Avenue Hayward California 94542 65 Copies
9. The subjects receiving cognitive feedback were presented with a menu after they pressed END The nand was displayed in the top half of the screen After choosing a cognitive or outcome feedback option the subject participating in the cognitive feedback experiment was provided with decision rule scales in the top half of the screen or consistency and or accuracy information in the columns preceding or following the estimated position entry column An experimenter was present for the entire length of the experiment to provide guidance to the subjects and ensure proper task completion by the subjects The subjects were given no time limit for completing the forty tasks involved in the experiment Subjects took anywhere from 50 minutes to 135 minutes to complete the experiment G DESIGN OF FEEDBACK 1 Outcome Feedback In the experiment version that incorporated outcome feedback only the computer instruction set indicated that the analyst could access outcome feedback in the form of the suspect ship s actual position by pressing the END key on the keyboard Once the subject had accessed outcome feedback he she was unable to change any positions he she had already entered In the experiment incorporating both outcome and cognitive feedback the analyst was able to access their accuracy or outcome feedback by selecting the accuracy menu choice on the feedback menu Outcome feedback was presented in similar fashion to
10. and Social Sciences 1986 SAS STAT Guide for Personal Computers Version 6 Cary NC SAS 1987 Schaefer Robert L and Richard B Anderson The Student Edition of Minitab Reading MA Addison Wesley Publishing Company Inc and Benjamin Cummings Publishing Company Inc 1989 Sengupta Kishore The Impact of Cognitive Feedback on Group Decision Making Case Western Reserve University 1990 62 Sticha Paul J John F Patterson and Jonathan J Weiss Approximate Reasoning Methods for Decision Aids Technical Report Griffis Air Force Base NY Decisions and Designs Inc 1982 Thompson J R R Hopf Weichel and R E Geiselman The Cognitive Bases of Intelligence Analysis U S Army Research Institute for the Behavioral and Social Sciences Alexandria VA Operating Systems Division of Logicon Inc 1984 Thompson Dave Henry Rueter and William Rainaldi and Joseph Orosy User s Manual for the Prototype Analyst Workstation PAWS Fort Huachuca Field Systems Research Laboratory U S Army Research Institute for the Behavioral and Social Sciences 1990 Tolcott Martin A F Freeman Marvin and Paul E Lehner 1988 Expert Decision Making in Evolving Situations IEEE Transactions on Sustems Man and Cybernetics v 19 no 3 pp 606 615 1989 63 BIBLIOGRAPHY Goodwin Robert Donald Feedback in Dynamic Decision Making An Experiment in Software Project Management Thesis U S Naval Postgraduate
11. by AI ENCOA DINAA utilized Bayes theorem to adjust and assess probabilities The Prototype Analyst Workstation PAWS developed in 1990 was similar to AI ENCOA in that it allowed the analyst to define alternative courses of action open to the enemy with the aid of MAU D Thompson et al 1990 Cohen Laskey and Tolcott developed a prototype decision aid in 1987 that could be personalized for each user Submarine commanders were used as the subjects and the idea that individual decision makers differ both song themselves and from task to task in the decision strategy they prefer Cohen et al 1987 was examined This personalized and prescriptive decision aid allowed the submarine commander to examine different decision methods weights and trade offs for each decision Tolcott Marvin and Lehner conducted research in 1989 in an attempt to further the investigation of cognitive behavior underlying intelligence analysis Tolcott et al 1989 This study revealed that an intelligence analyst will tend to remain with his her original estimate of a scenario regardless of update information presented Tolcott et al 1989 It appears as though military intelligence analysts will pay more attention to supportive than to contradictory evidence as it pertains to their original decision estimate Tolcott et al 1989 This research elicited another research question if update information could be defin
12. his top military personnel regarding just how much military hardware and personnel would be needed to accomplish this Then the President went to Congress with the Secretaries of Defense and Transportation and presented a proposal to attempt to finally put a noose on the cocaine flow from Colombia Bolivia and Peru The country of chief concern is Colombia since drugs harvested in Peru and Bolivia are transported through Columbia and then flown or shipped to the U S Situation Report 10 November 1991 In a sweeping majority last week Congress voted to increase the budget for the drug war ten fold This increase was voted in under the conception of the American people that the war could now be tackled in a short period of time and put an end to the drug trade in the Caribbean Scenario 46 You are on watch in the Joint Team Against Drugs Command in Florida Pressure has been increasing for your watch team to get the best intelligence possible out to the Fleet surface and air units who are responsible for patrolling marked areas in the Caribbean You have received intelligence indicating that a vessel capable of carrying three tons of cocaine is preparing to depart Colombia for the U S mainland A history of this vessel reveals that it has completed three runs from Colombia to the U S in the last six weeks Its tracks were around the eastern edge of Cuba the first time then around the western edge the last two times Track
13. navigation took 4 days each time You are directed to provide the most likely area the vessel is at the current time Taking into consideration the information provided in the most recent message traffic in what area would you send the aircrews and ships It is currently 11 Nov and you need to estimate the vessel s position Please provide a grid square number where you best estimate the suspect ship to be 47 Decision Support Information What is Decision Support Information Decision Support Information is diagnostic information provided by the system on your decision processes and that of the system You can access the decision support when making your decisions on the suspect vessel s positions In order to compute the information however the system needs at least 10 scores from you By accessing such information decision makers can derive better insight into their decisions processes This enables them to revise and improve their decisions or positions in this case through a what if mode of analysis How do I use Decision Support Information in Making Decisions Typically you the decision maker would use the decision support information as follows I Make some tentative decisions i e ship s positions 2 Ask the system for decision support information 3 Refine your positions accordingly You may wish to do this 1 3 sequence within a block as many times and with as many scores as you wish Th
14. suspect vessel position Each variable appeared to be weighted differently in each scenario presented to the analyst and the combination of the differently weighted variables varied as well This was to provide as realistic a task as possible and to optimally test the analysts decision making capabilities However the actual position of the ship as calculated by the system weighted the ship type speed weather and last contact variables equally and included a random error The proximity of the Cuban Navy variable was negligible in the calculation of the ship s actual position The values used for each of the variables as presented to the analysts are represented in Table 3 24 TABLE 3 EXPERIMENTAL CUE CHARACTERISTICS Last 1 12 hours Contact Ship Sailboat 6 8 knots Type Speed Yawl 10 12 knots Cargo Ship 19 20 knots Speedboat 40 knots Sea State Calm winds up to 10 knots Breezy winds up to 20 knots Heavy swells winds up to 45 knots Hurricane force gale winds up to 80 knots Cuban Navy nautical mi radius nautical mi radius nautical mi radius nautical mi radius E PRESENTATION l Task Description Information All subjects regardless of the type of feedback provided to them received the same basic information to enhance the naturalness of the experiment Background Information and Situation Report provided geopolitical information on the areas of concern for DOD and the United States Gove
15. the analyst and to give the task a realistic perspective The task was reduced to a single situation similar to the analysts daily tasks in the Fleet in the Situation Report and Scenario both which narrowed the scope of the illegal drug trafficking trade to a single suspect vessel The analyst was asked to provide his her best estimate of the location of the suspect vessel based upon the information provided Basic information provided on the suspect vessel included that it had completed three previous runs from Colombia to the U S over the past six weeks tracking east of Cuba the first time and west of Cuba the last two times The analyst was told the vessel was estimated to carry up to three tons of cocaine and that it was preparing to depart Colombia once again for the U S mainland This basic information on the suspect vessel provided the analyst with a brief history of the vessel the general geographic location of the scenario and an idea of the vessel s drug carrying capability 3 Task Variables Cues A realistic task necessitates variables and random weighting of the variables This task incorporated four variables weather ship type speed presence of the Cuban Navy and the last contact information on the suspect vessel All four variables were present throughout the task thus the analyst had information regarding each of the four variables each time he she was asked to perform the task of providing an estimated
16. the environment has assigned to the cues Cognitive feedback consists of three elements Task Information TI Cognitive Information CI and Functional Validity Information FVI Balzer et al 1989 Cognitive Feedback provides what outcome feedback cannot information on the decision making process as it occurs Whereas outcome feedback is information that describes the accuracy or correctness of the response cognitive feedback represents information regarding the how and why that underlies this accuracy Jacoby et al 1984 Naval Intelligence analysis occurs in a very dynamic ON Since it has been suggested that outcome feedback is dysfunctional in a dynamic environment Jacoby et al 1984 and that the type of feedback believed to be most beneficial most often is indeed cognitive Balzer et al 1989 perhaps the feedback with the highest utility for Naval intelligence and cryptologic l analysts could be cognitive ib Cognitive feedback appears to hold promise for improving the quality of intelligence analysis This could be of the utmost importance since many decisions made by analysts can affect the safety and lives of others 17 D HYPOTHESES Because of the importance of decision making in intelligence analysis it is imperative that more research be devoted to the types of decisions that analysts have to make what are optimum feedback mechanisms Thompson et al 1984 This research examined the following hypothe
17. the subjects for both sets of the experiment When the subject accessed accuracy the outcome feedback was displayed in the form of a two digit suspect vessel position as estimated by the system The system s estimate of the suspect vessel s position appeared in the right hand column of the data entry screen next to the suspect vessel s position estimated by the subject The outcome feedback in the format of accuracy is displayed in Figure 3 Outcome feedback or accuracy was calculated with the use of consistency and predictability information 2 Cognitive Feedback All subjects conducting the outcome and cognitive version of the experiment received an additional instruction set Decision Support Information which detailed the five choices of cognitive or decision feedback available to the analyst 32 Vessel Position 2 3 4 5 6 7 8 9 0 1 Figure 3 Outcome Feedback Displayed as Accuracy Sengupta 1990 To access both cognitive and outcome feedback the analyst pressed the END key which then provided them with a menu of information feedback options Choices numbered 1 through 5 on the menu provided cognitive feedback information Choice number 6 provided outcome feedback in the form of accuracy as described in the previous subsection The experimental model required that the analyst had provided EEA for at least 15 message sets before any of the cognitive feedback options could be acce
18. will be described in this section and section III G respectively 28 l Geographic Area A navigational chart identical to those used in the Fleet by analysts conducting actual tasks was provided with the task area delineated in grid format Thirty six squares approximately 60 nautical miles X 60 nautical miles each Were numbered to represent a continuous scale similar to latitude and longitude Each square was unique The grid square numbering system allowed for greater ease in statistical analysis without detracting from Sie actual continuous latitude longitude system employed in the Fleet The geographic area of the experiment defined by the grid square was covered with acetate to allow marking of the area by the analyst with a water soluble Marked Rulers or navigational plotters dividers were provided to the subject for distance calculation Each subject was provided with a piece of scrap paper and a pencil or pen for any time distance calculations oe Calculators were not permitted as a general rule 2 System The exp riment was conducted on IBM compatible 286 386 computers in private office or computer laboratory areas Each subject was given a one page Computer Instruction Set which explained the screen on which the vessel s estimated position in the form of a grid square number would be entered Instructions on how to receive outcome or cognitive feedback were included in the appropriate instr
19. E OF HOW TO USE THE PROGRAM ASK THE EXPERIMENTER 45 BLOCK ONE Background Information The United States has been fighting the drug war for over three years yet the effectiveness of the war is questionable The Presidential election is only one year away and the current administration wishes to ensure continual funding for the anti drug effort World politics have taken a backseat to domestic issues The current administration realizes that domestic policy has not been its strong suit in the press as presented to the American people One issue that is repeatedly spotlighted by the media is that of the crack babies being born to cocaine addicted mothers at an alarming rate Crack babies are not just born in the inner city ghetto atmosphere but to mothers of all social strata in the U S Recent news commentaries have revealed that the importation of cocaine and erack into the U S is at an all time high Congress is beginning to ask what is going on with the drug war Senior military officials know that their anti drug campaign has been relatively successful and fruitful In light of the small number of U S armed forces dedicated to the anti drug campaign the total amount of captured contra band is phenomenal But in comparison to the total amount of cocaine and crack smuggled into the U S annually it is barely the tip of the iceberg The President saw a need to step up the intensity of the drug war He elicited advice from
20. T SUBJ WX REPORT FOR 11 NOV 1991 PARTLY CLOUDY SKIES WITH WINDS FROM THE NNW AT UP TO 45 KTS CAUSING HEAVY SWELLS AND WHITECAPS FM USS SCUTTLEBUTT TO JTAD USCOMSOLANT SUBJ CUBAN NAVAL EXERCISE ACTIVITY CUBAN NAVAL EXERCISE UNITS SIGHTED WITHIN 20 NM OF SUSPECT VESSEL 51 QUESTIONS TO BE ANSWERED AFTER COMPLETING BLOCK ONE cognitive feedback l Describe in words or equations what decision rule you followed in making your own estimates 2 Distribute 100 points among the three variable you used for reaching your overall estimate in accordance with the importance you assigned them total will add up to 100 Ship s speed Ship s last contact Weather Cuban Naval Presence 100 Total 3 What do you think the actual weights the system used for each of the variables was Ship s speed Ship s last contact Weather Cuban Naval Presence 100 Total 4 In this task did you request decision support information at any time from the system Y N 5 If YES try to describe how you used decision support information in making your decisions QUESTIONS TO BE ANSWERED AFTER COMPLETING BLOCK ONE outcome feedback l Describe in words or equations what decision rule you followed in making your own estimates 2 Distribute 100 points among the three variable you used for reaching your overall estimate in accordance with the importance you assigned them total will add up to 100 Ship s speed Ship s last co
21. YES how long ago in years months 5 To what extent were the concepts of decision support information clear to you as explained by the instructions and the experimenter 1 2 3 4 5 6 7 8 9 Not clear Very at all clear 6 To what extent was decision support information helpful in improving your own decision 1 2 3 4 5 6 7 8 9 Not at all Very helpful helpful 7 To what extent was information about the ship s actual position helpful in improving your own decision 1 2 3 4 5 6 7 8 9 Not at all Very helpful helpful 56 e 7 8 Now that you have completed the task can you think of any other factor other variables etc that may have influenced you in making your decision 9 How clear were the instructions regarding the task l 2 3 4 5 6 7 8 9 Not clear Very at all clear 10 How would you present the task differently 11 How seriously did you take this task generally 1 2 3 4 5 6 7 8 9 not very seriously seriously 12 How easy was this system to use 1 2 3 4 5 6 7 8 9 Not at all Very easy easy 57 TASK DEBRIEF SURVEY Outcome Feedback 1 How would you rank the difficulty of your task during the past 40 scenario snapshots 1 2 3 4 5 6 7 8 9 gt not very at all difficult difficult 2 How would you rank the clarity of your task 1 2 3 4 5 6 7 8 9 not very at all clear clear 3 How closely does the task resemble what you do ona day to day basis in the fleet 1 2 3 4 5 6 7 8
22. ain information relevant to a specific problem without becoming bogged down in irrelevant information Patten 1974 Patten s study focused on the information and data relevant to the analysis process but not the analysis process itself Sticha Patterson and Weiss researched and determined Approximate Reasoning Methods for Decision Aids in 1982 which led to the development of a prototype decision aid for Air Force target nomination The reason for studying approximate reasoning methods was to hopefully achieve a system that could facilitate an intelligence analyst s efforts Sticha et al 1982 Hence the actual intelligence analysis process became the focus of the research An Intelligence Aid for Estimating Enemy Courses of Action AI ENCOA was developed in 1985 in an attempt to provide cognitive support to intelligence analysts by altering decision rule bases Lehner et al 1985 AI ENCOA software was developed as a prototype decision aid to assist Army tactical intelligence analysts assigned with choosing feasible enemy courses of action This prototype utilized Multi Attribute Utility MAU models and enabled the analyst to assign weights to each alternative Another decision support software package created for Army intelligence analysts was DINAA The Dynamic Intelligence Assessment Aid by Probus and Donnell in 1986 DINAA allowed the analyst to determine probabilities for each avenue of 12 approach as determined
23. ance ANOVA amongst the performance factors was conducted This was calculated using the General Linear Models procedure in SAS SAS 1987 The results are shown in Table 5 TABLE 5 ANOVA OF MODEL AND ERROR Dependent Degrees sum of Mean F Value Pr gt F Variables of Squares Square Freedom La 1 22 4 720 0 079 0 0001 1 904 0 019 1 22 0 624 0 010 0 0001 0 398 0 004 G 1 22 2 606 0 043 2 16 0 0003 1 968 0 020 A The results indicate that subjects receiving cognitive feedback had a significantly higher achievement score than those receiving outcome feedback F 1 22 4 05 p lt 0 0001 Subjects receiving cognitive feedback had a significantly higher consistency score than those receiving dutona feedback only F 1 22 2 56 p lt 0 0001 Subjects receiving cognitive feedback also had a significantly higher matching index than those receiving outcome feedback only F 1 22 2 16 p lt 0 0003 We thus conclude that subjects receiving cognitive feedback performed better than those receiving outcome feedback only The null hypothesis is therefore rejected The statistical results of the ANOVA of the performance indicators also support the hypothesis that cognitive feedback positively affects the performance of intelligence analysts B DEBRIEF RESULTS The debriefing questionnaires and task block surveys revealed information on the subjects perceptions of the experiment Overall
24. bsolete Unclassified ts Approved for public release distribution is unlimited THE IMPACT OF COGNITIVE FEEDBACK A ON THE PERFORMANCE OF INTELLIGENCE ANALYSTS i by Geraldine 8 Krotow Lieutenant United States Navy B S United States Naval Academy 1986 Submitted in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE IN INFORMATION SYSTEMS from the NAVAL POSTGRADUATE SCHOOL March 1992 Author Geraldine S Kro Approved by Kishore Sengup esis Advisor IR Jop s Second Reader David R Whipple Chairma 5 Department of Administrative Sciences ii ABSTRACT Human judgement and the process of decision making have been studied in depth for the past century More recent research has revealed that feedback is a primary element in the decision making process Feedback has been categorized according to its role in decision making Some categories of feedback include cognitive feedforward and outcome Cognitive feedback may hold the most promise for positively affecting the decision making process Naval Intelligence analysis is a complex process which involves human judgement and decision making on a daily basis This thesis sought to determine that cognitive feedback would enable intelligence analysts to make optimal choices more consistently than if they were presented with just outcome feedback Naval Intelligence analysts were the subjects of an unclassified exper
25. capture the analysis process of analysts in the Navy was devised The task was presented to active duty Naval Intelligence analysts in a format similar to what is used in the Fleet The entire experiment was unclassified and the geopolitical scenario was fictitious yet effort was taken to design a realistic task environment that the analysts would be familiar with C RESEARCH QUESTION Previous studies of military intelligence analysts have rev sied that analysts will tend to seek confirmation of their decisions Tolcott et al 1989 If this confirmation can be provided in the form of cognitive feedback will the analysts use the feedback to reach more correct and optimal decisions Does the form of feedback matter or will there be a noticeable variance between subjects provided with and without cognitive feedback Answers to these questions were sought through the design and implementation of the experiment and were the focus of this research Analysts were randomly divided into two groups cognitive feedback and outcome feedback Each group was presented with the same exact task but the information presented as feedback varied dependent upon which group the analyst was assigned to The results of each group were compared to one another and the effects of different types of feedback on the analysts decisions were collected D CONTRIBUTION The importance of the decision making process to intelligence analysis is recog
26. carefully You will receive the scenario background information and message packages manually and enter your estimated position into the computer You may also be able to use the information on the computer to receive feedback on your decisions and this will be specified in the computer instructions 3 After you complete the entire task both blocks please fill out the debrief questionnaire and the demographic information sheet provided Please be honest and as clear as possible in your responses Turn your entire packet into the experimenter 43 Cognitive Feedback COMPUTER INSTRUCTION SET You will be presented with four columns on your screen There are ten rows in each column one for each message set for a total of forty message sets per block The message sets will be presented manually Each set is on a separate piece of paper The message sets incorporate information that will enable you to estimate the target ship s location Specifically you will be given information on the ship type speed last contact in hours weather and Cuban Naval presence in the area You can enter your grid square number choice for each message set next to the appropriate message set number You need not analyze the message sets in any particular order You can maneuver about the screen with the cursor The up and down arrows allow you to select the message sets you choose to analyze and also allow you to change answers if you ch
27. cision is made Hogarth 1987 The lens model as depicted in Figure 2 represents the differences between actual value weights assigned to pieces of pertinent information by the environment and the value weights assigned to the pertinent Figure 1 Conceptual Judgement Model Hogarth 1987 Pieces of information by the decision maker These pieces of information which are considered pertinent to a specific decision or task are formally referred to as cues YO eath xpt By Xp YO matd yt do xg BH Figure 2 Brunswick Lens Model c f Dudycha and Naylor 1966 Libby 1981 The lens model uses identical polynomial equations for each side of the model the environment and the decision maker The differences lie in the assigned values or weights for each of the cues All of the variables used by the lens model are listed and defined in Table 1 TABLE 1 LENS MODEL VARIABLES c f Libby 1981 Name pefimition Environmental Predictability Matching Index gt Predictability Rige First the lens model attempts to capture the status of the environment The environment is represented by equation 1 Y a b X b X D Xp 19 Yo the actual model imposed by the environment is a compilation of the weights by bp assigned to the cues Xj Xp by the environment Likewise the decision maker s view of the task is captured in an identical equation Y wi
28. departure for this study was attained a Judgement Model Hogarth s conceptual judgement model divides the process of decision making into seven basic steps 1 task environment 2 Schema 3 Acquisition 4 Processing 5 Output 6 Action and 7 Outcome Step 7 outcome provides information that will be pertinent to future decisions This outcome of the decision making process is also considered a form of feedback for future judgmental scenarios Hence outcome or feedback is a means by which future judgements can be affected In becoming p rtinent information to the decision making process feedback establishes its vital role in the decision making process The conceptual model of human judgement is depicted in Figure 1 Feedback plays a crucial role in the judgement process as it provides criteria to both the task environment and the judgement schema The lens model as interpreted by Brunswick continues the description of the element of feedback in mathematical detail b The Brunswick Lens Model The Brunswick Lens Model allows the environment to be observed through a lens of imperfect cues Libby 1981 and assigns weights to each of the cues in a specific decision making scenario The Lens Model is readily adapted to situations in which the decision making occurs in a probabilistic environment and the accuracy of a decision is contingent both on the individual decision maker and the environment in which the de
29. dication of how 10 well an individual Y matches the environment Y through e this lens of imperfect cues Libby 1981 The Brunswik Lens Model lends well to the process executed by Naval Intelligence and Cryptological analysts By examining the analyst s decision and mathematically comparing it to the environment s actual outcome an index of achievement for intelligence analysts can be arrived at for a specific type of analysis task The effects of various factors such as feedback on the decision making process can be examined by comparative analysis of achievement indices for multiple instances of a generic task B INTELLIGENCE ANALYSIS AS A DECISION MAKING PROCESS While eaveltigence analysis can be a complex decision making process involving human judgement intelligence analysis has not been AT investigated by psychologists for more than thirty years in the manner human judgement and decision making have been Tolcott et al 1989 A preliminary literature review reveals somewhat sporadic research in the area of intelligence analysis with the majority of the material focused on developing decision support aids for the intelligence analyst In 1974 Patten attempted to initiate a method of organizing data and information received by an intelligence system to help analysts Patten s concern was the following that intelligence analyst s working on specific problems must be able to selectively obt
30. ed as feedback and if feedback could be further delineated into different types of feedback would an analyst s perception of update tatermation or feedback be affected by the specific type of feedback provided This question is addressed in section II C Feedback and its Role in the Decision Making Process l Decision Making and Intelligence Analysis The process of intelligence analysis is comparable to the standard decision making process but even more pronounced due to the high levels of stress dynamic environments and the crucial nature of the decisions arrived at After evaluating large amounts of incoming data Adelman et al 1984 the analyst must provide information on which commanders decisions are made validating the analysts position as a critical one in the chain of intelligence and operational events Lewis and Copeland 1982 It is clearly recognized that the analyst routinely faces an intricate and complex task that may be gargantuan in its proportions Luckie et al 1968 Essentially the analyst is tasked with repeatedly quantifying assessed probabilities Probus and Donnell 1986 Very often the analyst is relying solely on intuitive feelings with little or no substantiated feedback on his her decision making process 14 2 Intelligence Analysis as a Process Naval Intelligence analysis utilizes the decision making process Intelligence analysis can therefore be studied with the lens model and concept
31. es are consistency One or all of these added tools to 40 the environment of intelligence analysis could be beneficial and lead to improved analysis results C FUTURE RESEARCH Future research could use this study as a point of departure to determine which forms of cognitive feedback are most beneficial to intelligence analysis This could lead to the development of a Decision Support System prototype for U S Naval Intelligence activities 41 APPENDIX NAVAL INTELLIGENCE OFFICERS A LENS VIEW OF INTELLIGENCE ANALYSIS INSTRUCTION SET A Introduction a The purpose of this study is do examine how Naval Intelligence Officers and Specialists process complex information 2 Your participation in this experiment is voluntary and the results of your participation will be used for research purposes only Please take the experiment seriously and attempt to answer the queries as realistically as possible B Your Task 1 Your task involves three basic steps They are a Read the background information for your scenario provided in your task booklet You may need to read it over two or three times before you feel thoroughl y comfortable with it The scenario will provide information regarding the task you are asked to perform For example you will be told the history of a particular drug smuggling ship that you will later be asked to provided an estimated location for b Examine individually each of the 40 mes
32. esses The research question was investigated in an experimental setting designed to most closely represent the type of decisions made by analysts in the Fleet A realistic and fairly complex task of moderate to difficult complexity was created in an effort to employ the broadest range of the analyst s decision making capabilities B Experimental Design An experiment involving comparison of results between two sets of subjects was devised One set of subjects 12 subjects was given outcome feedback only another set 12 y subjects was given cognitive and outcome feedback The presentation of feedback was designed to provide an optimal amount of information to the subject with minimal complexity in the presentation format The subjects for each part of the experiment were randoml y assigned to receive outcome or outcome and cognitive feedback Subjects receiving cognitive feedback had a longer instruction set to read and understand before they were able to begin the experiment Outcome feedback was presented as accuracy whereas cognitive feedback was presented as Decision Support Information Both types of feedback and how they were designed and presented to the subject are described in the following sections Ca PARTICIPANTS Subjects chosen for this experiment were all on active duty in the United States Navy They are currently serving or have served in positions requiring intel iegence or cryptologic analysis Bot
33. ets will be presented manually Each set is on a separate piece of paper The message sets incorporate information that will enable you to estimate the target ship s location Specifically you will be given information on the ship type speed last contact in hours weather and Cuban Naval presence in the area You can enter your grid square number choice for each message set next to the appropriate message set number You need not analyze the message sets in any particular order You can maneuver about the screen with the cursor The up and down arrows allow you to select the message sets you choose to analyze and also allow you to change answers if you choose to At anytime during the experiment you may find out what the correct answers were for message sets you have already completed By pressing END you will be provided with the correct ship positions Once you access this information you may not change any answers you have already entered After you have completed Block One complete the Block One questionnaire and press HOME to move to Block Two After Block Two complete the Block Two questionnaire the task debrief survey the demographic information sheet and inform the experimenter that you are finished The line at the bottom of the screen will provide you with information on how to maneuver about the screen and how to receive additional information IF AT ANYTIME DURING THE EXPERIMENT YOU ARE NOT SUR
34. h Enlisted Personnel and Commissioned Officers were used in the experiment All subjects have received training by U S Navy schools and have had experience at making intelligence decisions The actual positions and billets held by the experiment s subjects varied from operational intelligence billets afloat to standard administrative type billets ashore 20 TABLE 2 EXPERIMENT SUBJECT DEMOGRAPHICS Demographic Mean standard Deviation Age years 0 9 Analysis Experience years How long since The subject demographics were calculated with the conducted analysis in the Fleet years assistance of Minitab Schaefer and Anderson 1989 D TASK The task of quantifying assessed probabilities Probus and Donnell 1986 to various cues or pieces of information is central to the intelligence analyst s decision making process This is academically and professionally acknowledged as a difficult task and has been examined and incorporated into the development of different systems to aid intelligence analysis such as The Dynamic Intelligence Assessment Aid DINAA designed by Probus and Donnell in 1986 and described in the User s Manual for the Prototype Analyst Workstation PAWS by Thompson et al in 1990 Both DINAA and PAWS represent attempts to first capture and then aid the process of intelligence analysis This highlights the first challenge faced when attempting to prove the hypothesis of t
35. hic Representation 27 Fee SETTING o e al e ce A A a 28 l Geographic Area 29 2 SYSTEM e aoe ce e a aa ry A 29 3 Data Capture Entry es 30 G DESIGN OF FEEDBACK 6 31 l Outcome Feedback 31 2 Cognitive Feedback 32 IV EXPERIMENTAL RESULTS wie 37 A TASK RESULTS 56 5 we 37 B DEBRIEF RESULTS s 1 1 1 eee ee ee 39 Va CONCLUSIONS 1 we ee ew a 40 Bid SUMMARY lt 6 2 sb 8 e a Re Te oe as A ee 40 B CONTRIBUTION Ce o a pea a ae as eh a AS 40 C FUTURE RESEARCH APPENDIX LIST OF REFERENCES INITIAL DISTRIBUTION LIST vi 41 42 61 65 TABLE TABLE TABLE TABLE TABLE LIST OF TABLES Lens Model Variables Experiment Subject Demographics Experimental Cue Characteristics Means and Standard Deviations of Performance ANOVA of Model and Error vii 21 25 37 38 Figure Figure Figure Figure Figure 1 2 3 4 5 LIST OF FIGURES Conceptual Judgement Model EA A 7 Brunswick Lens Model Lo HA AAA ee ee a 8 Outcome Feedback Displayed as Accuracy 33 Analyst s Decision Weights gett Bi wei Wee a 34 Consistency Information 35 viii 1 INTRODUCTION A BACKGROUND
36. his study the experiment must include a task similar in complexity to 21 the task performed by a Naval Intelligence analyst on a daily basis The validity of the experiment hinged upon the data collected from the analysts and the data could only be as realistic and meaningful as the task itself 1 Selection of Task The task of intelligence analysis requires the analyst to provide a decision based upon whatever information is available To represent this task and ensure the robustness of the experiment a task similar to that performed by analysts in the Fleet was devised The task requires the analyst or subject to provide his her best estimate of a suspect ship s position based upon given information The given information was dependent upon whether or not the subject had access to solely outcome feedback or cognitive and outcome feedback 2 Description of Task The actual task presented to the analysts involved an anti drug smuggling situation Anti drug smuggling tasking was chosen due to its relevancy in military operations and intelligence The task was first presented to the analyst froma political standpoint of the situation in the Background Information The Background Information provided current domestic and international political and social information as it pertained to the influx of illegal drugs into the United States The purpose of the Background Information was to create a basis of familiarity for
37. iment which captured a realistic task performed routinely by analysts in the Fleet The experiment revealed that cognitive feedback produced the most accurate and optimal results and indicates that intelligence analysis would benefit from decision support systems that incorporate the element of cognitive feedback lii II TABLE OF CONTENTS INTRODUCTION A BACKGROUND B EXPERIMENTAL TASK C RESEARCH QUESTION oo oeoa a D CONTRIBUTION THEORETICAL PREMISE A THE DECISION MAKING PROCESS l Complexity 2 Decision Making Models a Judgement Model b The Brunswick Lens Model INTELLIGENCE ANALYSIS AS A DECISION MAKING PROCESS 1 Decision Making and Intelligence Analysis 2 Intelligence Analysis as a Process FEEDBACK AND ITS ROLE IN THE DECISION MAKING PROCESS 1 Outcome Feedback 2 Cognitive Feedback HYPOTHESES we iv 11 14 15 15 16 17 18 LITE METHOD 00 0 4g el aa a A a Be 19 A INTRODUCTION a sos s e A E Re 19 B Experimental Design e e ees 19 Es PARTICIPANTS c a a A asa as 20 i Ds TASK 6 a e e ee de a de a a 21 j l Selection of Task os ee 4 ae Ow 4 22 2 Description of Task 22 3 Task Variables Cues 66s 23 E PRESENTATION we ew te et ee 25 l Task Description Information 25 2 Geograp
38. ing a position and thus greater ease in calculations involving the positions of the suspect vessel The grid square did not however detract from the continuous nature of the latitudinal and longitudinal scales of measurement Continuous values for the position s of the suspect vessel were preserved with the grid square as each position remained unique and numerically higher or lower than any other grid square The grid square numbering maintained the continuity of the number line What was most realistic in the experimental setting was that the message sets presented multiple cues to the subjects in a format familiar to the subjects This enhanced the viability of the results captured by the experiment as it allowed the analysts to complete a task as familiar as possible to them Experiment debriefing revealed that nearly every subject believed the experiment and task to be highly realistic and believable F SETTING Subjects received experiment Background Information Scenario Situation Report Message Sets Computer Instruction Set and Decision Support Information manually in a loose leaf binder The experimenter presented the subject with the binder after briefly explaining the geographical area of the experiment with the aid of a navigational chart Background Information Scenario Situation Report and Message Sets were described in sections III D and III E Computer Instruction Set and Decision Support Information
39. is sequence allows you to delay asking for your accuracy and enables you to continue to refine your positions Types of Decision Support Information For illustrations of Decision Support Information see III G 2 The system will provide you with the following 5 types of decision support information 1 INFORMATION ON YOUR DECISION RULE Decision makers are sometimes unable to specify precisely a particular decision rule in this case weights assigned to specific information regarding the suspect vessel The system will track the weights you are using in 48 formulating your positions and will display them through a stacked bar chart How do I use it l Make sure the weights displayed are actually the ones you want applied 2 If not revise your scores and see how the weights change 3 Iterate between 1 2 until the system shows weights you actually want applied 2 INFORMATION ON YOUR CONSISTENCY Sometimes after decision makers have specified their decision rules they are unable to apply them consistently The system will calculate the scores you would have given had you been completely consistent with your decision rule How do I use it 1 Check your scores against the consistency scores 2 Revise your scores if you need or wish to 3 Iterate between 1 2 till your scores match with or are close to the consistency scores 3 INFORMATION ON THE SYSTEM S DECISION RULE Instead of trying to fig
40. is unlimited 2b DECLASSIFICATION DOWNGRADING SCHEDULE 4 PERFORMING ORGANIZATION REPORT NUMBER S 5 MONITORING ORGANIZATION REPORT NUMBER S 6a NAME OF PERFORMING ORGANIZATION 6b OFFICE SYMBOL 7a NAME OF MONITORING ORGANIZATION Naval Postgraduate School tf applicable Naval Postgraduate School 55 6c ADDRESS City State and ZIP Code 7b ADDRESS City State and ZIP Code Monterey CA 93943 5000 i Monterey CA 93943 5000 8a NAME OF FUNDING SPONSORING 8b OFFICE SYMBOL 9 PROCUREMENT INSTRUMENT IDENTIFICATION NUMBER ORGANIZATION If applicable 8c ADDRESS City State and ZIP Code 10 SOURCE OF FUNDING NUMBERS Program Element No Project No Task No Work Unit Accession Number 11 TITLE Include Security Classification THE IMPACT OF COGNITIVE FEEDBACK ON THE PERFORMANCE OF INTELLIGENCE ANALYSTS UNCLAS 12 PERSONAL AUTHOR S Krotow Geraldine S 13a TYPE OF REPORT 13b TIME SERR 14 DATE OF REPORT year month day 15 PAGE COUNT Master s Thesis From March 1992 74 16 SUPPLEMENTARY NOTATION The views expressed in this thesis are those of the author and do not reflect the official policy or position of the Department of Defense or the U S Government 17 COSATI CODES 18 SUBJECT TERMS continue on reverse if necessary and identify by block number SUBGROUP Impact of cognitive feedback on the performance of intelligence analysts 19 ABSTRACT continue on reverse if necessary and identify by block number Hu
41. man judgement and the process of decision making have been studied in depth for the past century More recent research has revealed that feedback is a primary element in the decision making process Feedback has been categorized according to its role in decision making Some categories of feedback include cognitive feedforward and outcome Cognitive feedback may hold the most promise for positively affecting the decision making process Naval Intelligence analysis is a complex process which involves human judgement and decision making on a daily basis This thesis sought to determine that cognitive feedback would enable intelligence analysts to make optima choices more consistently than if they were presented with just outcome feedback Naval Intelligence analysts were the subjects of an unclassified experiment which captured a realistic task performed routinely by analysts in the Fleet The experiment revealed that cognitive feedback produced the most accurate and optima results and indicates that intelligence analysis would benefit from decision support systems that incorporate the element of cognitive feedback 20 ee ero OF ABSTRACT 21 ABSTRACT SECURITY CLASSIFICATION RNCLASSIFIED UNLIMITED Lj SAMEASREPORT LJ DTIC USERS Unclassified Za NAME OF RESPONSIBLE INDIVIDUAL 22b TELEPHONE Include Area code 22c OFFICE SYMBOL DD FORM 1473 84 MAR 83 APR edition may be used until exhausted SECURITY CLASSIFICATION OF THIS PAGE All other editions are o
42. nized and the need to determine which types of feedback are optimal has been previously stated Thompson et al 1984 This str attempted to further determine if cognitive feedback is relevant and beneficial to the process of intelligence analysis Chapter II provides a literature review of research in the area of intelligence analysis and lays the theoretical premise for the study Chapter III describes in detail the experimental method Chapters IV and V discuss statistical results of the experiment and conclusions drawn from the experiment respectively Results of this study and any follow on studies may be used to develop more advanced decision support system prototypes for the Naval Intelligence and other military intelligence communities 11 THEORETICAL PREMISE A THE DECISION MAKING PROCESS l1 Complexity Human decision making is a complex process Studies of this process have revealed that the complexity of decision making is dependent upon many individual factors such as intuition anticipatory tendencies and judgemental heuristics Hogarth 1987 A large portion of the decision making process involves human memory and the capability or lack thereof of the human mind to process information Hogarth 1979 The human being is a fallible decision maker Hogarth 1979 The human mind is unable to effectively process continuous information from a multitude of sources especially in a stressful environment A stressful envir
43. ntact Weather Cuban Naval Presence 100 Total 3 What do you think the actual weights the system used for each of the variables was Ship s speed Ship s last contact Weather Cuban Naval Presence 100 Total 53 DEMOGRAPHIC SURVEY NAME AGE __ SEX RANK DESIGNATOR COMMISSIONING DAD a eel COMMISSIONING SOURCE YEARS IN INTELLIGENCE FIELD PREVIOUS OPINTEL EXPERIENCE DUTY STATION DATES ALL OTHER PREVIOUS ASSIGNMENTS INTELLIGENCE SCHOOLS ATTENDED NMITC BASIC NMITC OPINTEL ETC FULL TIME WORK EXPERIENCE IN YEARS HIGHEST DEGREE EARNED HIGH SCHOOL ASSOCIATE S BACHELOR S GRADUATE OTHER SPECIFY HOW LONG AGO IN YEARS DID YOU COMPLETE YOUR HIGHEST LEVEL OF EDUCATION _____ HOW FAMILIAR ARE YOU WITH COMPUTERS GENERALLY 1 2 3 4 5 6 7 8 9 Not at Very all familiar familiar 54 HOW MANY HOURS PER WEEK DO YOU USE COMPUTERS YOUR GENERAL COMMENTS REGARDING THE EXPERIMENT 55 TASK DEBRIEF SURVEY Cognitive Feedback 1 How would you rank the difficulty of your task during the past 40 scenario snapshots 1 2 3 4 5 6 7 8 9 not very at all difficult difficult 2 How would you rank the clarity of your task 1 2 3 4 5 6 7 8 9 not very at all clear clear 3 How closely does the task resemble what you do on a day to day basis in the fleet 1 2 3 4 5 6 7 8 9 does not resembles resemble at all very closely 4 Have you performed a similar task in the past Y N a If
44. onment places the human mind in a position to ignore some information and process other information dependent upon existing heuristics Hogarth 1987 In applications where decision making is vital such as diagnostic medicine and tactical military scenarios it is imperative that the information assigned the least importance is indeed worthy of such a low value Each piece of pertinent information contributing to a decision must be appropriately presented to the decision maker ina manner that will ensure the information is processed with due value It is beneficial to fully understand the importance of information presentation in the decision making process This understanding can be accomplished by studying various decision models as described in subsection II A 2 2 Decision Making Models Researchers have attempted to capture the human decision making process through observation and mathematical modeling Two models of decision making are the conceptual judgement model Hogarth 1987 and the lens model as described by Brunswick Libby 1981 The conceptual judgement model and the lens model appear to approach the task of decision making from the different aspects of psychology and mathematics Yet the two models are inextricably linked by the element of feedback a vital part of the decision making process Section II C identifies and defines feedback and its varieties in more detail By examining each of these models a point of
45. oose to At anytime during the experiment you may access decision support information on the accuracy your decision making process Simply press END and a decision support information menu will be presented to you at the top portion of your screen You may select up to six different types of feedback but once you chose feedback that displays the accuracy of your decisions you may not change any decisions already completed You will receive further information on the decision support information and how to use it before you begin the experiment All decision rule information requires that you have entered at least 10 positions before you can access this information After you have completed Block One complete the Block One Questionnaire and press HOME to move to Block Two After Block Two complete the Block Two questionnaire the task debrief survey the demographic information sheet and inform the experimenter that you are finished The line at the bottom of the screen will provide you with information on how to maneuver about the screen and how to receive additional information IF AT ANYTIME DURING THE EXPERIMENT YOU ARE NOT SURE OF HOW TO USE THE PROGRAM ASK THE EXPERIMENTER 44 Outcome Feedback COMPUTER INSTRUCTION SET You will be presented with four columns on your screen There are ten rows in each column one for each message set for a total of forty message sets per block The message s
46. ovided an estimated position could be given correlated consistency positions Consistency was i calculated by a multiple correlation between the variables and the analyst s estimation of the weights of the variables Consistency was presented to the analyst on the data entry menu in the left hand column next to the analyst s estimated suspect vessel positions An example of the presentation of consistency is depicted in Figure 5 Vessel Position 34 34 21 22 25 25 52 53 46 45 O vw 03004 wn FP Figure 5 Consistency Information Sengupta 1990 3 Information on the System s Decision Rule This function provided the subject with the actual weights for each variable or cue as applied by the system to achieve the correct or system answer This was calculated in a manner similar to Decision Support Information but the 35 multiple regression was between the actual weights of the variables and the assigned values of the variables Information on the system s decision rule was represented on a horizontal bar graph Information on the System s and the Subject s Decision Rule a combination of choices one and three this function allowed the subject to graphically compare his her applied decision weights with the weights utilized by the system Information on the Subject s Decision Rule and Consistency a combination of choices one and two this function presented the subject with both the weights he she was u
47. rieties of feedback such as outcome and cognitive What types of feedback will best suit the process of intelligence analysis The answer becomes clear when the two most basic types of feedback outcome and cognitive are examined 1 Outcome Feedback Outcome feedback is the most basic type of feedback and can be described as the results of a decision process that has already occurred The results are the outcome s of decisions previously made hence outcome feedback of a current decision does not facilitate changing or altering the current decision Outcome feedback provides information on previous decisions only current decisions are unaffected by outcome feedback from their results Outcome feedback has been challenged for its validity as a type of feedback as it can only enhance future decisions vice the current decision making process Balzer et al 1989 Several research studies demonstrate that outcome feedback may have little to no utility in an uncertain learning environment Balzer et al 1989 Hence outcome feedback may not be optimal in the intelligence analysis environment as intelligence analysis frequently occurs in an uncertain environment 16 2 Cognitive Feedback Cognitive feedback provides the decision maker with information about the decision currently being made Cognitive feedback allows the decision maker to know what their consistency rating is what weights they have assigned to cues and what weights
48. rnment In this experiment Background Information stated that the focus of the President is on the anti drug efforts of the U S This set the stage for the scenario in which the subject was asked to provide a position for the 25 suspect vessel Scenario provided information as to what billet the subject was to place him herself in and for what purpose they were to provide an estimated position of the suspect vessel In keeping with the experimental rule of ten scenarios per cue Balzer and Doherty 1991 the analyst was asked to Rie an estimated position of the suspect vessel forty separate times during the experiment Each scenario was independent of any other and incorporated the four variables or cues provided to the subject The four variables were last contact ship type speed weather and Cuban Naval exercise proximity to the location of the suspect vessel The actual values of each variable hours knots sea state proximity of Cuban Navy in nautical miles were varied in each scenario Thus each scenario used the same four variables but in a different combination of values for each instance Variables were presented to the analysts in the form of message sets which are very familiar to intelligence personnel An example of a message set is in the Appendix Since the entire experimental scenario and background information was unclassified so were the message sets the origin of each message and its destination
49. sage packages provided Each message set is on a separate piece of paper and consists of three messages similar to those you use in the Fleet The scenario will take place in the Caribbean Each message set will provide information on the ship s type speed the last contact in hours weather in the Caribbean and Cuban Naval presence in the vicinity of the suspect vessel A typical message set is composed of three messages whose bodies of information appear as follows SUSPECT SHIP LOCATED 15 HOURS AGO IN THE VICINITY OF GRID SQUARE 373 TRAVELING AT 3 KTS WEATHER REPORT FOR 15 JANUARY CLEAR SKIES WINDS FROM THE WEST AT 10 12 KTS NO WHITECAPS MINIMAL WAVES 42 CUBAN NAVAL UNITS SIGHTED WITHIN 1 NM OF SUSPECT VESSEL c After you read and analyze each message set you are to provide a grid square position number of where you best estimate the target unit to be located You will use the chart provided and the navigation set to determine this REMEMBER each message set is independent of all others hence you will estimate a new position for each message set You will complete two blocks of 40 queries for a total of 80 position estimates d This is a not a simple task and the answer may not seem easy to arrive at Do not get discouraged and remember that you were chosen for this experiment because of your expertise at decision making in a complex environment 2 Please read the computer instructions
50. ses in an effort to answer the previous question Cognitive feedback is more effective than outcome feedback in intelligence analysis tasks Intelligence analysts provided with cognitive feedback during their decision making process will exhibit improved performance and will make optimal vice satisficing decisions The research question was examined through an experiment using Brunswik s interpretation of the lens model and mathematical analysis It was imperative to first determine whether or not the subjects were affected by cognitive feedback in any way Further analysis determined exactly how the analysts were affected positively in that they were making more accurate and consistent decisions or negatively in that they were making decisions which were increasingly inaccurate and inconsistent with previous decisions The effect of cognitive feedback on Naval Intelligence and Cryptological analysts was the focus of this study 18 III METHOD A INTRODUCTION Naval Intelligence analysts make complex decisions on a daily bast sete their analysis is as an individual analyst for an operational command such as a squadron or ship or they are part of a large watchstanding team that collectively analyzes data for the Department of Defense DOD Naval Intelligence analysis routinely involves complex decision making and provides a viable arena for examining the effects of cognitive feedback on complex decision making proc
51. ssed Cognitive feedback information was presented to the subject in the following formats l Decision Rule Information This provided specific weights assigned to the variables by the analyst for the previous scenarios which the analyst had provided an estimated 33 position for This function would reveal to the subject which of the variables they were placing the most and least values on to make their decision The calculation of the weights assigned to the variables was accomplished x by the following method as in Sengupta 1990 a Weights assigned to the variables were calculated by a multiple regression of variable values and the analyst s estimation of the weights b The weights were then transformed into a percentage representation of the total value of the weights c The weights transformed into percentages were then displayed on a horizontal bar graph as shown in Figure 4 0 YA 100 Your decision Positive Le Weights Negative HI IA 100 Weights Negative Vessel type MN Last contact Sea state EE Cuban Navy His Figure 4 Analyst s Decision Weights Sengupta 1990 4 2 Consistency Information This function provided the subject with the positions they would have chosen for the suspect vessel if they were indeed applying their decision rule consistently As with choice one Decision Rule 34 Information only the message sets for which the analysts had pr
52. th the weights b b assigned to the cues x x by the decision maker Both the environment and the decision maker are using the same set of cues X X to which weisnks are then assigned In the most ideal situation the decision maker employs consistent weights for specific cues dependent upon the nature of the task Other factors that affect the decision making process and are thus represented in the lens model include consistency and environmental noise Environmental noise is also referred to as predictability Libby 1981 Both consistency and predictability utilize similar variables in their calculations R consistency measures how consistent the decision maker is in predicting the environment It is calculated by a multiple correlation between the cues and the decision maker s judgements Ras predictability measures how predictable the environment is If noise or predictability is low the probability of the decision maker attaining a reasonable task accuracy is low Libby 1981 Predictability can be calculated by employing a multiple correlation of the cues and the actual environmental values The accuracy of decisions can be measured through the lens model in the form of decision achievement ra TI GRoR 2 Consistency R and predictability R can be calculated to determine the overall accuracy of the decisions made by the decision maker Decision achievement is an in
53. tilizing as well as his her degree of consistency Accuracy this function served as the outcome portion of this version s feedback as it provided the subject with the actual vessel position As in the outcome feedback only version of the experiment once the subject accessed accuracy or outcome feedback he she was not able to revise any previously entered vessel positions 36 IV EXPERIMENTAL RESULTS A TASK RESULTS The statistical analysis of the experiment supports the hypotheses by revealing that the analysts presented with cognitive feedback and outcome feedback performed significantly better than the analysts who received outcome feedback only Hence cognitive feedback impacted the intelligence analysis process pertinent to the experimental task in a positive manner An immediate and clear measure of the performance of the analysts is represented in Table 4 TYPE OF FEEDBACK Cognitive 841 106 914 068 967 041 Feedback Outcome 658 192 811 077 717 180 Feedback The measures of performance are the achievement index or TABLE 4 MEANS AND STANDARD DEVIATIONS OF PERFORMANCE accuracy r the consistency index R and the matching s index G In all three measures the analysts receiving e cognitive feedback performed markedly better than the analysts receiving outcome feedback only 37 To more closely examine these results an analysis of vari
54. ual judgement model and may yield improved performance results with the provision of feedback to the analyst Effective intelligence analysis can be defined as an internal concept driven process vice an external data driven process Katter et al 1979 and thus as an internal process has the needs of conceptual judgement including feedback Hogarth 1987 The intelligence analyst is continually faced with making probability assessments concerning both categorical events and events which lie along a continuum Barclay and Randall 1975 Therefore the decision making process used by the analyst could be ideally studied with the aid of Brunswik s interpretation of the lens model Cc FEEDBACK AND ITS ROLE IN THE DECISION MAKING PROCESS Technically feedback is the process by which an environment returns to individuals a portion of the information in their response output necessary to compare their process strategy with a representation of an ideal strategy Balzer et al 1989 Feedback as pictured simplistically by Hogarth in his conceptual judgement model is essential to the decision making process Feedback is relevant to the decision making process and it is widely accepted that Better performing individuals would be more likely to access and use feedback information Jacoby et al 1984 It is fair to conclude from this that feedback could play an important role in improving intelligence analysis There are many va
55. uction set Additional computer instructions were presented to the subject on the screen of the computer they were using reminding the analyst to oido each message set individually and to enter each two digit estimated position of the suspect vessel into the computer after they had analyzed each message set The last instruction on the screen directed the subject to strike any key when ready upon which time the four column screen for position entry was displayed on the monitor 3 Data Capture Entry The system captured the subject s estimated positions when the subject entered their estimated position for each message set The subject was presented with a monochrome screen with four columns of ten entry places each To the greatest extent possible a graphical display was used that had been found to be effective for approximate comparison of quantitative information Brehmer 1984 The entry places for the subject s estimated positions were numbered 1 40 and further computer instruction information was presented on the bottom line of the screen The columns for data entry covered the bottom half of the screen The top half of the screen remained blank for subjects receiving outcome feedback 30 only When the subjects receiving solely outcome feedback pressed END they were presented with the actual position of the the suspect vessel accuracy or rar in the right hand columns next to their estimated vessel position
56. ure out from several examples what rule the system is using it is more effective if it is displayed in a bar graph format How do I use it 1 Use the information to get an idea of what decision rule the system has been following 4 INFORMATION ON THE SYSTEM S DECISION RULE AND YOURS This is actually a combination of decision support information 1 and 3 It enables you to compare your decision rule with that of the system and thereby emulate the system better 49 How do I use it 1 Check weights you have given versus weights given by the system 2 Revise your positions if you need to 3 Iterate between 1 2 until your weights match with or are close to the system s weights 5 INFORMATION ON YOUR DECISION RULE AND CONSISTENCY This is actually a combination of decision support information 1 and 2 The idea here is to let you revise your weights without losing your consistency at the same time How do I use it 1 Make sure the weights displayed are actually the ones you want applied 2 If not revise your positions and see how the weights change 3 Check your scores against the consistency scores 4 Revise your scores if you need or wish to YOU ARE NOW READY TO PROCEED WITH THE TASK 50 MESSAGE SET ONE FM USS SPIKE TO JTAD SUBJ VESSEL SIGHTING SUSPECT VESSEL SIGHTED BY FANTAIL LOOKOUT APPROX 7 HRS AGO AT GRID SQUARE 68 MOVING 18 KTS FM NOAA TO JTAD USCOMSOLAN
Download Pdf Manuals
Related Search