User Manual
Contents
1. 4 Lowest value 49 00 a response by clicking on the Highestvatue Bran Best estimate 50 00 Submit My 80 Interval button the question will be removed from your Outstanding Questions list and you will be unable to modify your estimate again Submit My 80 Interval Return To Question List IKI Group Response screen for revisiting a question User Manual ACERA 2010 3 Getting Started Facilitator 3 1 Logging In To log into the system direct your Internet browser to http www acera unimelb edu au elicitation tool html To log in as a facilitator enter the Search ACERA Search following details in the New ACERA home Participant Registration and click Elicitation Tool the Register button Participant Registration and Login New Participant Registration Ifyou have not previously regis seis as a participant please enter your name and email address into the fields below Once registered you will be Fi rst N a me facilitator automatically logged o into the system and allocated a Participant ID Number First Name facilitator La st Name login Last Name login E Email Address markab unimelb edu e Email Address f Existing Participant Login markab unimelb edu au Ifyou have previously registered please enter your Participant ID Number into heia ad box below ify you have forgotten your Participant ID Number you may enter the email address you regis paral2. 68462 8 45000 AE 11 ACERA 2010 User Manual 5 Appendix System Flowcharts Participant Functionality Flowchart 5 1 Answer Question Finalise Question Re visit Question Show Interval Submit My 80 Interval Accept Interval Answer Questions Group Response Show Interval Again Submit My 80 Interval Group Response ACERA 2010 User Manual Facilitator Functionality Flowchart Top Level 5 2 Save Cancel Add New Question Revisit Question Export Data ACERA 2010 User Manual Facilitator Functionality Flowchart Manage Question 5 3 Hide True Value Show True Value Exclude Participant Re Include Participant Crop Results Request Resubmission User Manual ACERA 2010 6 Appendix ET Documentation To ensure that the documents can be appropriately targeted to their respective audiences the following document suite has been produced for ET User Manual this document is targeted at participants and facilitators and explains how to use the native functionality of ET Process Manual this document is targeted at facilitators and to a lesser extent domain experts and explains how to prepare for run and interpret the results of an elicitation Administration Manual this document is targeted at system administrators and explains how to manage import and export data from the ET database Requ
3. Question 5 of the question text This will take you to the Answer Question screen To log out of the system close your browser window Refresh My Question List Question List screen User Manual ACERA 2010 ET EN The Answer Question screen allows you to enter your lowest highest and Elicitation Tool best estimates for the question as Question Details i well as your confidence in your Question 1 What percentage of the Australian population is female e Please enter your estim ates in units of of Population a n Swe r s e Use a minimum value of 30 e Use a maximum value of 70 yo tine ses 47 a Once you have entered your you ne highest 5 gt o e a responses into the four text fields a E ai click the Show Interval button This E p will display your response as an ECOS interval relative to a numerical scale m as well as showing your interval mel outon when converted to an 80 confidence level save your values If you wish to modify any of your Answer Question screen showing as the estimates as intervals estimates simply re enter the values in the appropriate text fields and click the Show Interval Again button The interval display will be updated to reflect your changes When you are happy with your interval estimate click the Accept Interval button 2 3 Finalising Your Answer If the facilitator has released the results for the question you will be take
4. enter your Participant ID Number into the text box below If you have forgotten your Participant ID Number you may enter the email address you registered with into the text box simply close the browser window Participant ID or Email Login screen Like Answering a Question The facilitator can control how many questions are released at any one EM Pac wh time For this reason there may be a Elicitation Tool zero or more questions displayed on Question List You are registered as Participant ID Number 15 John Smith smith test com your Outstanding Questions screen when you log in Outstanding Questions You have 5 outstanding question s Q1 What is your estimate of a person s chance of dying from a shark attack whilst participating in surf sports in Australian coastal answer Question 1 T i i k h waters over the next 12 months O a n swe r a q u e st O n C C t e Q2 What is your estimate of the total number of people living in or visiting Australia in the next 12 months Answer Question 2 Q3 What is the current surf sports participation rate in Australia Answer Question 3 An swe r Qu est l O n b u tto n to t h e r l g ht Q4 What is your estimate of the number of shark attacks on people whilst participating in surf sports in Australia over the next 12 Answer Question 4 months o Q5 What percentage of shark attacks on surf sports participants in Australia result in a fatality Answer
5. vith into the text box Participant ID or Email You will then be taken to the Facilitator Login Question Bank screen This screen lists the currently created questions and allows you to add new questions manage existing questions and export data The main section of the screen Search ACERA les lists current questions including al question ID description state Elicitation Tool Question Bank Administration and release Manage Questions i 2 The release column will display On this page you can add new questions edit or release existing questions and export the results of the elicitation different buttons depending on Question ID Description State Release Edit Manage 1 What is your estimate of a person s chance of dying from a shark attack whilst i i participating in surf sports in Australian coastal waters over the next 12 Results Revisit question Manage t h e state of t h e q u e st O n T h e months 2 What is your estimate of the total number of people living in or visiting CA y Australia in the next 12 months A state can be one of four 3 What is the current surf sports participation rate in Australia Results Revisit question y Manage 4 Whats your estimate of the number of shark attacks on people whilst a y d 1 ff participating in surf sports in Australia over the next 12 months Results ct terme i e re nt Va u e S 5 What percent
6. Resu Resu ACERA 2010 Revisiting Questions Elicitation Tool Manage Question The group results for question 11 are displayed below Refresh Results Release Results To Participants N Return To Question Bank How many people die each year from car accidents in Australia Initial estimates are in yellow final estimates are in green and re visited estimates are in orange people 4000 0 Participant 1 Participant 2 Participant 3 Participant 4 el Participant 5 a a Average H l e S All participants have answered this question Not waiting on resubmission from any participant No participants have been excluded by crapping Crop results to improve readability Crop Results from 0 00 min 0 00 to 5000 00 max 5000 00 Releasing results to participants Search ACERA Search Search ACERA Search 5000 0 Once a participant has finalised their estimate for a question it will be removed from the participants Outstanding Manage Questions Questions screen As the facilitator you can give the participant one more chance to Release Edit Manage e E change their answer by clicking Revisit question Manage mr a the Revisit question button Revisit question Manage Revisit question Manage E N E o Doing so will make the question re appear on the participant s Export Outstanding Questions screen The participant will then be able to
7. acera Australian Centre of Excellence for Risk Analysis User Manual Elicitation Tool 2010 Australian Centre of Excellence for Risk Analysis User Manual ACERA 2010 Contents E A A aaa 3 LES CONVCNMONS E O dic e EA ai 3 T2 Feedback ON CNIS COCUMEAL a 3 Zo Getting Started Participation 4 al BOI MOI aa 4 Zils ANS WEGA QUCSUION eop a 4 23 PIMalsino YOUN ANS WET iia 5 2 40 REVISTA O O QUESTO irc aii ADA ASA IA AA A Aris 6 3 Getuno Started Fai CA OR a 7 ts A 7 IL AAN CWS Ques oa ion aE 8 Dios RECESO RS ee eaten 9 Ds REVISTAT QUESTIONS cti ii bat at AA AA AAA AAA ARA E 9 4 Advanced Results Display Facilitator s sssnssnsnnnnunnnnnnnnnnnnnnnnnnnnnnunnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn 10 AL MANAGING QUESTIONS ii a A AAA A A E ta 10 42 EXPOLIO DAI INE 11 5 Appendix System Flowcharts ia 12 Delt PArticipant FUNCIONA ELO WEONES E O 12 5 2 Facilitator Functionality Flowchart Top Le Vel nnoncicioinininmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 13 5 3 Facilitator Functionality Flowchart Manage Question irsrsssessssssssessesssnssssnssssnssssnsssssessssssnssseass 14 6 Appendix ET Documentation 15 User Manual ACERA 2010 1 Introduction The purpose of this document is to provide Elicitation Tool ET users with sufficient information to operate the system There are three types of system users Participants Facilitators and Administrators Participants respond to questions set by
8. age of shark attacks on surf sports participants in Australia Results aran result in a fatality a Initial the question has not yet Export been released to participants Export Question Data Export Participant Data Export Answer Data Cl iC ki n g t h e Re l ea se b u tto n wi Question Bank make the question appear on uestion Ban the participant s Outstanding Questions list Released the question has been released to participants for the first time Once released a question can no longer be edited Results the question has been answered and finalised Press the Revisit Question button to allow participants to re visit the question to change their previous answers A y User Manual ACERA 2010 Re visited the question has been re released to participants 3 2 Adding a New Question Elicitation Tool Question Bank Administration Add the question details below Description Units Minimum value Maximum value True value if known Save Cancel ki lometres 0 9000 Add Question screen Add Question To create a new question click the Add New Question button This will take you to the Add Question screen From here you can set all of the necessary values for a question Each text field is described below Description the question for example What is the distance from Melbourne to Darwin Units the units the participant s resp
9. change their previous estimate After submitting a re visited estimate the participant cannot change their response again t The facilitator may choose to release the results to the participants before all responses have been received This is to accommodate situations where participants are temporarily unavailable and the facilitator does not want to hold up the workshop User Manual ACERA 2010 4 Advanced Results Display Facilitator 4 1 Managing Questions The facilitator has access to a number of advanced features available from the Manage Question screen These features include providing status information on participants as well as the ability to manipulate the display of the results Below the interval graph on the Manage Question screen are five lines of text that give status information about the participants Line 1 which participants have yet to answer the question Line 2 which participants have yet to resubmit a response Line 3 which participants have submitted an initial answer but have yet to finalise their response Line 4 which participants have yet to submit a revisited response Line 5 shows if any participants have been excluded due to cropping There are four advanced functions that can be performed Elicitation Tool e 7 Cropping you can crop the interval display for tiep 2 AS e _ the question by reducing the maximum value revo 9 _ and or increasing the minimum valu
10. e Type the cs new values into the appropriate text fields and mission from any participant click the Crop Results button All participants who submitted an initial answer have also finalised this question All participants who submitted an initial answer have also re visited this question CO Exclude Participant Select the participant s ID A eel cre ay from the drop down menu and click the De Exclude Participant button The excluded eee _ participant s interval will no longer be shown Advanced features on the Manage Question screen on the interval display and the average interval will be updated To re include a participant select the participant ID from the drop down menu and click the Re Include Participant button The participant s results will be redisplayed on the interval display and the average interval updated Show Hide True Value this button appears if a True Value was entered when the question was created Click Show True Value to display a vertical black line on the interval display that is positioned on the true value Click Hide True Value to hide it Note that the participants cannot see the true value on their screens only the facilitator Request Resubmission this option appears before the results are released to the participants If a participant makes a mistake when giving their initial estimates select their ID from the drop down list and click Request Resubmission The questio
11. g Questions list with a Revisit Question button On clicking this Question 1 What percentage of the Australian population is female The group results for this question are displayed in the graph below Initial estimates are in yellow final estimates are in green and re visited estimates After discussion led by the facilitator you may choose to modify your interval estimate Change your lowest highest and best estimates as required and click the Submit My 80 Interval button when you are done This will finalise your estimates for the question Note that you cannot view your new interval like on the Answer Question screen You will then be taken back to the Outstanding Questions screen Search ACERA Search MEVISIL WUGSUVIT I A of Population 30 0 Participant 1 Participant 2 Participant 3 Participant 4 Participant 5 Average button you will be taken back to the Group Response screen where you can adjust your interval in the same Ifyou would like to adjust your 80 confidence level interval based on the other estimates modify the values in the lowest highest or best estimate text boxes below and press the Submit my 80 interval button way as you did when finalising your Alternatively if you would like to keep your interval as itis just press the Submit my 80 interval button without modifying any values response Once you have submitted T 38 0 46 0 54 0 em 70 0 q e o
12. irements Documents this document is targeted at system developer and specifies the ET system requirements Excel ET this is an Excel spreadsheet that can be used to collect the elicitation responses from remote participants This soreadsheet is prepared by a facilitator and distributed to participants as an off line questionnaire via email These responses can then be imported into the ET database to assist with data aggregation and visualisation 15
13. n immediately to the Group Response Elicitation Tool screen Otherwise you will be taken to View Question Results You are registered as Participant ID Number 5 John Smith example example com the Pause screen and will have to wait View Results for Question 11 for the facilitator s instructions to Question 11 How many people die each year from car accidents in Australia proceed Cl ick the View G roup Results Please wait for the facilitator before continuing When instructed to press the View group results button below to view the group results for this question b u tto n to p ro ce e d to t h e Gro up Otherwise press pa Return to question ia button to return to the question list Response screen View Group Results J Return To Question List On the Group Response screen you will see your interval estimate for the Pause Screen question identified by your Participant ID as well as the interval estimates of the other participants There are four different types of intervals that can be displayed and each of which is displayed using a different colour Yellow intervals indicate initial estimates Initial estimates are responses made at the first Answer Question screen that have yet to be finalised Green intervals indicate final estimates Final estimates are responses that have been finalised on the Group Response screen Orange intervals indicate re visited estimates Re visited estimates are
14. n will reappear on the participant s Outstanding Questions screen The option disappears once the Release Results To Participants button is clicked Note that changes to the interval display on the Manage Question screen do not change the data stored in the system 10 User Manual ACERA 2010 4 2 Exporting Data From the Question Bank screen the facilitator can export three different types of data by clicking the appropriate Export Data button at the bottom of the screen All data is exported in CSV Comma Separated Values format and can be opened using software such as MS Excel The three types of data that can be exported are Question Data lists all the questions currently in the question bank Participant Data lists all the participants that have registered Answer Data lists each participant s responses for each question Sample formats for each of the different data types are shown below Question Data Format Question Max True Question Text No Value Value Value What percentage of the Australian population is female of Population 70 50 31 re visited How many kilometres of coastline does Australia have Including the ten largest islands km 10000 47070 released In what year did Kevin Rudd the ee f Year in YYYY current Prime Minister of Australia F 1980 1998 results orma enter Federal parliament Participant Data Format Answer Data Format ESES 15000 50000 a JEF 10954 6 68462 8 GH 10954 6
15. onse should be in This is useful for questions that may have more than one unit of measurement for example you could specify the above question in kilometres or miles Also note that the text should all be in lower case as itis appended to Please enter your estimates in units of on the Answer Question screen Minimum value the smallest value the user can input and the value that will appear on the left of the scale Maximum value the largest value the user can input and the value that will appear on the right of the scale True Value entering the true value allows the facilitator to see the true value on the results graph as a vertical black line The participants cannot see the true value on their results graph Once you are finished click the Save button to save the question and return to the Question Bank screen Clicking Cancel will return you to the Question Bank screen without adding the question to the list Once you have added a new question it will appear on the Question Bank screen with the Initial state If you wish to change the question in any way click the Edit button However once the Release button is clicked the question can no longer be changed User Manual 3 3 Once participants have provided their Releasing Results initial estimates for a question the results can be released to the participants To do this click the Manage button for the question You will be taken
16. reens names are highlighted using italic text Buttons and clickable icons are highlighted using bold text Key concepts are highlighted using underlined text 1 2 Feedback on this document Please forward any suggestions or feedback regarding this document to Andrew Speirs Bridge speirs unimelb edu User Manual ACERA 2010 2 Getting Started Participant 2 1 Logging In To log into the system direct your Internet browser to http www acera unimelb edu au elicitation tool html If you are using the system for the first time enter your first name last name and email under New Participant Registration and click the Register button If you have already registered type Search ACERA your Participant ID or email address under Existing Participant Login Elicitation Tool Participant Registration and Login and click the Login button New Participant Registration Once logged in you will be taken to lf you have not previously registered as a participant please enter your name and email address into the fields below Once registered you will be automatically logged onto the system and allocated a Participant ID Number YO u r Ou ts tan ding Questions scree n First Name This lists all the questions the Last Name Email Address facilitator has currently released Existing Participant Login To log out of the system at any time If you have previously registered please
17. responses that the facilitator has asked participants to review The black interval represents the average of all participants currently displayed User Manual ACERA 2010 Search ACERA Search Elicitation Tool Pilianses WUGSUVIT IA Question 1 What percentage of the Australian population is female The group results for this question are displayed in the graph below Initial estimates are in yellow final estimates are in green and re visited estimates are In orange of Population I T T T T 1 30 0 38 0 46 0 54 0 62 0 70 0 Participant 1 Participant 2 a a Participant 3 gt Participant 4 H gt Participant 5 k Average n Ifyou would like to adjust your 80 confidence level interval based on the other estimates modify the values in the lowest highest or best estimate text boxes below and press the Submit my 80 interval button Alternatively if you would like to keep your interval as itis just press the Submit my 80 interval button without modifying any values Lowest value 47 63 Highest value 52 30 Best estimate 52 00 Submit My 80 Interval Return To Question List lt lv ES Group Response screen before finalising a question 2 4 Sometimes the facilitator may ask A Revisiting a Question you to review a previously finalised question In this case the question Elicitation Tool will reappear in your Outstandin
18. the facilitator Administrators manage the database of elicitation data Participant instructions can be found in Section 2 Facilitator instructions can be found in Sections 3 and 4 Administrator instructions can be found in the separate ET Administration Manual It is assumed that this software will be run in conjunction with a facilitated risk analysis workshop using the approach documented in the ET Process Manual The facilitator prepares for the workshop by entering their questions into ET The facilitator leads workshop participants through a series of questions where participants respond with a numerical estimate in the form of an interval Each interval is elicited using a 4 step procedure that consists of Lowest estimate Highest estimate Best estimate Confidence level the degree of confidence that the interval created contains the true value Whilst the beginning and end of the workshop may be tailored to meet the specific needs of the elicitation the core of the workshop agenda will follow the outline below e Participants register login into the system e Participants answer each question as it is released by the facilitator o Submit their initial estimate for the question o Wait for facilitator to release the group responses o Review and discuss group responses o Adjust estimates if required and submit their final estimate 1 1 Conventions The following typographical conventions are followed in this manual Sc
19. to the Manage Question screen where you will be able to view the participant s initial estimates of the question as well as release the question s results and perform other actions To release the results click the Release Results to Participants button The participants will now be able to view the estimates of the other participants on their Group Response screen The participants can then discuss their responses adjust them if necessary and finalise their estimate for the question 3 4 Elicitation Tool Question Bank Administration On this page you can add new questions edit or release existing questions and export the results of the elicitation Question ID Description 1 What is your estimate of a person s chance of dying from a shark attack whilst participating in surf sports in Australian coastal waters over the next 12 months 2 What is your estimate of the total number of people living in or visiting Australia in the next 12 months 3 What is the current surf sports participation rate in Australia 4 What is your estimate of the number of shark attacks on people whilst participating in surf sports in Australia over the next 12 months 5 What percentage of shark attacks on surf sports participants in Australia result in a fatality Add New Question Export Question Data Export Participant Data Export Answer Data Allowing a question to be revisited State Resu Resu Resu
Download Pdf Manuals
Related Search