Home

Meeting the Increased Mobility Demand D7.3 - SIMPLI-CITY

1. sssssssss 70 8 3 3 Main Usability Issues Revealed During the Evaluation 70 8 4 Task Sheets for Usability Evaluation of the App Marketplace from Developers olor TEC 74 8 5 Transcription of the Debriefing Interviews ccc cccceeeeeeeeeeeeeeeeeeeeeeeeeeeeeeees 76 9 Usability Inspection from Developers Point of View eeeeessseeeeeeeeeeeeee 78 9 1 Approach taken for the Usability Inspection sseeeeseeeeeeeseee 78 9 1 1 Aim of the Usability Inspection o4 ca oet e edat eu eb died ete ed beer deret dur 78 9 12 Method Applied Arrasi iei pen ict eter eoe a iat esetete convi Sus 78 9 1 3 Teatmof EValuators rU D n lan ates 78 9 1 4 Usability Inspection Sessions sssssesessssseeesseeeeennneenenennnnn 79 9 1 5 Context IMMOMMANON 33 2 oi Sees stum hus Cea oe tu LE LAU Aca 79 9 2 Summary of the Results of the Usability Inspection essssssesssse 80 9 3 Detailed Documentation of the Findings cececeeecceeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeees 80 9 3 1 Task 1 Develop an App Using the SIMPLI CITY System 81 9 3 2 Task 2 Develop a Service Using the SIMPLI CITY System 85 9 3 3 Task 3 Utilise a Service from the SIMPLI CITY Service Marketplace for Greating QIN eom axe 89 9 3 4 Task 4 P
2. Test User User 2 User 1 User 3 User 4 User 5 User 6 User 7 Pre fest Session Session Session Session Session Session Usability Eval Session 1 2 3 4 5 6 Expertise in Usage of App Stores high medium high high high medium medium Expertise in A Software Development medium high medium medium medium high high Expertise in l Java Coding high high medium medium medium high high Expertise in Working with Windows medium high high low high low low Expertise IH low low n low n low n Developing for Android 9 Q P g 5 x i Familiar with IntelliJ IDEA no yes yes yes no yes yes Preferred Keyboard English German German German English German English Knowledge about SIMPLI CITY heard of nothing nothing used heard of heard of nothing Participated in Thinking A a T Aloud Test Ever Before x p Yes 9 yes Four of the test users self assessed their software development expertise as being medium three of them stated that they do have high expertise in software development Four of the test users stated that their expertise in coding with Java is high the other three test users self assessed their expertise with Java as being medium Three of the test users did not have much experience in working with Windows and two of the test users were not familiar with the IntelliJ IDEA None of the test users had much expertis
3. 135 14 Usability Evaluation of the Use Case related SIMPLI CITY App 137 14 1 Approach taken for the Usability Evaluation sseeseesseeeeeeeee 137 14 1 1 Aim of the Usability EvaluatlOh hoe e e ea cete Etat nies 137 14 1 2 Method Applied eseeesssessssesessssessssee eene 137 14 153 FACIWISIOD S cia aaa eA CAE MADE ACAD S CA ACE 137 TAAA TeSt USCIS cette teens Richie i tcu eec ed t A eL LE Md deed 137 14 1 5 Usability Evaluation Sessions eesssssseeeeeeeeennnen 138 14 2 Main Issues Revealed During the Usability Evaluation 140 14 2 1 Issues Noted by the Facilitator sesesseseeeeeennenee 140 14 2 2 Usability Issues Revealed by the Test Users ssseessusssss 140 14 3 Detailed Documentation of the Findings 141 T4 3 1 Usability MeltiGs scere ree ee Fee oto Het oe ere bd eret deel a chal basse sweets 141 14 4 Task Sheets for Usability Evaluation of the WP7 related App 144 15 Summary and Conclusions 2 2 tton utn dais qute dade ict taste dun exte ote nia hddendd cuenten ieee 146 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 71146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPL
4. ssseesessss 20 5 2 X Results of the Evaluation of the Use Case 7 related App ssssse 20 5 2 1 Results of the Usability Inspection by Experts seessseeeeeee 20 5 2 2 Results of the Usability Evaluation by Test Users sseeessssss 21 6 Detailed Results of the Technical Evaluation seeseeeeeneeeeeee 22 7 Usability Evaluation of SIMPLI CITY s Developer Tools eseeessssss 42 7 1 Approach taken for the Usability Evaluation seseseeeeeeeeeeese 42 7 1 1 Aim of the Usability EVvaliallOh ci ere tete e al et ex edit edis 42 Pele Method Applied acie i E a E ERI MER DIM DERE 42 Z3 Team f acl alOIS ucc tue sachet P pe e t p o eats oM Od 42 7 1 4 Technical Setup for the Usability Evaluation sssessseeeeeee 42 7 1 5 Usability Evaluation Sessions is oer teen Oron do eden ci duonmoE 43 Le Mest USES aa RD E 43 Fon EE Urin m tT 45 7 2 Summary of the Results of the Usability Evaluation essseesssssssss 45 7 3 Detailed Documentation of the Findings sssssseeeenn 46 7 31 Usability MOLTIOS isch cee tnc ca ceeds teer veinte onsite nri 46 7 3 2 Main Positive Aspects Mentioned by the Test Users sssssssss 49 7 3 3 Main Usability Issues Revealed During the Evaluation
5. 50 7 4 Task Sheets for Usability Evaluation of SIMPLI CITY s Developer Tools 56 7 5 Transcription of the Debriefing Interviews cccceeeeeeeeeeeeeeeeeeeeeeeeeeeeeeees 59 8 Usability Evaluation of the Developers Part of the App Marketplace 64 8 1 Approach taken for the Usability Evaluation cccccceeeeeeeeeeeeeeeeeeeeeeeeeeeeeeaeees 64 8 1 1 Aim of the Usability EValHallOR ene utc oe o D qu ute UR c Ot unde ded fede 64 S12 Method Applied o ene tr eto Noa tasto te can aa aaa 64 8 1 3 Team of Facilitators noe adeo E oce usus 64 8 1 4 Usability Evaluation Sessions sssssseeeeeennnnne 65 Blo MES CUS ENS rai LEER C MALLA La E LU RC so 65 T Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 5 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 81 6 Evaluation Tas Karo echo ai ee tu dedu didi wel aat iei aaa ag Ii ca dae 66 8 2 Summary of the Results of the Usability Evaluation ssssesssssssss 67 8 3 Detailed Documentation of the Findings esssssseeeeenn 68 8 3 1 Usability MelrieS carino hiceds nete ied ru e ares duudedu ieu ee ee nda Odi c n Rude 68 8 3 2 Main Positive Aspects Mentioned by the Test Users
6. eessssssssseeene 9 1 3 Document Status and Target Audience eeessssseeeeeeeeeenen 9 1 4 Abbreviations and Glossa esset et ete t A E rM M ME 9 1 5 Document Structure ssssssssssssssssssssseeeeeeeeeeeennn nennen nnn nnnenenrn nnn nennen 9 2 SIMPLECITY S Evaluation ApDFOACDhi i riae oet RE RE LE MEN DERE ME ceME M upezRa M uenziuc 10 3 Results of the Technical Evaluation seeesesssesssseeeeeeeee eene 12 4 Results of the Evaluation from Developers Point of View ccceeeeeeeeeeeeeeeeeeeees 16 4 1 Results of the Usability Evaluation by Test Users seseeeessssee 16 4 1 1 Results of the Usability Evaluation of SIMPLI CITY s Developer Tools 16 4 1 2 Results of the Usability Evaluation of the Developers Part of the SIMPLI CITY App Marketplace osito that deste ite eile ata ade ul dele Lue io atc rent d 17 4 2 Results of the Usability Inspection by Experts 17 4 3 Results of the Functional Evaluation from Developers Point of View 18 5 Results of the Evaluation from Road Users Point of View sseeeeesssss 19 5 1 Results of the Evaluation of the Road Users Part of the App Marketplace 19 5 1 1 Results of the Usability Inspection by Experts ccceeceeeeeeeeeeeeeeeneeeeeeees 19 5 1 2 Results of the Usability Evaluation by Test Users
7. 14 55 Task 1 e 14 18 Test user starts with task 1 e Types in the address and comes to the login page e The browser suggests the address from test user 2 e 14 21 task was completed the test user says it was very easy Task 2 e 14 21 Test user starts with task 2 e User 3 sees an app list but it s not clear if it is the first page clicks on the simpli city logo e The page refreshes e User 3 clicks on 2 apps and reads the descriptions e Test user s answer for completion of the task It is an overview of the application and can install it on my computer e test user states that this task was difficult Task 3 e 14 25 Test user starts with task 3 e types in navigation in the search area enter does not work clicks the search button and waits for the result not sure if further waiting is necessary test user thinks maybe the internet connection is low and waits a bit longer no result e test user types in route in the search area finds one app Increased mobility and clicks on the App e test user states that there should be more description and there should be a note how big is the App e test user says that the task is finished but it was not easy test user rates it close to difficult Task 4 e 14 30 Test user starts with task 4 e going back to the overview e test user knows how to find more information because test user did it before in task 2 e Clicks on Environmental Awareness Rising e there is no
8. In addition one test user User 0 participated in the dry run of the evaluation Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 43 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report session All of them answered the introductory questionnaire did the 4 test tasks and provided their feedback in the debriefing interview and feedback questionnaire However the evaluation session with the first test user was handled as pre test and after this session the material specifically the task descriptions and the introductory questionnaire was slightly modified according to the lessons learnt from this pre test All 7 test users were students of Computer Science at the Technical University of Vienna All of them were between 25 and 30 years old and there was only one woman among them These demographic facts resemble the target user group for SIMPLI CITY s developer package quite well According to a recent survey the average age of software developers in Europe is between 26 and 30 years and only about 6 of software developers are female Table 14 gives an overview of the relevant skills and knowledge of the test users as stated by them in the introduction questionnaire Table 8 Overview of the Test Users relevant Skills and Knowledge
9. Size LP B Published date 24 07 2015 Description Media Simpli city 2014 2015 T Mobile A k O 7 4 BEZE devel tempos21 es Search Ing Travel 24 Comments lr sir ir c X co sustainable mobility for supporting the ever growing need of the society to enhance the driver s Figure 15 Screenshot 04 T Mobile A X A 46 15 47 gt Increased Mobility WP7 Increased Mobility WP7 Tn Category Traffic y Size c 4 Version 1 0 Published date 24 07 2015 Description Figure 17 Screenshot 16 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 109 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 12 Usability Evaluation of the Road Users Part of the App Marketplace This chapter presents the results of the Usability Evaluation of the SIMPLI CITY Application Marketplace from road users point of view In Section 12 1 the Usability Evaluation approach is described briefly Section 12 2 gives a summary of the results of the Usability Evaluation and Section 12 3 outlines the findings in detail Furthermore Section 12 4 includes the Task Sheets which were provided to the test users during the evaluation sessions Section 12 5 provides a transcription of the Debriefing Interviews conducted with the test users at
10. SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Figure 32 Test Place Arranged in SRM s Office for the Usability Evaluation Sessions Four usability evaluation sessions were done end of September 2015 The average duration of each Usability Evaluation Session was 25 minutes All the Usability Evaluation Sessions followed the same process First the facilitator welcomed the test user described briefly the SIMPLI CITY system and explained the purpose of the session Then the facilitator gave an overview of the test procedure and asked the tester for her his permission that the session can be recorded and the records can be used for later analysis and reporting purposes After the test user signed the consent form the facilitator encouraged him to ask questions whenever anything is unclear In a short introductory interview the facilitator obtained from the test user some demographic information gender and age and some information regarding relevant skills and knowledge such as familiarity with smartphones Android and expertise with Thinking Aloud tests After this introductory part the main part of the test session could begin in which the test user was asked to complete the following evaluation tasks e Task 0 Use the SIMPLI CITY information App to find out the SIMPLI CITY speech server e Task 1 Insert settings related to your car car plate code of permission emission level value and power source e Task 2 Insert a de
11. SIMPLI CITY WP7 Public D7 3 Evaluation Report 5 2 2 Results of the Usability Evaluation by Test Users The Usability Evaluation of the Increased Mobility app took place in SRM s office in Bologna Italy Several Usability Issues were found during these sessions with test users and the most frequently observed reason for confusion of the test users was lack of confirmation of the user s input by the system All test users stated that they would like to get more feedback from the system In general it can be said that the usability evaluation confirmed the prototype stage of the Increased Mobility app and revealed a lot of useful hints for further improvement of the system More detailed information about the Usability Evaluation of the WP7 related SIMPLI CITY app can be found in Section 14 of this document 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 21 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 6 Detailed Results of the Technical Evaluation The Technical Evaluation which was conducted by the SIMPLI CITY partners at the end of the project looked at the fulfilment of the requirements which were specified in deliverable D2 3 in the first project year The developers of the SIMPLI CITY system were asked to provide information wh
12. The Cloud Based Information Infrastructure provides a robust API which is accessible via REST All the defined functionalities are provided however it would be good if additional query or search functionality would be implemented For now only JSON based queries are allowed External Data Sources Requesting data from OpenGovernment OpenStreetMap i e external data sources is quite stable and works as expected However since these data sources are outside of the controlled environment we are not able to cover all errors e g the services are under control of a different provider who might change the API or remove the service at all without prior notice This will influence the overall stability as errors may appear unexpectedly Hence if the project will be continued contracts to external data sources providers need to be established SLAs REST Proxy for invoking backend services from outside of the SRE The REST Proxy is a well document and robust component It provides all the requirements as defined in the project documentation However although backend services are under control of the SRE the REST Proxy is not always able to figure out the cause of an error e g if a backend service failed or just returned NULL the only thing the REST Proxy can do is forwarding that information to the requester Hence it would be good to provide additional error handler or monitoring components In addition so far the REST Proxy do
13. 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Action Sequence for Yes Success Failure Story Comment Idea for Improvement Completion of Task 1 Copy the bundled App User will know what to x Sending the developed Rename Send Bundle to Send to device ed pin usmg do at this step SEE to o pida IS After clicking on Restart ADB Android Debug pene Bundel E E an espe Bridge display a message that ADB has been SIPP restarted When clicking the Send Bundle button display a window with the current upload speed and then a dialog upload complete Add confirmation dialog asking if the developer really wants to send the app to the device The publish button is missing together with username password combination This functionality is needed for immediate publishing of the app to the marketplace At the moment this is done using the web UI of the marketplace and not via the plugin User will know that this X was the right action Run the Personal User will know what to X To test the deployed app Mobility Assistant or do at this step the developer has to run close and run if it was the PMA which the already running developer knows how to do as they are familiar with SIMPLI CITY basics User will know that this X was the right action Document Date i Page D7 3_Evaluation_Report_v1 0_For_Approval docx 2015 10 30 Status For Approval 84 146 http www simpli
14. 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report p Sounds p SIMPLICITY lt gt SIMPLICITY SOUNDS Should notify after Speech Re Y v v Refresh Refresh Location unknown Location unknown Position Position Refreshing data E C3 mm e aim mm Figure 26 Screenshot S4 Figure 27 Screenshot S5 Figure 28 Screenshot S6 e E EZ lt gt SIMPLICITY p SIMPLICITY E Sp Settings Navigate Increased Mobility Sounds Select origin Navigate Server Select destination Map Apps Settings Select time slot Settings Select transport mode Show route 0 D Figure 29 Screenshot S7 Figure 30 Screenshot S8 Figure 31 Screenshot S9 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 136 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 14 Usability Evaluation of the Use Case related SIMPLI CITY App This chapter presents the results of the Usability Evaluation of the Use Case related SIMPLI CITY app from road users perspective In Section 14 1 the Usability Evaluation approach is described briefly Section 14 2 gives a summary of the results of the Usability Evaluation and Section 14 3 outlines the findings in detail Furthermore Section 14 4 includes the Task Sheets which were given to the test users during
15. Initials Name Organisation MB Mauro Borioni SRM GL Giuseppe Liguori SRM MK Michaela Kargl FGM DW Doris Wiederwald FGM 11 1 3 Context Information Information related to the context in which the Usability Inspection was carried out is presented in the following Table 19 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 101 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Table 19 Context information Users User of this product may be anybody who owns an Android Smartphone and wants to use SIMPLI CITY Apps It is assumed that the users of this product have the following relevant skills knowledge and experience e Good English language skills but not necessarily English native speaker e Familiar with surfing the internet using the Smartphone e Experienced in getting Apps from Google Play Store Tasks The main tasks that users is expected to perform with this product are e Login to the SIMPLI CITY Application Marketplace e List all the Apps in the Marketplace e List those Apps in the Marketplace which are already installed on the Smartphone e List those Apps in the Marketplace which are not installed on the Smartphone yet e Select a non installed App and request App installation e Once installation conf
16. What was the best thing User 0 think the easiest was to delete the service That was the easiest part But the best do not know the best was that everything worked from the first time And even though there were some errors it still worked do not know why Moderator And what was the worst thing User 0 The worst thing well did not know whether I m done or not with this first stuff with the plugin It was hard like I just copied some parts and then did not know whether I m done or not Moderator What would you have expected User 0 do not know just some kind of confirmation Actually normal plugins are usually installed with some kind of installation menu next gt next gt next and then check some checkboxes to not install some crapware and then usually get the confirmation installation complete and then a checkbox like start it and then see that everything works Moderator So you would have expected something like that User 0 Yes it would be better Test user User 2 Pre Test Moderator How was it User 2 I liked it Moderator So regarding the SIMPLI CITY system what did you find especially good Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 50 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserve
17. Xt This is an app Increased Mobility WP7 Traffic 0 Comments bed c 2 t Figure 18 No introductory text on the App Marketplace DE sreco asmone EE A hig evell tempos21 com sir ty demo public applicationSearch 2 C Q Suchen raudvrts r A o 6 simplicityss we aca e E ve A Simplicity 2014 2015 Figure 19 Search result for navigation Missing system feedback Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 119 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report E simis city Application Ma x amp Q hups devell tempos2i com iimpli cty demc publsc spplicationDetad C Q Suchen ra 964 0 Environmental Awareness Rising Category Travel t t stalled ae insta B we Version 5 8 8 A Published date 09 12 2014 Description habits adopting eco behaviour 24 Comments 24Rating ktt subject wero Simplicity 2014 2015 ys Simpli city Application Ma X a p bd id a 4a gt 0 a Il Q hittps devell tempos21 com simpli city demo public apphcationDetail zu in Ci simplicity Information o Application installation requested OK Description Media Comments Figure 21 Pop up window appearing after clicking
18. 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report TASK 4 Find out more detailed information for any of the Apps you are interested in Tell the moderator when you have finished this task Then answer the following short question Overall this task was Very Difficult Very Easy TASK 5 Install any of the Apps on your Smartphone Tell the moderator when you have finished this task Then answer the following short question Overall this task was Very Difficult Very Easy TASK 6 Log out from the SIMPLI CITY App Marketplace Tell the moderator when you have finished this task Then answer the following short question Overall this task was Very Difficult Very Easy u u u u u u u 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 122 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 12 5 Transcription of the Debriefing Interviews In the debriefing interview the test users explained what they liked at the SIMPLI CITY App Marketplace and what they perceive to be the main issues that should be improved Figure 22 Test Users answering the Moderator s Questions during Debriefing In
19. Ecolndex service and communicated to it via REST Proxy Using the OSGi standard underneath SRE has proven to be a robust extensible platform Especially it should be mentioned that a wealth of information and documentation is available on the Internet as the OSGi platform is widely used SIMPLI CITY takes advantages of OSGi functionality such as hot redeployment of services management functionality and much more however lacks elasticity and fault T Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 97 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Partner Opinion and Comments tolerance where a new instances of SRE could be started should load increase Such functionality was foreseen as optional in the original set of requirements however implementing it would be beneficial especially in the real production used of SIMPLI CITY PMA Personal Mobility Assistant The PMA was used by TIE to help other partners implement use case apps and during the work on the Application Runtime Environment ARE This component has proven to be quite complex and requiring help from the developing partner The component needs more maturation and a better set of documentation for smoother integration Nevertheless the authors of the component were ready to
20. It is not possible to install the app even pushing on the Install button It seems itis not active at all and no message is provided to inform the user about that It s also not possible to check the installation process 1 3 X X X 3117 1 App Marketplace is not designed as a responsive website gt the page view does not adjust automatically to the size of the display and thus the page is only partially displayed on the small display of the mobile phone 8 and 11 2 75 X X X X 01 03 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 106 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Reported by Corresponding Description of the Observed Usability Violated Severit Evaluator Screenshot s y to be found in MB GL mk DW Section 11 4 Heuristic Sometime it happens that the user is thrown out by the App Marketplace and has to re login Not clear when 3 connection is timed out i e when you 1 2 75 X X x n a have to re enter the password jumps back to Login page very quickly after a minute or so without explanation When the user push the button back to move from a page to the main menu 5 it seems that a logout process happens 3 2 75 X X n a since if the user wants to visit another
21. RICERCHE FIAT Centro Ricerche FIAT Italy Public D7 3 Evaluation Report ascona Ascora GmbH Germany Gy TECHNISCHE SS 2 UNIVERSITAT N DARMSTADT 3 SA Technische Universitat Darmstadt Germany FGM forschungsgesellschaft mobilit t Forschungsgesellschaft Mobilit t Austria worldline Atos Worldline Spain 39 Sse RETI E MOBILITA SRM Reti e Mobilit Italy Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 3 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report Executive Summary The evaluation strategy followed within SIMPLI CITY combined elements of technical evaluation usability evaluation and functional evaluation that are all three combined within this report The Technical Evaluation describes the grade of fulfilment of the requirements specified within deliverable D2 3 at the beginning of the project Usability Evaluation and Functional Evaluation describe the usability and the functionality of the system from users point of view However the SIMPLI CITY system addresses two user groups software developers and road users Hence the parts of SIMPLI CITY that are relevant for software developers and the parts of SIMPLI CITY that are relevant for road users were evaluated separately Within the SIMPLI C
22. Tell the moderator when you finish this task Then answer the following short question Overall this task was Very Difficult Very Easy 17 Adjective ratings related to SUS scores 25 worst imaginable 38 poor 52 acceptable o k 74 good 85 Excellent 100 best imaginable Source Bangor A Kortum P Miller J Determining What Individual SUS Scores Mean Adding an Adjective Rating Scale in Journal of Usability Studies JUS Vol 4 Issue 3 May 2009 pp 114 123 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 144 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report TASK 2 INCREASED MOBILITY gt Insert a destination within Bologna Select an area within the city centre of Bologna or reachable from your current location by passing through the city centre Tell the moderator when you finish this task Then answer the following short question Overall this task was Very Difficult Very Easy E u u u List of suggested addresses 1 Via Parigi 2 Piazza Maggiore 3 Via Ugo Bassi 4 Via Marsala 5 Piazza Malpighi 6 Via Zamboni 7 Mia Mascarella 8 Via Solferino 9 Via Goito 10 Via Monte Grappa TASK 3 Use the SIMPLI CITY information App to find out the SIMPLI CITY web site address
23. Tell the moderator when you finish this task Then answer the following short question Overall this task was Very Difficult Very Easy Q Q Q Q u Q u 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 145 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 15 Summary and Conclusions The evaluation strategy followed within SIMPLI CITY aimed to evaluate the prototypes developed within SIMPLI CITY against the requirements specified within deliverable D2 3 Requirements Analyses Report and to validate the applicability of the SIMPLI CITY prototypes in a real world setting Evaluation of the SIMPLI CITY Prototypes Against the Specified Requirements The Technical Evaluation of the SIMPLI CITY prototypes showed that the requirements which were specified with priority Must Have or Should Have were almost completely fulfilled by the prototypes developed in the course of the project Only 1 of the 38 Should Have requirements which were defined during the specification phase of the SIMPLI CITY system at the beginning of the project was not fulfilled This was mainly due to the fact that the specification details changed slightly during the course of the project The not fulfilled requirement is U188 Composition of services According to the SIMPLI
24. Will Not U73 iOS support x Have For Now Will Not U74 Blackberry support x Have For Now Will Not U75 Windows phone 7 Have support For Now Could U76 Tablet support X Have eat Will Not U77 adn fits into the x Have p For Now ay Usually resources are cached as far as The battery lifetime of Will Not f possible However in general the 975 e PMA ISIOngEr than x battery lifetime depends on the used pve 1 day d p For Now device Environment Will Not U79 ince to exchange x Have For Now P Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 29 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report fueg Explanation why not or only partially Requirement implemented Priority yes part no we 3 Must U80 Profile in the cloud x Have Storage of data in the Must vs cloud 5 Have Environment Connectivity U82 Gateway to data x Has not been implemented since this rid Sources requirement was of low importance For Now U83 Interaction with head Out of scope for the project as no use He up display case was supporting such requirements For Now U84 Verc ees E Out of scope for the project as no use Could exchange 9 case was supporting such requirements H
25. With that do not want to say that is a complex platform because once you did a task do it again is pretty easy Moderator And what was the worst thing User 1 The worst thing was the first contact with the application you face an unknown environment and you have to be focused to understand what to do in each moment to accomplish the task As I said before this only happens at the beginning think that with 1 hour will be able to master the platform User 2 Test Session 2 Moderator How was it User 2 Most of the time the application was intuitive enough to guide me without much effort but there were also some tasks specially the creation of tests that felt a little bit frustrating without any help First of all the tab changing from open tests to tests management or something similar was not intuitive and the fact that the tab s title is very small just adds difficulty up then you have to make up for yourself that you need to create a new TestGroup and that a single test can contain more than one question Aside from this which was answering the third question the worst part the application felt messy the colour scheme and the iconology are too simplistic It all felt the same very plane very boring This does not influence the core functionality but it does not motivate you to use repeatedly the service Moderator What was the best thing User 2 In my opinion the best part was the id
26. and the user should be able to get additional information about all available functionality Users should be warned when errors are likely to occur The application should provide feedback to the user when the user s input has not been understood and ask for more information if the user s input has been ambiguous or inconsistent The application should use a dialogue strategy based on confirmation 6 Help users recognise diagnose and recover from errors Error messages in the application should be expressed in plain language no abbreviations no codes precisely indicating the problem and constructively suggesting a solution 7 Customisation and shortcuts The application should provide basic configuration options and should give expert users access to advanced configuration options The application should provide shortcuts to the most frequent tasks and should allow their customization or definition The application should provide suitable messages that match the level of a variety of users The application should allow step by step actions for novices and more complex inputs for advanced users 8 Aesthetic and minimalist design The application should use simple and clear language and short sentences the information given should be concise correct and relevant 9 Minimise the user s memory load The user should not have to remember information from one part of the dialogue to another Objects and options should be visible a
27. app page the login is requested When entering text in the communication field and then pressing Search App Login Page appears 3 4 and What is the Search App Button for 5 29 i aoa as the communication field is there anyway 7 Search button not visible when 4 and 11 225 E 03 04 entering text in the communication field The install button is too far on the right 8 side and thus not visible when reading 11 2 25 X 116 the app description For the App Environmental Awareness Rising a button like Installed label is displayed The user would expect to get the possibility to uninstall the App when clicking on the button Installed 9 However this is no clickable button it 2 4 2 25 X 18 just looks like a button Furthermore also it is marked as installed the App Environmental Awareness Rising cannot be found among the Apps installed on the mobile phone User cannot enlarge or reduce the 10 screen the zoom is not possible 11 2 X n a There is no information about the 11 marketplace and no description of the 5 and 10 2 X x 01 functionality no help No support at Login Page in case 1g password has been forgotten 3 2 X Hee 13 The compatibility with OS and devices 10 2 T TA is not foreseen 14 Button Search App and Search 8 1 75 T 04 redundant 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 3
28. did not know where to find the Publish button and it was there next to the binary would place it outside User 3 guessed that the Publish button was inside the binary tab because the Submit button was there in the creation of the application User 4 I noticed that the Publish button was in the binary tab because other applications had a green check with published next to it in the binary column User 2 I would expect to find the Publish button outside next to the list 8 3 3 2 Main Usability Issues Noted by the Observer In the following the main usability issues as noted by the observer during the test are listed in the order of occurrence Task 0 Submit Add the new binary e The actions to be done to upload the binary should be sequential or simplified Task 1 Create a set of tests e The user interface does not tell you how to proceed e The users do not know if they have to create a group and what is the meaning of it e The users do not know how to assign the tests to a group Task 2 Assign tests user to application Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 71 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report e The users try to drag and drop from tests to users
29. from users to tests and even in zones from the applications that are not in the droppable section After the Usability Evaluation the described observations were provided to the respective developers and they were actively working on it in order to improve the provided functionality and resolve the detected issues 8 3 3 3 Screenshots Documenting Potential Usability Issues f Gem lela s Simpli city Application M x X C fi D devel tempos21 es simpli city demo developer applicationDetail zu Ww v a E B z AM s Mpli city esis My Apps Add App v test app Published Media amp video Comments Description Binaries Media t Upload binary Add binary Please read the related document Publication and Review Guidelines before uploading a binary Android 17107 2015 V 0 0 1 pre alpha w Published f Unpublish Simpli city 2014 2015 Figure 8 Submitting binary Issue during Task 0 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 72 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 1 Simpli city Application M x gt Q fi D devel tempos21 es simpli city demo approver testCreationList zul Nw E A e simplrcityssies Approve for testing Approve for publication Tests Management User User Approver Doc
30. the use of the Material Design but does not inforce a unified look amp feel Unified Look amp Feel for U24 third party developers see requirement U23 above User Experience Behaviour System reaction to context U30 Reaction to who is in The vehicle sensors would be able to recognize the users However the car via sensors the current use cases are independent from the driver identity 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 14 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report Requirement Explanation Regarding Partial Fulfilment User Experience Behaviour Interaction with the system Automotive quality The built in speech recognizer in Android is used The recognition U45 voice recognition quality in the automotive environment depends on the device automotive acoustic connectivity and on the speech recognition provider either built in models for in car use provided by Google or from a 3rd party The built in speech recognizer in Android is used The recognition High speech recognition quality in the automotive environment depends on the device U46 connectivity and on the speech recognition provider either built in rag provided by Google or from a 3rd party No evaluation of word error rate WER has
31. when you finish this task TASK 2 Perform the tests 1 Perform the tests a Add comments if needed Tell the moderator when you finish this task TASK 3 Approve the application as valid for publication 1 Check results of testing if needed 2 Approve the application for publication Tell the moderator when you finish this task TASK 4 Publish the application Tell the moderator when you finish this task Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 75 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 8 5 Transcription of the Debriefing Interviews User 1 Test Session 1 Moderator How was it User 1 All right All the information needed to perform the tasks was explained step by step and it was very clear Moderator What was the best thing User 1 For me the best thing is the overall functionalities offered by the platform for all the roles developer approver and tester in a single platform in a centralized way Another thing to point out is that the learning curve is quite fast It is true that some tasks were more difficult and complex than others and facing a new environment unknown and with a lot of functionalities is always complicated at least the first time
32. CITY developers this requirement was not covered within the final prototypes since now the composition of services happens in apps and not on the level of services Evaluation of the Applicability of the SIMPLI CITY Prototypes in a Real World Setting In order to evaluate how easy the SIMPLI CITY prototypes can be used by real users i e software developers and road users several Usability Inspection activities involving experts as well as Usability Evaluation activities involving test users were conducted during the project The Usability Inspection and Usability Evaluation were designed as formative evaluation in order to allow the SIMPLI CITY developers to implement further improvements to the system based on the findings of the evaluation Usability Inspection Evaluation was conducted in phases starting from June 2015 to September 2015 For all components of the SIMPLI CITY system several useful hints for improvement of their usability were obtained by the Usability Inspection Evaluation Many of these improvements suggested by the results of the Usability Inspection Evaluation were implemented by the SIMPLI CITY developers immediately However of course not all of these usability issues could be solved during the runtime of the project but the remaining usability issues are well documented and constitute valuable contributions for further developments of the SIMPLI CITY prototypes In order to evaluate how well the SIMPLI CITY prototype
33. Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 3 Results of the Technical Evaluation At the beginning of the project during the specification phase of the SIMPLI CITY system in total 212 requirements were defined in deliverable D2 3 Requirements Analysis Report Of these 115 were defined as Must Have requirements 38 as Should Have 37 Could Have and 22 Will Not Have For Now In order to be able to assess to which grade the prototypes developed within the SIMPLI CITY project fulfil these requirements a Technical Evaluation Questionnaire was prepared at the end of the project In this questionnaire the developers of the SIMPLI CITY prototypes were asked to state for all these requirements whether they were fulfilled completely partially or not at all Furthermore the developers were asked to explain for all Must Have or Should Have requirements which were not or only partially fulfilled the reason why these requirements were not implemented completely The detailed results of the Technical Evaluation Questionnaire can be found in the Section 6 of this document The SIMPLI CITY developers stated that 143 of the 212 requirements were completely fulfilled by the prototypes developed in the project Of the 115 Must Have requirements 99 were completely fulfilled Further 16 of the 115 Must Have requirements were fulfilled partially
34. Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 9 2 Summary of the Results of the Usability Inspection As the usability inspection involved evaluation of development tools Application Design Studio and Service Development API plus SIMPLI CITY marketplaces it was taken care that all evaluators invited to the session already had an ample experience with different development tools IDE integrated development environments and so on which they use in their daily work Thus it was ensured that they could subjectively and partly objectively judge whether they could use the tools being tested in their real life work They could compare the tools being discussed with alternatives they have used in the past and they were able to compare the implementation of standard features ways of interface organisation responsiveness developer friendliness and other aspects The unanimous opinion of the evaluators was positive They liked what the SIMPLI CITY development tools allow them to do for development of SIMPLI CITY services and apps and how these services and apps can be later marketed on the SIMPLI CITY marketplaces Specifically the evaluators noted positively that the development tools are based on one of the best freely available IDEs IntelliJ IDEA which together with Eclipse is one of the most widely used IDEs in the industry Furthermore the e
35. Requirement implemented Priority yes part no Could U140 Promotional aspects X Have Permit to spread the Could U141 Word X Have App Marketplace Quality checking Could pide certification x Have iot Must U143 App publication X Hav Must U145 App removal X Have Must U147 Easy to publish X Have n Must U144 App versioning X Have Will Not U146 Call for developers x Have For Now Service Marketplace Must U148 Service registration x Have Service extension Must Mg and modification 7 Have Must U150 Service management X Have c Must U151 Service versioning X Have Service Level Agreement SLA Must U152 SLA support x Have Usage of an official Must U153 SLA standard x Have Simple SLA Must U14 description standard 5 Have y Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 35 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report tuifillsd Explanation why not or only partially Requirement implemented Priority yes part no Developer studio SDK API Provision of JAVA Must U155 Api x Have m Will Not U156 DLE of C C x Have For Now Standard U157 programming x Hn ave interface Easy access to API U158 through the x e ave developer studio D
36. Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Action Sequence for Yes Success Failure Story Comment Idea for Improvement Completion of Task 1 Make any modifications User will know what to X Developers should already Manifest Editor needs to be stretched to fit the on the original example do at this step have some SIMPLI CITY window development background i e know how SIMPLI CITY works how it is organised and how to develop services and apps within its framework Add an example of fully featured app Now it is a skeleton project containing all needed files but these files do not contain much logic It would be better for developer understanding if it included a fully fledged app Add more comments to the example app files User will know that this X describing necessary steps in the app code was the right action Generate the bundled User will know what to X As the developer is ready App out of the project do at this step to try her his app on the using SIMPLI CITY gt device she he will come Send App to Device to this menu option and then Build User will know that this X was the right action Document Date i Page D7 3_Evaluation_Report_v1 0_For_Approval docx 2015 10 30 Status For Approval 83 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No
37. U108 device sensors 5 Have Access to remote Must U109 sensors e g traffic x Have sensors Access to car components Remote control of car components when the U110 functionality is s Out of scope for the project as no use dl available e g air case was supporting such requirements nite For Now conditioning heating battery charge timing Provision of web site Could UH of the car x Have Data sources Must U112 Support of open data X Have Handling of Must wits multimedia data Have Data sources Data services Configuration of the M TT U114 frequency of update X functions which iMd it Must of the data from data y Have unnecessary to update or poll Sources background services regularly Transformation Must oe support Have Must U117 Data filtering X Have Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 32 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report Jess Explanation why not or only partially Requirement implemented Priority yes part no Must U118 Data correlation x Pav E Must U116 Unified data model x Have Sp Must U119 Data summarisation x Have Handle of data Must U120 streams a Have Data sources Stora
38. a 5096 completion rate due to a bad choice on selecting one of the two buttons to submit an application One of them drives the user to fill up complex information that the users did not know In Task 1 one of the users gave up and asked for help since he had run out of ideas Table 15 Task Completion Success Rate Participant Task 0 Task 1 Task 2 Task 3 Task 4 User 1 4 4 4 4 4 User 2 4 4 4 4 4 User 3 4 4 4 4 User 4 4 4 4 Success Rate 5096 75 100 100 100 Y successfully completed not completed help needed for completion 8 3 1 2 Time on Task TOT The amount of time needed by each participant to finish their tasks depended on their background knowledge about learning on the platform task description and quality of user interface Task 0 required participants to create an application by filling up the name description language target countries and keywords upload a promotional image and submit the binary file and adding the media icon and screenshots On average this task took 4 minutes and 10 seconds while the fastest participant required only 2 minutes and 30 seconds and the slowest required 6 minutes and 8 seconds Task 1 asked participants to create a new set of tests for the application created in the previous task Then they should assign it to the application and assign also the user that will perform these tests The participants should be able
39. a password username and do at this step for authentication is common password practice there is no open user User will Know that this x registration for the App Was the right action Marketplace login credentials are provided by SIMPLI CITY personnel on request Go to Add App fill all User will know what to x This step is self explanatory the required fields and do at this step add some media images User will know that this x was the right action Click Create App then User will know whatto x Developers should already have Binaries and Upload do at this step some SIMPLI CITY development Binary and select the background i e know how SIMPLI previously generated User will know that this x CITY works how it is organised full bundle zip was the right action and how to develop services and apps within its framework Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 92 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public 9 3 5 Task 5 Put a Service on the SIMPLI CITY Service Marketplace D7 3 Evaluation Report Action sequence for Yes Success Failure Story Comment Idea for Improvement completion of Task 5 Open IntelliJ IDEA User will know what to x Developers start their IDEs daily
40. ave make money The system allows the consortium to Should pie make money from 2 Have apps Apps are free but U130 supported i e red asi ave hat model Free and premium Could U131 versions of apps Have Guarantee and U132 minimum service life ule i Have time support time Business model Finance Services The system allows U133 service developers to x fuod ave make money The system allows U134 the consortium to T Could make money from Have services Services are free but U135 supported i e red X oe lave hat model Free and premium Could vise versions of services Have Guarantee and Storage is possible but logic needs to be s Could U137 minimum service life X performed by an external payment H Fac es ave time support time provider during commercialization phase Business model Sales U138 Provision of apps Only number of installations available Must statistics X information from crashes is not available Have Provision of services Must page statistics A Have Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 34 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report unen Explanation why not or only partially
41. been performed User Experience Feedback Available only from mobile marketplace app not from the marketplace u59 Rating of apps web interface Backwards compatibility Backwards Some checks on the version and classes used by the App are done U100 compatibility ofapbs during the installation process to prevent crashes but a full list of older p y pp libraries with versions is not maintained on PMA side U102 Backwards The API version can be changed when editing the App Manifest but compatibility of API App Design Studio will provide the last library available System scalability I This functionality as such was not covered within the final prototype U124 Scalability Ore service however the overall Service Runtime Environment has been designed platform in a way that allows provisioning of scalability features in the future 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 15 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 4 Results of the Evaluation from Developers Point of View Evaluation of the SIMPLI CITY system from developers point of view included 3 activities 1 Usability Evaluation by test users 2 Usability Inspection by experts 3 Functional Evaluation by SIMPLI CITY developers The fol
42. do any further steps see screenshot Figure 21 Task 6 Log out from the SIMPLI CITY App Marketplace no usability issues encountered logout task was straight forward for all test users 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 118 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 12 3 3 3 Screenshots Documenting Usability Issues am Simpli city Application Ma X https devell tempos21 com simpli city demo public applicationSearch zu QC O suche o604 604 9 zz D SIMplt city zea T Search g se Environmental Awareness Rising Travel 24 Comments dy d cC X The domain adc sed is the eco sustainable mobility for supporting the ever growing need of the society to enhance the driver s The eco sustainability from the other side takes main direct advantages to the driver as saving fuel and money and improving efficiency that stimulate herhim to change driving habits adopting eco behaviour 4 test app Media amp video 0 Comments t t t 2 Desciption test app Fake App Media amp video 0 Comments bed t Media player All formats ali sources Prog3 Communication 0 Comments t 2 2 t Here we have to add a description Amnesty Safety 0 Comments y C C
43. do at this step User will know that this x was the right action Open an existing User will know whatto x Developers should already have some SIMPLI CITY Service do at this step SIMPLI CITY development background i e project know how SIMPLI CITY works how it is User will know that this x organised and how to develop services and was the right action apps within its framework Generate the bundled User will know whatto x The developer knows the main dialog for service out of the do at this step working with bundling of services is located project using Maven here Projects Your project gt Lifecycle gt package User will know that this x was the right action Open the location of User will know whatto x Developers should already have some the bundle under do at this step SIMPLI CITY development background i e Project location target know how SIMPLI CITY works how it is and copy the JAR file User will know that this x organised and how to develop services and somewhere to be used was the right action apps within its framework later Document Date i Page D7 3_Evaluation_Report_v1 0_For_Approval docx 2015 10 30 Status For Approval 93 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Action sequence for Yes Success Failure Story Comment Idea for Imp
44. driver identity Have dialog The vehicle sensors would be able to U30 Reaction to who is in x recognize the users However the Should the car via sensors current use cases are independent from Have the driver identity U31 Reaction to time of the x Must day Have Must U32 Reaction to sensors x Have 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 25 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report JUIN eI Explanation why not or only partially Requirement implemented Priority yes part no Reaction to KPIs from U33 the car like level of x Must fuel and kilometres Have done U34 Reaction to history of 2 Must usage Have Reaction to traffic information like traffic Must U35 jams train schedules x H ave road works accidents and strikes User Experience Behaviour Interaction with the system Natural speech Must yee recognition Have Results oriented U37 instead of service app X AI a ave recognition oriented Could U38 Disambiguation X Have U39 Provision of a limited i Could number of alternatives Have Could U40 Friendly voice X Have Could U44 Voice quality x Have Input interactions with system via multimodal Must U41 Ul o
45. following sections provide a summary of the results of the evaluation of the road users part of the SIMPLI CITY App Marketplace and of the SIMPLI CITY PMA with the Use Case 7 related App More information about these evaluation activities can be found in the respective Annexes of this document Note Information regarding the Evaluation of the SIMPLI CITY PMA with the Use Case 8 related Apps is not included in this document but can be found in the dedicated deliverable D8 3 Evaluation Report 5 1 Results of the Evaluation of the Road Users Part of the App Marketplace This section gives an overview of the results of the evaluation of the road users part of the App Marketplace For detailed information about the usability inspection by experts please refer to chapter 11 of this document For detailed information about the usability evaluation by test users please refer to chapter 12 of this document 5 1 1 Results of the Usability Inspection by Experts The Usability Inspection of the SIMPLI CITY App Marketplace from road users point of view was done by 4 evaluators applying the usability inspection method Heuristic Evaluation in Bologna and Graz in July August 2015 The version of the SIMPLI CITY App Marketplace which was delivered to the evaluators for Usability Inspection still needed some improvements before it could be considered as usable and functioning as expected The potential usability problems which were revealed b
46. help and solved many issues TUDA Service Runtime Environment and the Cloud based Information Infrastructure Partner TU Darmstadt used the Service Runtime Environment and the Cloud based Information Infrastructure The SRE component and the Cloud based Information Infrastructure component have proven to be mature well documented components offering robust functionality within a well defined set of interfaces exposed via REST web services Deploying TUDA s bundles into the SRE worked without any problems The given examples have shown to be a very good extension to the documentation Both components provided an easy way to interact and fulfil partner s tasks IBM Partner IBM had a direct interaction with the following two components only as IBM is involved in the data layer part i e no interaction with WP5 6 Cloud based Information infrastructure for storing and accessing data The Cloud based Information infrastructure for storing and accessing dynamic historic and remote data from heterogeneous sensors and actuators The infrastructure has been demonstrated to be scalable in the context of the WP7 scenario where thousands of data points are collected on an hourly basis The response time of the infrastructure is as defined during the requirement specification One of the main next steps to be considered is the synchronisation stage when accessed modified updated by different users in same time while this was not defined
47. images of the route and audio Must U210 multimedia X feedback are provided as other streams H ave information are not supported Reproduction of Should U211 X streaming audio Have Korkar Au No updates are sent to the PMA as this Must U212 Re X is not supported by PMA User needs to proximity of Points of Have request update actively Interest Social network Must ns integration i Have CRF Scenario Use Case Topic II 1 Reporting to the end U214 user about eco X d dita ave driving information Vehicle information gt Must U215 available to the x Have system Provision of real time feedback to the user Must UNS in order to improve Have his her performance Access toa ioumiev Information related to history trip eco related Scc aden y driving data included are available with Must U217 i 9 x the Eco Live assistant app choosing the data MS Hg ine option Trip History not through a web Hee specific web portal portal Comparing U218 eco performances x lg of different drivers Reward drivers U219 through the eco X icd A ave driving contest Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 41 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 7 Usability Evaluation of SIMPLI CITY s Developer Tools This cha
48. install button This is confusing That s all could say User 4 Evaluation Session 4 Moderator How was it User 4 Well you said at the beginning that it is just a pre alpha state So it s o k You can log in you can find some apps but they have to add more search words guess So not only the description but maybe some extra words whatever Well and if this works with the install button it s good From the design you can see that it is not far in the development yet So the search is separated from the button and so on But I think it s on a good way Moderator Was there anything that you find especially good User 4 Not really It s not bad but there s also nothing especially good Moderator Was there anything that you find especially bad User 4 Well bad was the search You do not find anything just typed in two words from the task description And navigation or routing should find a navigation app and it did not so that was bad Moderator Anything else So if you would be the developer of this stuff what would you improve first User 4 I would improve the design It is not It does not look very good and appealing But at the current stage it s just a technical proof think so they do not need a fancy stylish website It works that s o k And I think would improve the details page Just add a couple of screenshots of the app describe just what this app does
49. location on local file do at this step have some SIMPLI CITY system development background User will know that this X i e know how SIMPLI was the right action CITY works how it is organised and how to develop services and apps within its framework Open Project Structure User will know what to X This specific step should Ctrl Alt Shift S and do at this step be memorised by the choose JDK 1 7 if not developer While not self already selected evident it is described in the user manual User will know that this x Document Date i Page D7 3_Evaluation_Report_v1 0_For_Approval docx 2015 10 30 Status For Approval 86 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report was the right action Action sequence for Yes Success Failure Story Comment Idea for Improvement completion of Task 2 Make any modifications User will know what to X Developers should already on the original example do at this step have some SIMPLI CITY development background User will know that this X i e know how SIMPLI was the right action CITY works how it is organised and how to develop services and apps within its framework Use Maven Projects User will know what to x Majority of modern Your project Lifecycle do at this step projects use Maven for package
50. majority of comments which were received from partners during this functional evaluation of SIMPLI CITY s developer tools is related to possible future enhancements some of which were considered as optional nice to have features in the earlier stages of the project and were skipped during the implementation phase because the development of other more important must have features occupied time and resources To summarise it can be said that apart from the fact that there s strong demand for good documentation and guidance overall SIMPLI CITY partners software developers were quite satisfied with the way they had to work to develop within the SIMPLI CITY framework The organisation of components and the overall architecture had proven to be up to the expected industry standards 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 18 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 5 Results of the Evaluation from Road Users Point of View Within SIMPLI CITY evaluation from road users point of view included e Evaluation of the SIMPLI CITY PMA with the Use Case 7 related App e Evaluation of the SIMPLI CITY PMA with the Use Case 8 related Apps e Evaluation of the part of the SIMPLI CITY App Marketplace that is targeted at road users The
51. needed by each participant to finish their tasks were due to their knowledge about App stores and their experience in usage of smartphones Table 27 Participants Time On Task Participant Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 User 1 3 6 4 2 7 3 User 2 3 3 2 4 5 1 User 3 3 4 5 3 3 2 User 4 2 4 2 2 5 1 User 5 2 2 2 3 5 2 Average TOT in minutes 2 6 3 8 3 2 8 5 1 8 12 3 1 4 Rating of the Overall System Usability After the completion of all tasks the participants were asked to fill in a standardised feedback questionnaire SUS System Usability Scale As explained in Section 7 3 1 4 SUS yields a single number representing a composite measure of the overall usability of the system SUS scores have a range of 0 not usable at all to 100 perfectly usable system Analysis of the answers of the test users to the feedback questionnaire reveals that the road users part of the SIMPLI CITY App Marketplace has got a SUS score of 61 7 According to Bangor A et al this SUS score can be translated to users perceiving that the usability of the system is acceptable o k This indicates that there s still some substantial improvement of the system s usability necessary 12 3 2 Main Positive Aspects Mentioned by the Test Users During the debriefing interview at the end of the Usability Evaluation Sessions the test users stressed two features of the SIMPLI CI
52. not push on the screen to activate the microphone button and they did it only when guided by the facilitator In general the use of the microphone button seemed not to be intuitive and often the voice of the test user overlapped with the system voice Although task 3 was successfully accomplished by all the test users some test users would have appreciated if they were allowed to access the SIMPLI CITY website by simply taping on the address on the screen 14 2 2 Usability Issues Revealed by the Test Users During the Usability Evaluation Sessions the test users mentioned several issues and aspects of the SIMPLI CITY system which should be improved 1 Interaction with the system Users 1 and 2 The start of application is too slow and the screen remains dark for too long time It is not clear if the system is starting or not Users 1 3 and 4 The system does not confirm the actual acquisition of data No feedback is provided to the user Users 2 and 3 The system does not show the user what data have already been inserted and saved 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 140 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report User 1 Saving data already inserted is not possible at user willing User 2 A Return home feature is not pr
53. of Services and Human Computer Interaction For more information please refer to the project website at http www simpli city eu Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 8 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 1 2 Deliverable Purpose Scope and Context The purpose of this deliverable is to outline the evaluation approach taken within WP7 of the project SIMPLI CITY to describe the evaluation related activities conducted by the consortium and to provide a detailed overview of the results of these evaluation activities 1 3 Document Status and Target Audience This document is listed in the Description of Work DoW as public The results of the evaluation activities conducted within WP7 of the SIMPLI CITY project might be interesting for all parties who want to use and exploit parts of the SIMPLI CITY system since these evaluation results give valuable hints for further improvement of the system 1 4 Abbreviations and Glossary A definition of common terms and roles related to the realization of SIMPLI CITY as well as a list of abbreviations is available in the supplementary document Supplement Abbreviations and Glossary which is provided in addition to this deliverable Further information can be found at http www simpli c
54. of course it s however a strong motivation to publish to the market place Test user User 1 Test Session 1 Moderator How was it User 1 At first was not 100 sure what is the purpose or the aim of the tasks However since they were chained together the picture became more clear afterwards Moderator What did you find especially good Was there anything User 1 The tasks were pretty straight forward except of the bundling which involved some extra steps It was quite handy was not aware first that the IDE is actually communicating and interacting with an external marketplace in the background without me noticing but it seemed that the publishing worked pretty straight forward it seems very nice In my experience it can be rather difficult to publish a service in an open marketplace like that So if it stays that easy it would be a very handy thing to have Moderator Was there anything that was especially bad User 1 The documentation was not 100 intuitive and without Maven background and without IntelliJ background which had before assume that would have had more difficulties in locating specific elements and packaging the bundle But this cannot be avoided I think because it s just so that if you are familiar with a tool it s good but if you re not you have to get used to it if you want to work with it So you cannot really get around this but the documentation could be a bi
55. of the project Talkamatic Speech Backend MMDI While creating the Application Runtime Environment partner Ascora came in contact with the Voice Recognition and Speech Output system of partner TALK The use of the component turned out to be very easy and straight forward A good documentation and very good communication with partner TALK helped Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 96 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Partner Opinion and Comments while using the component Partner Ascora is planning to use this very component for their own WP8 mini project which is explained in more detail in the respective document The components provide an excellent way for applying voice interaction to apps within and beyond SIMPLI CITY An enhanced support of additional languages could enhance the components after the project s ending Application Runtime Environment For the creation of the ARE Push Service which is the connection between the Application Runtime Environment and the Service Runtime Environment the Message Handler component of the ARE had to be used which proved to be as fully function as expected Apart from that the Service pulling from the Service Runtime Environment had to be used Ecolndex Calcula
56. phone Done 10 47 finished user states that it was easy Test User 2 09 09 2015 Timeline e Introduction 11 14 11 20 Explaining example task 11 20 11 24 User Test 11 24 11 43 Debriefing Interview 11 43 11 49 Task 1 e 11 24 Test user starts with task 1 e Types in the address comes to the login page Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 127 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report e The browser repeats the address from test user 1 e 11 26 task completed the test user says it was relatively easy but the address is very long Task 2 11 27 Test user starts with task 2 it s not clear for the test user how he shall solve this task clicks on the first app User 2 does not know exactly what the website is about test user says this task was very difficult Task 3 11 30 Test user starts with task 3 types in navigation in the search area no result types in destination in the search area no result back to the app list clicks on the last App Increased Mobility test user states that this task was not so easy Task 4 11 32 Test user starts with task 4 going back to the overview first App is the only one that has stars in the ranking the button does not look like a button clicks on the first app i
57. stated that their experience in browsing in the internet is high one test user self assessed that his experience is medium None of the test users was involved in the development of the SIMPLI CITY App Marketplace One of the test users was working for other parts of SIMPLI CITY three of the test users had heard of SIMPLI CITY before and one test user did not know anything about SIMPLI CITY Two of the test users had participated in a usability study before but none of them in a Thinking Aloud Test 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 112 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 12 1 2 5 Evaluation Tasks The test users were asked to complete the following tasks e Task 1 Login to https devel1 tempos21 com simpli city demo as a user e Task 2 When you ve logged in to https devel1 tempos21 com simpli city demo find out what this website is about e Task 3 Search for Apps that support you with navigation and help you to find the best route for driving to any destination Hint The site offers a search function e Task 4 Find out more detailed information for any of the Apps you are interested in e Task 5 Install any of the Apps on your Smartphone e Task 6 Log out from the SIMPLI CITY App Marketplace Sect
58. the SIMPLI CITY app is not available on the Google Play Store e 16 38 test user stops trying task 5 on the smartphone 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 132 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 13 Usability Inspection of the Use Case related SIMPLI CITY App 13 1 Approach Taken for the Usability Inspection Starting from July 2015 a Usability Inspection was conducted by SRM every time when a new release of the WP7 related SIMPLI CITY app Increased Mobility was available The Usability Inspection method used was task based Heuristic Evaluation as described in detail in Section 11 1 4 The results of each of these Usability Inspections were communicated to the developers immediately after each inspection so that the app could be improved accordingly Table 28 below gives some general information regarding the expert who did the Usability Inspection the place where the inspection was conducted and some relevant technical details about the device used for the Usability Inspection Table 28 Usability Inspection s of the Increased Mobility App General Information Evaluator Name Organisation Giuseppe Liguori SRM Date s of Usability Inspection July September 2015 Place of Usability Ins
59. the destination street in Bologna No GUI interaction allowed white screen no list of cities available 6 Increased mobility gt Settings gt Configure the emission 1 3 3 level value gt specify the value gt euro 3 GUI No possibility to come back to the main menu The app must be restarted 7 Increased mobility gt Settings Configure the Power 1 3 3 source value gt specify the value gt diesel GUI No possibility to come back to the main menu The app must be restarted 8 Increased mobility gt Map gt Position the position is 1 S5 3 always unknown 9 Increased mobility gt Map gt refresh button does not 1 S6 3 work and the position is always unknown 10 Increased mobility gt Settings gt Configure the Power 1 3 2 source value gt specify the value gt diesel GUI No feedback is given to the input of the emission value 14 refer to Section 11 1 5 for details on the utilised List of Heuristics 15 Related screenshots are provided in Section 13 2 1 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 134 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Violated Corresponding Description of the Observed Usability Issue H
60. the evaluation sessions 14 1 Approach taken for the Usability Evaluation 14 1 1 Aim of the Usability Evaluation This Usability Evaluation was done in order to find out the main usability problems novice users encounter while trying the Increased Mobility app It was a usability evaluation both on Vocal and on Graphic interface The results of this Usability Evaluation give valuable hints for further improvement of the SIMPLI CITY system 14 1 2 Method Applied For the Usability Evaluation of the Increased Mobility app the well established Usability Evaluation method Thinking Aloud Test was applied A test user is asked to speak out loud all her his thoughts while using the system to complete a given test task By observing the test user s way of working on the task and by analysing the reasons for confusion hesitation or mistakes of the test user valuable hints for improvement of the usability of the system can be obtained 14 1 3 Facilitator The Usability Evaluation of the Increased Mobility app was organised and conducted by Giuseppe Liguori working in SRM and strongly involved in the SIMPLI CITY project activity He dealt with the preparation of the testing sessions and ensured proper functioning of the technical equipment and software He guided the test users through the session as moderator and took notes of revealed usability issues 14 1 4 Test Users In total 4 test users participated in the Usability Evalu
61. the install button Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 120 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 12 4 Task Sheets for Usability Evaluation of the App Marketplace from road users point of view TASK 1 Login to https devel1 tempos21 com simpli city demo as a user Hints Your username is consumer Your password is also consumer Tell the moderator when you have finished this task Then answer the following short question Overall this task was Very Difficult Very Easy TASK 2 When you ve logged in to https develi tempos21 com simpli city demo find out what this website is about Tell the moderator when you have finished this task Then answer the following short question Overall this task was Very Difficult Very Easy TASK 3 Search for Apps that support you with navigation and help you to find the best route for driving to any destination Hint The site offers a search function Tell the moderator when you have finished this task Then answer the following short question Overall this task was Very Difficult Very Easy QO u u u u u u Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 121
62. the requirements but would be a great extension for future work Sensor Abstraction component within the Apps which are deployed in the ARE The Sensor Abstraction component is a nicely designed component which allows easy access to arbitrary sensors via a single interface e g it provides the means to read the current user s location phone s GPS calendar contact list as well as the multiple car sensors fuel level etc ASC Service Registry Partner Ascora used the Service Registry while developing the Service Marketplace as well as the Service Development API which is integrated into the Application Design Studio In both cases the Service Registry proved to be a fully functional yet dynamically extendable component which provided all necessary interfaces to access and maintain the list of the available Services to 2nd party partners project partners For usage in SOA Environments the Service Registry as well as its parent component the Service Runtime Environment itself are more than feasible to be re used Sensor Abstraction Ascora has used the Sensor Abstraction and Interoperability components during its WP8 work for connecting to car based sensors The components have proven to demonstrate a good start on how to access sensor systems and on how to get values of them For further development an abstraction from the car manufacturer implementations of the OBD protocol would further enhance the component beyond the scope
63. through that dialog box and finding the correct functionality Also it is not possible to open the Development Guide while this dialog is active 2 1 9 Important development related operations lack any visual feedback Participants were often wondering if the requested operation succeeded or not 2 2 Problems were detected in SIMPLI CITY plugin functionality 2 2 1 New SIMPLI CITY project setup is incorrect The users had to perform additional unexpected steps such as Java SDK setup in order to have a working solution 2 2 2 User Authentication Handling is missing Actual service registry instance needs to have user authentication enabled 3 Task 2 Find the created Backend Service in the Service Marketplace 3 1 The need to login is not obvious Functionality should not be completely hidden but rather greyed out if the user is not authorized or logged in 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 52 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 3 2 The search menu is case sensitive what is unexpected for many users 4 Task 3 Delete created service 4 1 Delete Service is located incorrectly The participants had difficulties finding that functionality in the SIMPLI CITY menu item 4 2 System feedback is missing Whe
64. to generate project dependency the project User will know that this x control and building Developers are likely to be already familiar with Maven Described is a standard way of building a project Document Date i Page D7 3_Evaluation_Report_v1 0_For_Approval docx 2015 10 30 Status For Approval 87 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Action sequence for Yes Success Failure Story Comment Idea for Improvement completion of Task 2 Go to SIMPLI CITY User will know what to X Once a service is Rename menu item Bundle amp persist to Upload Bundle Service and do at this step developed the developer Use a bigger window for the Bundle amp persist persist and select naturally wants to deploy it dialo Bundle it to the Service Runtime 9 Environment for tests Move Service Runtime Environment URL setting to this dialog from the separate settings dialog Add a list of SRE servers to persist the bundle if the developer wants to deploy to local or to remote servers then they just need to select from the list instead of retyping the URL User will know that this X was the right action Rename Delete service to Uninstall service this option uninstalls the service from the SRE Rename Bundle It to Deploy service this option dep
65. 0 Status For Approval 107 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Reported by Corresponding Evaluator Screenshot s to be found in MB GL mk DW Section 11 4 Description of the Observed Usability Violated Heuristic Severity No explanations under the respective apps on how to download them integrate comments etc Within the app mask it is not provided the information about the size of the app The developer s name contacts and reputation are not displayed A link to SIMPLI CITY website is missing No first time registration possible at Login Page It s not clear how it will be possible to rate the apps No explanation what the Documentation Link contains Why is there a link to the Publication and Review Guidelines at the site dedicated to consumers road users that s confusing Unclear not explained what the communication field is for Entering text there is possible no search function nothing happens 01 03 There is no introductory text on the App Marketplace Design is not consistent it is not clear which elements are clickable buttons which elements are links and which elements are expandable menu headings The area allocated to include the logo of the apps is too big considering the size 01
66. 01 SIMPLI CITY WP7 Public D7 3 Evaluation Report Table 9 Task Completion Success Rate Participant Task 0 Task 1 Task 2 Task 3 User 1 4 4 Jv User 2 E j 4 Jv User 3 j 7 vA 4 User 4 NA NA NA NA User 5 4 7 4 4 User 6 7 4 4 Jv User 7 4 4 4 J Success Rate 57 43 100 100 Y successfully completed not completed help needed for completion 7 3 1 2 Ease of Task Ratings After each task the participants rated the ease of completing the task by answering the Single Ease Question SEQ Overall this task was on a 7 point rating scale ranging from Very Difficult 1 to Very Easy 7 Table 10 shows the ratings given to the single tasks by the test users and provides an average of these ratings for each task Table 10 Ease of Task Ratings Participant Task 0 Task 1 Task 2 Task 3 User 1 6 4 7 4 User 2 5 5 7 7 User 3 3 4 7 5 User 4 6 5 6 6 User 5 7 4 7 6 User 6 4 5 6 5 User 7 6 6 7 5 Average Rating 5 1 4 5 6 5 5 6 Percent Easy Rating 6 or 7 5096 13 88 50 All participants perceived Task 2 Find the created Backend Service in the Service Marketplace to be the most easiest one and Task 1 Try to develop a Hello World Service by using the SIMPLI CITY system to be the most difficult test task Some of the test users also perceived Task O Install the SIMPLI CITY p
67. 3 3 1 Main Usability Issues Mentioned by the Test Users During the debriefing interview at the end of the Usability Evaluation Sessions the test users mentioned several aspects of the SIMPLI CITY system which should be improved The main issues were 1 Introductory text is missing User 2 When you arrive on the website where you have the selection of apps there should be a brief explanation where am That would be the most important thing saying This is SIMPLI CITY and you can download various apps here A brief explanation that would be the most important thing User 8 did not know what to expect when I typed in the link That s why was confused was not sure whether this is the first page would expect a bit of explanation on the first page what is the website about or something There was just a list of apps Of course if you actually want to search for an app it s quite easy because you see the list at once and do not need a lot of time But maybe a sentence or two at the beginning would be useful for people who have not visited the web page before User 5 Maybe something on top about this app store or app marketplace whatever some background information 2 App Marketplace should work on a Smartphone User 2 There was one important thing on the Android on the screen of the mobile it was the install button was not visible You had to scroll so that shouldn t be the case Us
68. 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 11 Usability Inspection of the Road Users Part of the App Marketplace This chapter presents the results of the Usability Inspection of the SIMPLI CITY App Marketplace from road users point of view This Usability Inspection was done in July August 2015 in Bologna and Graz In Section 11 1 the Usability Inspection approach is described briefly Section 11 2 gives a summary of the results of the Usability Inspection Section 11 3 outlines the findings in detail and finally Section 11 4 presents screenshots taken by the evaluators to document identified potential usability issues 11 1 Approach taken for the Usability Inspection 11 1 1 Aim of the Usability Inspection This Usability Inspection was done in order to reveal potential usability issues of this part of the SIMPLI CITY App Marketplace which is dedicated to users Usability Inspection of that part of the App Marketplace which is dedicated to software developers was done separately The results of this Expert Evaluation give valuable hints for further improvement of the SIMPLI CITY system 11 1 2 Team of Evaluators The Usability Inspection of the road users part of the App Marketplace was done by the four evaluators indicated in Table 18 who did not do the Usability Inspection as a team work but each of them inspected the application independently Table 18 Evaluators of the App Marketplace
69. 4 J J z V User 2 J z z z 4 User 3 J J V 4 a V User 4 V V V J V User 5 4 J V 4 z V Success Rate 100 80 60 80 0 100 Y successfully completed not completed help needed for completion 12 3 1 2 Ease of Task Ratings After each task the participants rated the ease of completing the task by answering the Single Ease Question SEQ Overall this task was on a 7 point rating scale ranging from Very Difficult 1 to Very Easy 7 Table 10 shows the ratings given to the single tasks by the test users and provides an average of these ratings for each task Table 26 Ease of Task Ratings Participant Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 User 1 7 2 2 6 2 7 User 2 6 3 4 1 1 7 User 3 7 5 4 6 5 7 User 4 7 6 2 7 1 7 User 5 7 5 5 7 4 7 Average Rating 5 1 3 1 2 9 3 5 1 8 5 3 Percent Easy Rating 6 or 7 100 17 17 67 0 100 12 Jeff Sauro 10 Things to know about the Single Ease Question SEQ online at https www measuringu com blog sed10 php visited 12 06 2015 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 114 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 12 3 1 3 Time on Task TOT The variations in time
70. 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 12 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report Requirement Explanation Regarding Partial Fulfilment User Experience Behaviour System reaction to context To a particular degree the system recognizes which information is relevant to the user and services are enabled to forward information to the end user App However the Apps are responsible of showing such notifications needed for end users U27 Proactive behaviour Marketplace This is in fact executed by the PMA not the marketplace The App download Into the marketplace sends a push message to notify a request for a new U69 gece installation U70 App installation This is in fact executed by the PMA not the marketplace U71 App uninstallation This is in fact executed by the PMA not the marketplace Security ues Third pariy 20280810 the aadar e renan ee cara sas I mac i ed for simplicity reason Access to components Full Java libraries are provided for PMA side of the App For MMDI U105 Access tothe dialog a rind Aii oe ae Br due to system restrictions when creating projects on IntelliJ that only allow to select one SDK Business model Sales Only number of installations available inf
71. ASK 2 Find the created Backend Service in the Service Marketplace The URL of the Service Marketplace is http simpli city eu 3030 The username for the Service Marketplace is testuser The password for the Service Marketplace is testuser123 Tell the moderator when you finish this task Then answer the following short question Overall this task was Very Difficult Very Easy TASK 3 Delete created service 1 Use SIMPLI CITY menu item in IntelliJ Idea for that 2 Verify in Service Marketplace that your service is gone Tell the moderator when you finish this task Then answer the following short question Overall this task was Very Difficult Very Easy u u u u u u u Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 58 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 7 5 Transcription of the Debriefing Interviews Figure 7 In the debriefing interview the test users explained what they liked at the SIMPLI CITY system and what they perceive to be the main issues that should be improved Test user User 0 Dry run Moderator How was it User 0 Well it was fine The first task was kind of puzzling and it was kind of complicated but then it was easy In general it was fine Moderator
72. Back Express 5 5 8 0 3644 was used for screen recording A Logitech 920 HD Pro webcam and the related software were utilised for general video and audio recording of the session especially for recording of the feedback interview 7 1 5 Usability Evaluation Sessions The Usability Evaluation Sessions took place at the Technische Universitat Wien Vienna Austria The pre test was done on Tuesday 23 06 2015 and 6 usability sessions were conducted on Wednesday 24 6 2015 The average duration of the Usability Evaluation Sessions was 46 minutes with the longest session lasting for 61 minute and the shortest session taking 36 minutes All Usability Evaluation Sessions followed the same process First the moderator welcomed the test user introduced the team and explained the purpose of the session The moderator gave an overview of the test procedure and asked the test user for her his permission that the session can be recorded video audio and screen recording and these records can be used for later analysis and reporting purposes After the test user signed the consent form the moderator encouraged the test user to ask questions whenever anything is unclear In a short introductory interview the moderator obtained from the test user some demographic information gender and age and some information regarding relevant skills and knowledge of the test user such as e g familiarity with Windows Android Java and IntelliJ IDEA knowledge
73. Cit yt t 1 T 9 eph 8 mg public String getCityflame return This is Arbitrary test ArbtranSeraceimet E a Arbitrarylctrator hs public String belloWord s bs z return prin is b 9 verty D AsdararySerica a Persist Service bundle ES nat Ci resources O ie D test public void deplo Duge AR bityary 1 0 SNAPSHOT jar Select JAR C Plugs Dose 7 By ean gt O genented iources Marsfest kendService maniestam Select marstest amp compier D mavencstatus Sy deploy Ill Aritrary 1 0 SNAPSHOT yar paste nde D e mandestum 2 mM pomami Bunde mlermation jar Bid dies Service ID SbGed45 45ce MBc b te 35e Scc CIT DI teeBleckendierice mi Bundles 22 4 Ames Vh External Libranes bot als mU nin m HE a p ei shed with ex oe Hte wea Pgh TODO E Maven projects need to be imported import Changes Enable Auto import 7 minutes ago Figure 4 Overloaded Plugin Interface during Task 1 see Issue 2 1 8 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 54 146 Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 http www simpli city eu SIMPLI CITY WP7 Public D7 3 Evaluation Report SIMPLI CITY Service Marketplace x 3 mero ew 62 simplicity 7 E Hon 0 results found l SMEO Senn Qe Temene auem SMPUCTIY Ev 0 Marsa gil en testactendser i SMPU CITY Deve EN PX Ns Figu
74. Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 7 4 Task Sheets for Usability Evaluation of SIMPLI CITY s Developer Tools TASK 0 Install the SIMPLI CITY plugin for IntelliJ IDEA 1 Locate simpli city services plugin zip and Manual pdf on the desktop in SIMPLI CITY Evaluation folder 2 Open Manual pdf page 17 18 Section 4 2 and follow the instructions a Hint The link to IntelliJ Idea folder is provided in SIMPLI CITY Evaluation folder as well 3 Start IntelliJ Idea and click Create New Project there should be a SIMPLI CITY project category Tell the moderator when you finish this task Then answer the following short question Overall this task was Very Difficult Very Easy 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 56 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report TASK 1 Try to develop a Hello World Service by using the SIMPLI CITY system The URL of the Service Registry is http simpli city eu 8888 cxf serviceRegistry 1 Start IntelliJ Idea 2 Create a new SIMPLI CITY Backend Service project groupld Artifactld ProjectName can be an almost arbitrary string project location should be workspace SIMPLI CITY lt you
75. D7 3 Evaluation Report 8 1 4 Usability Evaluation Sessions The Usability Evaluation Sessions took place at Worldline Avinguda Diagonal 200 Barcelona Spain The 4 usability sessions were conducted on Friday 17 07 2015 and on Monday 20 07 2015 The average duration of the Usability Evaluation Sessions was 45 minutes All Usability Evaluation Sessions followed the same process First the moderator welcomed the test user introduced the team and explained the purpose of the session The moderator gave an overview of the test procedure and asked the test user for her his permission that the session can be recorded screen recording and these records can be used for later analysis and reporting purposes After the test user signed the consent form the moderator encouraged the test user to ask questions whenever anything is unclear In a short introductory interview the moderator obtained from the test user some demographic information gender and age and some information regarding relevant skills and knowledge of the test user such as e g familiarity with App Stores Android Smartphones knowledge of SIMPLI CITY and expertise with Thinking Aloud tests After this introductory part the moderator explained to the test user how the Thinking Aloud method works and asked the test user to do one test task after the other While the test user was working on the task she he spoke out loud all her his thoughts and considerations t
76. D7 3_Evaluation_Report_v1 0_For_Approval docx 2015 10 30 Status For Approval 81 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Action Sequence for Yes Success Failure Story Comment Idea for Improvement Completion of Task 1 Write Maven options User will know what to X Since the majority of Add tooltips to the Maven option fields explaining i e group ID artefact do at this step modern projects use what they are However for an experienced ID and version Maven for project developer the fields are self explanatory dependency control and building developers are likely to be already familiar Add F1 or the Help button which would invoke help describing these options with Maven User will know that this X was the right action Write SIMPLI CITY User will know what to X Developers should already options and choose do at this step have some SIMPLI CITY location of the project development background i e know how SIMPLI CITY works how it is organised and how to develop services and apps within its framework User will know that this X was the right action Document Date i Page D7 3_Evaluation_Report_v1 0_For_Approval docx 2015 10 30 Status For Approval 82 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights
77. For these requirements Table 2 provides explanations regarding the state of fulfilment as stated by the SIMPLI CITY developers Of the 38 Should Have requirements defined in deliverable D2 3 only one requirement namely U188 Composition of services was not fulfilled According to the developers this requirement was not covered within the final prototype since the composition of services happened in apps not on the level of services Further 10 of the Should Have requirements were fulfilled partially For these requirements Table 3 provides explanations regarding the state of fulfilment as stated by the SIMPLI CITY developers Table 2 Partially fulfilled Requirements with Priority Must Have Requirement Explanation Regarding Partial Fulfilment User goals Tactical SIMPLI CITY allows balancing the traffic in a particular area by sophisticated traffic management For this the PMA acts as both a data source for the traffic manager as well as the end user device Help to balance the traffic telling the driver how to behave However SIMPLI CITY was not a vis in a particular area traffic management project Hence the focus was only partially on this requirement However an arbitrary traffic management algorithm could be integrated into the SIMPLI CITY Mobility Services Framework and used in the PMA U16 Frovide ordered driving In Same explanation as for requirement U15 the city
78. I CITY WP7 Public D7 3 Evaluation Report 1 Introduction SIMPLI CITY The Road User Information System of the Future was a project funded by the Seventh Framework Programme of the European Commission under Grant Agreement No 318201 Its results provide the technological foundation for bringing the App Revolution to road users by facilitating data integration service development and end user interaction This deliverable describes the approach taken regarding evaluation in WP7 It outlines the overall evaluation strategy gives an overview of the evaluation activities undertaken and describes the results of all these evaluation activities in detail 1 1 SIMPLI CITY Project Overview Analogously to the App Revolution SIMPLI CITY adds a software layer to the hardware driven product mobility SIMPLI CITY takes advantage of the great success of mobile apps that are currently being provided for systems such as Android iOS or Windows Phone These apps have created new opportunities and even business models by making it possible for developers to produce new apps on top of the mobile device infrastructure Many of the most advanced and innovative apps have been developed by players formerly not involved in the mobile software market Hence SIMPLI CITY supports third party developers to efficiently realise and sell their mobility related service and app ideas by a range of methods and tools including the Mobility Services and App Ma
79. ITY project evaluation was assigned to the two tasks T7 3 and T8 3 responsible for the evaluation of Use Case and Use Case II related parts of the SIMPLI CITY system respectively While the evaluation of the Use Case and Use Case II related SIMPLI CITY Apps could easily be assigned to the relevant work packages the evaluation activities related to the other parts of the SIMPLI CITY system were relevant for both Use Cases Therefore the evaluation of SIMPLI CITY s market places and SIMPLI CITY s developer tools as well as the Technical Evaluation of the whole SIMPLI CITY system were done in close cooperation of both evaluation tasks T7 3 and T8 3 This document includes a detailed description and the results of the common evaluation and the evaluation of the Use Case related SIMPLI CITY App The Technical Evaluation of the SIMPLI CITY prototypes showed that all 115 Must Have requirements which were defined during the specification phase at the beginning of the project were fulfilled whereby over 86 of these requirements were completely fulfilled In addition of the 38 Should Have requirements about 74 were completely fulfilled The Usability Evaluation was seen as formative evaluation in order to allow the SIMPLI CITY developers to implement further improvements to the system based on the findings of the evaluation Usability Evaluation was conducted in phases starting from June 2015 to September 2015 For all component
80. TY App Marketplace which they ve liked 1 the log in page is good User 2 Ya at the beginning it was in principle fine the log in was fine Also the log out was fine User 5 So first of all the SIMPLI CITY logo is great and also the log in page is o k So the first catch is o k 2 itis good to find the list of apps at the first page where you enter the market place User 8 It s very simple to find the apps because you just see them immediately after log in 13 Adjective ratings related to SUS scores 25 worst imaginable 38 poor 52 acceptable o k 74 good 85 Excellent 100 best imaginable Source Bangor A Kortum P Miller J Determining What Individual SUS Scores Mean Adding an Adjective Rating Scale in Journal of Usability Studies JUS Vol 4 Issue 3 May 2009 pp 114 123 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 115 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report User 1 The overview of the app list is actually good you see it directly when you start on the front page when you have entered in You have entries that tell you if there is a discussion on an app or an app itself offered This is quite fine 12 3 3 Main Usability Issues Revealed During the Evaluation 12
81. TY system which should be improved Create App button functionality and position Confusion between Add binary button and Upload binary button The place of the Submit button on the publication of an application The drag and drop in Tests Management tab is not intuitive Differences between test and group test are not stated clear Publish button was hard to find After the Usability Evaluation the results of the evaluation sessions were communicated to SIMPLI CITY s developers During the remaining time of the project their work is focused on solving these issues and improving the developed software gods Is 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 67 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 8 3 Detailed Documentation of the Findings 8 3 1 Usability Metrics 8 3 1 1 Task Completion Success Rate Overall performance of each participant was evaluated depending on his her ability to finish the task on his her own The task success rate score is the number of participants that finished the task successfully divided by the total number of participants We can see that Task 2 Task 3 and Task 4 have the highest completion rate due to the simplicity of the tasks and the familiarity with the evaluated system Task 0 had
82. U193 information from X bn ave apps to server Exchange of U194 information from X s ave server to apps U195 Proactive user x Personal Mobility Assistant PMA can Must notifications show notifications when asked by Apps Have This functionality has not been covered within T5 2 as prioritization of U196 Prioritization of T notifications should be done on auser s Must notifications Personal Mobility Assistant PMA PMA Have can show notifications when asked by Apps Will Not U197 db Ac Hom X Have y For Now Cross cutting concerns Small number of Must Utas device to support Have Compliance to Should Age regulations x Have Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 39 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report Table 6 Use Cases Requirements tnise Explanation why not or only partially Requirement implemented Priority yes part no Generic Provision of The info is personalized based on user i ds Must U201 personalized info x context on some factors position Have e g travel costs sensors etc IBM Scenario Use Case Topic 1 1 Diagnosis of U202 abnormal traffic X Muer R Have condition in real time Prediction of U203 abnormal
83. U210 Reproduction of Only images of the route and audio feedback are provided as other multimedia information streams are not supported Notification to end user tn ea No updates are sent to the PMA as this is not supported by PMA U212 about the proximity of 3 User needs to request updates actively Points of Interest Table 3 Partially fulfilled Requirements with Priority Should Have Requirement Explanation Regarding Partial Fulfilment User Experience UI and usability of Apps In most cases the response from the system is quick Also the Reasonable response microphone icon in the PMA shows that the system is listening to the U18 p user s voice However some visual feedback for showing that the time Me System is processing information e g via the cloud could help in cases where such processing is slow User Experience Look amp Feel of Apps During the project it turned out that design changes should be in the hand of the corresponding app developer in order to be successful in the market The reason is twofold Firstly app developers need to integrate their apps into their companywide look amp feel and secondly U23 Unified Look amp Feel apps need to have the possibility to react to new trends without being within the project bound to a UI restriction of the underlying framework For example Google has released the Material Design guideline which quickly led to several changes for most apps As such SIMPLI CITY recommends
84. User 3 Yes the mixture of the items so that you first have to select to publish so that you have a distinct menu for publishing and also included the deleting functionality So that should be in a separate menu entry That would be more clear to say manage the market place services or manage the published services so that you can go to your service and say that you want to delete it or suspend it User 8 It s really interesting and the system might be good to use once the manual is a bit more elaborated For a moment I was kind of puzzled when saw the dummy text in one of the chapters and thought that I ve seen this lorem ipsum text before test user laughs Translation from the German part of the discussion after the official end of the session Test user User 4 Test Session 3 Moderator First question How was it User 4 Rather easy besides some minor features like the counter intuitive search interface because normally you would expect that it works case insensitive But nevertheless had no All task were pretty easy Moderator Was there something especially good User 4 It simply worked as intended I guess so there is nothing pretty awesome and nothing really bad besides the search interface but nevertheless it worked o k Moderator Any other feedback from your side User 4 Actually the interface as I ve seen so far if would develop an applicat
85. ace services or manage the published services so that you can go to your service and say that you want to delete it or suspend it 2 the guidance material needs improvement User 6 The help how to bundle the service how to upload the service could be improved First did not get that had to push that bundle it button Maybe that could be a little bit more clearly mentioned or there could be another structure of the help 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 50 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report User 3 the system might be good to use once the manual is a bit more elaborated For a moment was kind of puzzled when saw the dummy text in one of the chapters User 1 The documentation was not 100 intuitive and without Maven background and without IntelliJ background which had before assume that would have had more difficulties in locating specific elements and packaging the bundle But this cannot be avoided I think because it s just so that if you are familiar with a tool it s good but if you re not you have to get used to it if you want to work with it So you cannot really get around this but the documentation could be better formulated and more intuitively designed 3 t
86. apter 5 1 and related chapters 11 12 V0 5 Michaela Kargl FGM 30 09 2015 added chapter 5 2 and related chapters 13 14 as well as the summary Contributions V1 0 Michaela Kargl FGM 15 10 2015 updated according to internal reviewers comments Final Version October 30 2015 M37 Note This deliverable is subject to final acceptance by the European Commission Disclaimer The views represented in this document only reflect the views of the authors and not the views of the European Union The European Union is not liable for any use that may be made of the information contained in this document Furthermore the information is provided as is and no guarantee or warranty is given that the information is fit for any particular purpose The user of the information uses it at its sole risk and liability 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 2 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Project Partners TU WIEN TECHNISCHE UNIVERSITAT WIEN Vienna University of Technology Vienna University of Technology Coordinator Austria TIE KINETIX total integrated e commerce TIE Nederland B V The Netherlands IBM Research Ireland Smarter Cities Technology Centre Talkamatic FREE DIALOGUE Talkamatic AB Sweden CENTRO
87. ation of the Increased Mobility app All of them answered the introductory questionnaire did the expected 4 test tasks and provided their feedback in the debriefing interview and feedback questionnaire Three out of the four test users were men and were working in SRM 2 employees 1 consultant while the fourth was a woman working as employee in the Municipality of Bologna All of them were between 20 and 50 years old None of the test users was directly involved in the development of the Increased Mobility app but some of them have been involved in the SIMPLI CITY project So they knew the P Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 137 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report main aim of the project and also the expected final outputs even if they were not familiar with the application itself Table 30 gives an overview of the relevant skills and knowledge of the test users as stated by them in the introduction questionnaire Table 30 Overview of the Test Users relevant Skills and Knowledge User 1 User 2 User 3 User 4 Test user MA EM MB Mo Usability Evaluation Session Session 1 Session 2 Session 3 Session 4 Gender M M M F Age 31 40 31 40 41 50 20 30 Glasses Pecu
88. ave Interaction with car Must U85 sensors e g car X Have sensors Privacy Must U86 Transparency X Havo Confidentiality Does Must U87 not give away data to X Have third parties Must U88 Data encryption X Have Certification Only U89 certified apps are Must allowed to access Have users data Security ORA Must U90 Availability X Have Must U91 Integrity X Have Secure access to Must uag system 2 Have Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 30 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report fulfilled Explanation why not or only partially implemented Requirement Priority yes part no Usage of https has been foreseen at the Third party access to design phase and the platform is ready Must ued the system x to use it Currently the plain http protocol Have is used for simplicity reason Table 5 Developer Requirements fulfilled Explanation why not or only partially ND Requirement implemented Priority yes part no Backwards compatibility Some checks on the version and classes used by the App are done U100 Backwards x during the installation process to prevent Should compatibili
89. bility Inspection revealed also some positive findings Table 21 in Section 11 3 presents all positive findings The main positive aspects noted by the evaluators were e The graphic approach and the navigation are good and easy e The search function is case insensitive and works also with fragments of words 11 3 Detailed Documentation of the Findings The following Table 21 lists all positive findings that were observed by the evaluators during the Heuristic Evaluation Table 21 Positive Findings Reported by Description of the Observed Positive Findings Evaluator GL MK DW In general the graphic approach and the navigation are good and easy Login works smoothly When clicking the Install or the Search button the user receives visual feedback from the button that clicking was successful button colour changes Search function is case insensitive and works also with fragments of words Download of file under the menu Documentation works Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 105 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Reported by Description of the Observed Positive Findings Evaluator MB GL MK DW Step back from App to Marketplace works but only once When entering anot
90. c d gT The Road User Information System Of The Future WP7 Use Case Meeting the Increased Mobility Demand D7 3 Evaluation Report Document Lead FGM Contributing Partners TUV ASC TIE TUDA IBMIL TALK WORLD SRM CRF Delivery Date 10 2015 Dissemination Level Public Version 1 0 Short Abstract Teaser This document provides an overview of the evaluation approach taken within SIMPLI CITY and describes the results of the WP7 related evaluation activities conducted within the project This project has received funding from the European Union s Seventh Framework Programme for research technological development and demonstration under grant agreement no 318201 SIMPLI CITY WP7 Public Document Status Deliverable Lead Michaela Kargl FGM Internal Reviewer 1 Philipp Hoenisch TUV Internal Reviewer 2 Daniel Burgstahler TUDA Type Public Deliverable Work Package WP7 Use Case Meeting the Increased Mobility Demand D7 3 Evaluation Report ID D7 3 Evaluation Report Due Date 30 09 2015 Delivery Date 09 11 2015 Status For Approval Document History V0 1 Michaela Kargl FGM 02 09 2015 Document structure V0 2 Michaela Kargl FGM 07 09 2015 added chapter 2 chapter 3 and related chapter 6 V0 3 Markus Garger FGM 08 09 2015 added chapter 4 and related chapters 7 8 9 10 V0 4 Michaela Kargl FGM 15 09 2015 added ch
91. city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 9 3 2 Task 2 Develop a Service Using the SIMPLI CITY System Action sequence for Yes Success Failure Story Comment Idea for Improvement completion of Task 2 Install the Service User will know what to X Installing IDE plugins as While the Service Design Studio can be easily Design Studio in IntelliJ do at this step described is common installed from the supplied zip archive it is worth IDEA following the practice checking if the plugin itself can be uploaded to the install instructions in main IntelliJ plugin repository where third party the provided ZIP User will know that this x plugins are located http plugins jetbrains com was the right action then installation would be even easier Open IntelliJ IDEA User will know what to x Developers start their Rearrange menu system of the IDE put the do at this step IDEs daily SIMPLI CITY menu item before the Help menu item User will know that this X was the right action Create a new SIMPLI User will know what to x This is a usual step when CITY Service project do at this step creating a new project following the New independently of the Project wizard under User will know that this X project nature File New Project was the right action menu option Choose between User will k
92. cked on the short app description in order to find out more details and as expected they were transferred to the page with the more detailed app description However since there was not additional information available on this page the test users were disappointed See screenshot Figure 20 The test users stated that they would have expected to find a more detailed description of the apps functionality screenshots of the app price information etc Task 5 Install any of the Apps on your Smartphone all test users first tried to find the app on the Google Play Store only after a hint from the moderator they tried to install the app directly from the SIMPLI CITY App Marketplace 4 out of 5 test users immediately after reading the task description took the smartphone and tried to do this task on the smartphone since they know from their experience that installing apps on a smartphone is done by accessing an app store with the respective smartphone only one of the test users knew that it is also possible to install apps on the smartphone by using the PC laptop all test users found the install button easily However after pressing the button they were puzzled as a pop up window opened telling App Installation requested and they had to press o k they were completely confused and not sure what to do now they did not know whether the app is now installed on the smartphone or whether they need to
93. covered during the evaluation sequence In general the evaluated developer plugin was accepted quite well by every user However due to different levels of experiences and background knowledge from the users such as a non existing familiarity with Windows 8 IntelliJ the used developer IDE and the pre installed unpack tool we noticed high variations of the users task completion times For example in the task 0 in which the users had to install a plugin for the IDE some users had major problems as they were simply not familiar with the security settings of Windows 8 i e it is not allowed to unpack a zip file directly into the program installation folder The average task completion time or Time on Task TOT for the task O was about 8 minutes with a standard deviation of 3 3 Further while some users were able to complete that task on their own 5796 others needed minor help from the technician The first task was to develop a simple hello world method integrate it into a backend service and deploy that service in the SRE We expected a task completion time of maximal 20 minutes However 2 out of the 7 users took longer than 20 minutes Even though most of the users claimed to have high expertise in software development only 4396 of the users were able to finish this task on their own Again this can be reduced to the fact that the users were not familiar with the used IDE or SIMPLI CITY software that they had to use The average c
94. d Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report User 2 Well especially good is to have a server side for publishing the service that you develop and then it make the cycle for the development and publication starting from scratch You just need a couple of installations stuff and then can work with this That is a good point the way to interact with developer and aligned it Moderator What do you think is the worst thing User 2 The worst thing is l m not sure maybe what s not good is stick to the IDEA The first difficulty is using the IntelliJ used it before several times and I do not like it So you need to support more than just one IDE And also this junction function when you generate a project can be richer like provide more functionality mean if want to have a more complex application what shall do You need more detail Moderator So what would you expect User 2 Well it should provide more specific things like if want an application for GPS should have a generated code interact with the Google API from someone to generate a map So that s what want So inside of the plugin you should have some key to generate code for user for developer that is the user of this system Something like this The system should support more than just a skeleton because I m the developer know how to write an interface and know how to write the implementation and
95. d not actually understand what it s for o k there is an App Marketplace and I can deploy services in there That s all the information that get of what the actual value is of this plugin and of the service platform Moderator Would you say that there is something that is really good User 7 It s easy The buttons the menu is very structured and the icon integration is good so that the icons are prominent and you can see the new plugin and the functionality of the plugin The deployment process itself could probably be simplified cause basically the process is create a jar file and deploy it there s no necessity to have this step in between that you have to compile it with Maven first and then deploy it you could just have a button deploy my thing and the process does it for you But yes like the the Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 62 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report IntelliJ integration was good But the value seems small it did not appear that there was any clear value for me the system has to deliver Moderator Was there anything that you would rate as especially bad User 7 The deletion process of the service was hm think there could be some menu integration for the popup fo
96. d organize them as memory of the just ended trial was vivid Furthermore the recordings were saved and the material and setup was prepared for the next Session 14 2 Main Issues Revealed During the Usability Evaluation 14 2 1 Issues Noted by the Facilitator The Usability Evaluation of the Increased Mobility app demonstrated a confusion and low confidence by test users while dealing with some tasks despite the fact that the tasks were substantially basic That was specifically noted by the facilitator during the tasks 1 and 2 but also task 3 raised some usability problems In general often test users were not sure that their input was really accepted by the application being missing a confirmation or a feedback by the application itself Furthermore in some sub menus it was not possible to return back to the previous menu and test users were annoyed when forced to close the application and restart it again in order to go ahead the task In task 1 the test users were not sure where the requested data was expected to be entered They proceeded by chance and felt in wrong paths and some test users did not succeed at all in accomplishing the task Maybe it would be more convenient to have only one section in which the user is allowed to enter user data rather than two as it was in the tested version of the application Also in task 2 test users needed some trials even those who successfully completed the task Some test users did
97. debriefing interview at the end of the Usability Evaluation Sessions the test users stressed two features of the SIMPLI CITY system which they ve especially liked e The good integration with IntelliJ IDEA e The easy way of publishing services in the Service Marketplace However the test users mentioned also some aspects of the SIMPLI CITY system which should be improved e The developer guidance materials shipped along with developed software have to be improved e The provided user interface should be made more intuitive e The software development sequence needs to be simplified 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 16 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 4 1 2 Results of the Usability Evaluation of the Developers Part of the SIMPLI CITY App Marketplace The Usability Evaluation Sessions for the part of the SIMPLI CITY App Marketplace which is dedicated to app developers took place at Worldline s office in Barcelona Spain The 4 usability sessions were conducted on Friday 17 07 2015 and on Monday 20 07 2015 The usability evaluation demonstrated that a developer can publish an application successfully with a minimum of effort and knowledge Nevertheless a number of important usability issues were discovered dur
98. ded by SIMPLI CITY personnel on request Search for an existing User will know what to x Searching by going through service do at this step the list of services or using a search field is common in modern web sites User will know that this x was the right action Document Date i Page D7 3_Evaluation_Report_v1 0_For_Approval docx 2015 10 30 Status For Approval 89 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report connect to real ones was the right action Action sequence for Yes Success Failure Story Comment Idea for Improvement completion of Task 3 Look for the ID of the User will know what to x Developers should already service and the list of do at this step have some SIMPLI CITY methods available development background i e User will know that this x know how SIMPLI CITY works was the right action how it is organised and how to develop services and apps within its framework Open IntelliJ IDEA User will know what to x Developers start their IDEs do at this step daily User will know that this x was the right action Open an existing User will know what to x Developers should already SIMPLI CITY App do at this step have some SIMPLI CITY project or create a new development background i e one User will know t
99. ding to Bangor A et al this SUS score can be translated to users perceiving that the usability of the system is acceptable o k This is quite good for a prototype system but indicates that there s still some improvement of the system s usability necessary 7 3 2 Main Positive Aspects Mentioned by the Test Users During the debriefing interview at the end of the Usability Evaluation Sessions the test users stressed two features of the SIMPLI CITY system which they ve especially liked 1 the good integration with IntelliJ IDEA User 3 I liked the service creation in IntelliJ with a separate plugin and menu structure so that you can access and find the needed methods in an easy way User 7 It s easy The buttons the menu is very structured and the icon integration is good so that the icons are prominent and you can see the new plugin and the functionality of the plugin 3 According to User 0 Brooke SUS A quick and dirty usability scale http hell meiert org core pdf sus pdf According to User 0 Brooke SUS A quick and dirty usability scale http hell meiert org core pdf sus pdf To calculate the SUS score first sum the score contributions from each item Each item s score contribution will range from 0 to 4 For items 1 3 5 7 and 9 the score contribution is the scale position minus 1 For items 2 4 6 8 and 10 the contribution is 5 minus the scale position Multiply the sum of the sc
100. discussion process are recorded in order to help improving the system 9 1 3 Team of Evaluators Table 17 Team of Evaluators for Usability Inspection from Developers Point of View Name Organisation Professional Background David Turon Bes TIE Experienced software developer with background in Java and Scala Fran Rodriguez Montero TIE Experienced software developer with background in Java and Net Arturo Brotons Cartagena TIE Experienced Java developer Vadim Petrenko TIE Senior software developer with background in Java 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 78 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 9 1 4 Usability Inspection Sessions The usability inspection session took place on the 26 of August 2015 at the TIE headquarters Taking about one hour there was a plenty of opportunity to discuss different aspects of the development package Evaluators were given an opportunity to speak and express their opinions which were recorded for further analysis 9 1 5 Context Information Product The SIMPLI CITY developer package which was inspected during this Expert Evaluation included the following components Application Design Studio Service Development API Service Marketplace Application Mark
101. dy but only one of these in a Thinking Aloud Test 8 1 6 Evaluation Tasks The test users were asked to complete the following tasks e Task 0 Create an application upload the binary and media needed into the SIMPLI CITY Application Marketplace and submit for the approval role developer e Task 1 Create a new set of tests and assign it to the created application in Task 0 assign it to a tester and submit for testing role approver e Task 2 Perform the tests created in Task 1 and submit the results role tester e Task 3 Check result of the testing and approve the application for publication role approver 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 66 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report e Task 4 Publish the application role developer Each of the tasks was explained just before it was performed in order to ensure that the test users have a clear idea on what was expected from them 8 2 Summary of the Results of the Usability Evaluation The usability evaluation demonstrated that a developer can publish an application successfully with a minimum of effort and knowledge Nevertheless a number of important usability issues were discovered during the evaluation sequence In general the evaluated Application Mark
102. e would expect a bit of explanation on the first page what is the website about or something There was just a list of apps but if you actually want to search for an app it s quite easy because you see the list at once and do not need a lot of time But maybe a sentence or two at the beginning would be useful for people who have not visited the web page before Moderator Was there anything that you found really bad User 3 Really bad is that it is not adjusted for smartphone Because if you download apps for smartphones also the webpage just be adjusted for smartphone was irritated with sweeping left and right This was very bad Moderator s there anything else that you would improve if you would be the developer of this stuff User 3 Well it s white and a little bit blue Maybe it could be a bit more colourful know that has not influence on how it works but it could also look better Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 124 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Moderator Anything else User 3 Well know that this is not done yet But with installation was not sure if it is installed or not on the phone this is confusing was not sure if should do anything else after pressing the
103. e aspects noted by the evaluators were e The graphic approach and the navigation were good and easy e The search function was case insensitive and worked also with fragments of words 5 1 2 Results of the Usability Evaluation by Test Users The Usability Evaluation Sessions for the SIMPLI CITY App Marketplace from road users point of view took place at FGM s premises in Graz Austria The pre test was done on Friday 4 9 2015 and 5 usability sessions were conducted on Wednesday 9 9 2015 During the debriefing interview at the end of the Usability Evaluation Sessions the test users stressed two features of the SIMPLI CITY system which they ve especially liked e The website is very clear and simple e The users liked the design and the logo of the App marketplace However the test users mentioned also some aspects of the SIMPLI CITY App Marketplace which should be improved e There should be some introductory text at the top of the page describing what this website is about some sort of Welcome text which explains to the user the features of the marketplace e The search function should be better not only full text search of the app descriptions but also finding suitable apps when typing in more general keywords such as e g navigation or routing e There should be used graphics and not only text in the list of the Apps e There should be more information about costs and size of the Apps e The website should be op
104. e above The evaluators were advised to work alone r Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 102 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report without communicating with the other evaluators and delivered a list of their positive findings as well as a list of the potential usability issues they have revealed The evaluators individual lists were consolidated and combined into the heuristic evaluation report which can be found in Section 11 3 of this document 11 1 5 List of Heuristics The list of heuristic usability principles is at the core of the Heuristic Evaluation method For the Heuristic Evaluation of the SIMPLI CITY App the evaluators used the following set of 12 usability principles as reference This list of Usability Heuristics for touchscreen based mobile devices with voice based user interfaces is compiled based on Jakob Nielsen s 10 general principles for interaction design the Usability Heuristics for Touchscreen based Mobile Devices the Usability Checklist for Voice based User Interfaces and Gerry Gaffney s Voice Interaction Checklist Table 20 List of Usability Heuristics Usability Heuristics for Touchscreen based Mobile Devices with Voice based User Interfaces 1 Visibility of syste
105. e anything in the system that you found to be especially good User 6 Yes found it good that IntelliJ automatically uploads the service into the marketplace and that there are no extra tasks that have to be done So like it when everything is done in one programme when there s only a button and it works Moderator Was there anything especially bad User 6 think the view could be a little bit over full if there are many apps in there Moderator So you mean the web view of the Service Marketplace User 6 Yes So this could be a little bit improved with an extra view Moderator Anything else that you would improve if you would be the developer of this stuff User 6 Yes the help how to bundle the service how to upload the service First did not get that had to push that bundle it button Maybe that could be a little bit more clearly mentioned or there could be another structure of the help Test user User 7 Test Session 6 Moderator ve got a few questions How was it User 7 Well in such a usability evaluation you concentrate also on what you say so you concentrate on various tasks simultaneously such as understanding the technology speaking out loud so you sometimes can t actually grasp what your task is You rather concentrate on the process than on the actual result And as far as the product goes well saw that there is a plugin that extends IntelliJ but di
106. e in developing Android Apps or Services http stackoverflow com research developer survey 2015 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 44 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report None of the test users was involved in the development of the SIMPLI CITY developer package One of the test users was working for other parts of SIMPLI CITY three of the test users had heard of SIMPLI CITY before and the remaining three test users did not know anything about SIMPLI CITY Two of the test users had participated in a Thinking Aloud Test before 7 1 7 Evaluation Tasks The test users were asked to complete the following tasks Task 0 Install the SIMPLI CITY plugin for IntelliJ IDEA Task 1 Try to develop a Hello World Service by using the SIMPLI CITY system Task 2 Find the created Backend Service in the Service Marketplace Task 3 Delete the created service Annex A of this document includes the task sheets which the test users got during the usability evaluation session 7 2 Summary of the Results of the Usability Evaluation The usability evaluation demonstrated that the developed tools already provide a reasonable support for SIMPLI CITY software developers Nevertheless a number of important usability issues were dis
107. ea of cooperative working developer tester and approver all working side by side dividing the work If that does not count I ll keep the fact that when you as a developer are creating a new app all of the fields you have to fill are visible at the same time without any need to scroll down creating a feeling that it s all compact all important all easy Moderator And what was the worst thing User 2 It may be deuced between the previously mentioned messy aspect of the test creation task and the position of the publish app button inside the binaries tab in developer role T Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 76 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report Aside from this had the overall feeling that wouldn t have known had already finished the task if it was not for external help lacked some operation succeeded pop ups for example User 3 Test Session 3 Moderator How was it User 3 The product is very good it has a simple user interface and is easy to use Still needs to be improved but it could be very competitive Moderator What was the best thing User 3 The user interface it was very intuitive Moderator And what was the worst thing User 3 The wo
108. ed Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report iunea Explanation why not or only partially Requirement implemented Priority yes part no During the project it turned out that design changes should be in the hand of the corresponding app developer in order to be successful in the market The reason is twofold Firstly app developers need to integrate their apps into their company wide look amp feel and secondly apps need to have the Unified Look amp Feel for or Should U24 X possibility to react to new trends without third port developers being bound to a UI restriction of the nave underlying framework For example Google has released the Material Design guideline which quickly led to several changes for most apps As such SIMPLI CITY recommends the use of the Material Design but does not inforce a unified look amp feel U25 Usage of UI guidelines Should within the project Have ur a Should U26 Intuitive Usability X Hav User Experience Behaviour System reaction to context To a particular degree the system recognizes which information is relevant to the user and services are enabled to Must U27 Proactive behaviour x forward information to the end user app H ave However the Apps are responsible of showing such notifications needed for end users Will Not T eee x Have For Now U29 d e i M The current use cases are independent Could i p from the
109. ed that this is rather unusual Task 3 Search for Apps that support you with navigation and help you to find the best route for driving to any destination Hint The site offers a search function all test users typed into the search field navigation routing route but only with the word routes the respective app is found as the search function finds only words that are included in the description of the app alltest users pressed the enter key when they had finished the entry into the search field all test users expected the search to be triggered by pressing the enter key after some hesitation they finally found the search button and clicked on it the test users were confused when the search function had not found a corresponding app because there was only an empty screen displayed and there was no feedback from the system see screenshot Figure 19 Thus the test users did not know whether the search is still working and they need to wait a bit longer or whether there is no such app available in the marketplace Task 4 Find out more detailed information for any of the Apps you are interested in Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 117 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report all test users cli
110. ence Generic Services User centric data services e g Must u59 Facebook events x Have calendar Marketplace U60 Payment for apps and T Could services Mock up Have Payment for apps and Will Not U61 services Fully X Have deployed For Now put Must U62 Versioning of apps X Hav Must U63 Upgrading of apps X Have Quality assurance Must uos Mock up x Have U65 Quality assurance x Could Fully deployed Have Marketplace easy to Must U66 se X Have Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 28 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report nen Explanation why not or only partially Requirement implemented Priority yes part no U67 Apps and services are x Must easy to buy Have Discovery of apps and Must vee services Have This is in fact executed by the PMA not U69 App download into the the marketplace The marketplace Must device sends a push message to notify a Have request for a new installation P This is in fact executed by the PMA not Must U70 App installation X the marketplace Have This is in fact executed by the PMA not Must U71 App uninstallation X the marketplace Have Environment Device Must U72 Android support X Have
111. ent the answer of the user is Its a marketplace for App development Task 3 10 30 Test user starts with task 3 types in navigation in the search area no result types in routing in the search area no result types in navigation app in the search area no result best route no results destination no results app 3 results clicks on test app no description clicks on the fake app no description searches for app again the User can t find a navigation app and gives up Task 4 10 34 Test user starts with task 4 clicks on the first app in the list reads the description test user thinks that he s found the right page and finished the task Task 5 10 36 Test user starts with task 5 entering Play Store on the Android smartphone Searching for Simpli City a lot of different apps are listed entering Simpil City no result entering eco driving Hint from Moderator You will not find it on the play store The User is opening a browser Chrome on his smartphone and types in the address of the simpli city marketplace e he finds the login page and zooms in e types in username and password e e User tries to install the app but the button is not reacting User says it was very difficult because he did not find the app on the PlayStore 10 43 Test user starts with task 6 Types on the button logout on the desktop Done Types on the button logout on the smart
112. ep some SIMPLI CITY development project background i e know how User will know that this x SIMPLI CITY works how it is was the right action organised and how to develop services and apps within its framework Generate the bundled User will know whatto x The developer knows the main App out of the project do at this step dialog for working with bundling of using SIMPLI CITY gt apps is located here Send App to Device and then Build User will know that this x was the right action Click Open Location User will know whatto x This step is described in the and copy the full do at this step manual bundle zip file somewhere to be used User will know that this x later was the right action Document Date i Page D7 3_Evaluation_Report_v1 0_For_Approval docx 2015 10 30 Status For Approval 91 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Action sequence for Yes Success Failure Story Comment Idea for Improvement completion of Task 4 Open the web browser User will know whatto x The URL has been communicated and go to the App do at this step to the developer Marketplace web interface User will know that this x https devel1 tempos2 was the right action 1 com simpli city demo Login with your User will know whatto x Using username and
113. er 3 Really bad is that it is not adjusted for smartphone Because if you download apps for smartphones also the webpage should be adjusted for smartphone was irritated with sweeping left and right This was very bad 3 Layout of the Search function is confusing User 5 I need a better search function A search field or so with written search inside User 4 From the design you can see that it is not far in the development yet So the search field is separated from the button and so on 4 Search should work with the enter key User 1 The search option should react on the enter button guess I m not the only one who is typing and then not using the mouse to click on anything but just firing at the return key 5 Search algorithm should be improved 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 116 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report User 4 Well bad was the search You do not find anything just typed in two words from the task description And navigation or routing should find a navigation app and it did not so that was bad they have to add more search words guess So not only the description but maybe some extra words 6 Installation procedure is co
114. er goals Tactical U14 Show real time carbon Must print Have SIMPLI CITY allows to balance the traffic in a particular area by sophisticated traffic management For this the PMA acts as both a data source for the traffic manager as well as the end user device telling the driver how to U15 E CIE behave However SIMPLI CITY was not Must P a traffic management project Hence Have area i the focus was only partially on this requirement However an arbitrary traffic management algorithm could be integrated into the SIMPLI CITY Mobility Services Framework and used in the PMA Provide ordered Must U16 driving in the city x Same explanation as above Have Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 23 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report iuHied Explanation why not or only partially Requirement implemented Priority yes part no Allow local authorities Must U17 to connect easily with X Have users User Experience Ul and usability of Apps In most cases the response from the system is quick Also the microphone icon in the PMA shows that the system Reasonable response is listening to the user s voice However Should ae time 7 some visual feedback for showing that Have the s
115. es not provide any load balancing This means in the case of high load many parallel requests the SRE is in danger to overload or even crash Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 95 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Partner Opinion and Comments Application Runtime Environment ARE The ARE for deploying single Apps on the smartphone is a fully functional component It provides the means to add additional Apps in an easy way In addition its multi model user interface allows users to interact either via touch or speech with the Apps Dialogue Interface and Multi Modal UI within the Apps The Dialogue Interface within the Apps which are deployed in the ARE on the smartphone provides a novel approach for designing user interactions By simply providing an additional small backend service running on a server additional functionality can be added to any App without the need of restarting it This component is well designed and provides useful feedback in the case of errors So far this component provides the required functionality as specified within the project s specification However additional functionalities like providing speech recognition without any backend servers would be quite useful this goes beyond
116. esent Users 1 2 3 and 4 There are sections i e Emission level and Power source where the user is cannot go back to the previous section Restart of the application is necessary User 2 The system does not react to user commands User 3 User is not allowed to close the application by a dedicated feature within The application can be closed only through a forced feature of the device 2 Logic design Users 1 2 3 and 4 Settings and user data are requested in different sections not linked logically User 3 The application allows the user to insert the destination even if the destination has already been inserted previously in a different section 3 Vocal system Users 2 3 and 4 It is not clear how the user can activate the microphone to insert expected data via voice input Users 1 2 and 3 The microphone activation is not comfortable User 3 It would be useful if the microphone showed that it is in active mode and receiving sounds e g by flashing User 2 The application does not work when using only the user voice 4 User assistance Users 1 and 3 The SIMPLI CITY website is not linked within the application User 2 A search by text feature is not present in the application 14 3 Detailed Documentation of the Findings 14 3 1 Usability Metrics 14 3 1 1 Task Completion Success Rate Overall performance of each participant was evaluated depending on her his ability to finish the task on her his ow
117. ether these requirements were fulfilled by the prototypes developed within the project and furthermore the developers were asked to explain for any not completely fulfilled Must Have and Should Have requirements why it was not or only partially fulfilled The SIMPLI CITY developers stated that 143 of the 212 requirements were completely fulfilled by the prototypes developed in the project Of the 115 Must Have requirements 99 were completely fulfilled and the remaining16 were fulfilled partially Of the 38 Should Have requirements defined in deliverable D2 3 only one requirement namely U188 Composition of services was not fulfilled According to the developers this requirement was not covered within the final prototype since the composition of services happened in Apps not on the level of services Further 10 of the Should Have requirements were fulfilled only partially The following tables which include all requirements as specified in the deliverable D2 3 provide the detailed results of the Technical Evaluation of the SIMPLI CITY prototypes Table 4 End User Requirements AES Explanation why not or only partially Requirement implemented Priority yes part no Usage types U1 Support of car drivers Must as end users Have Support of cyclists as Should U2 X end users Have Support of private Should U3 transport passengers X H ave as end users Su
118. etplace Users It is expected that users of this product belong to the following groups e software developers who programme services e software developers who programme Android applications It is assumed that the users of this product have the following relevant skills knowledge and experience Java standard edition Android Experience in developing native Android mobile apps Understanding of push notifications Understanding of a client server model Ability to learn and use third party APIs Tasks The main tasks that users perform with this product are e develop an App using the SIMPLI CITY system e develop a service using the SIMPLI CITY system e utilise a service from the SIMPLI CITY Service Marketplace for creating an App e put an App on the SIMPLI CITY Application Marketplace e puta service on the SIMPLI CITY Service Marketplace Test The Usability Inspection was done in the following technical Environment environment e Evaluators used screen sharing to watch the presentation performed by Arturo Brotons in real time which he was doing using an ordinary laptop of average configuration e Development tools being evaluated were running locally on Arturo s laptop and the Service and Application marketplaces were running on Ascora s and WORLD s external development servers 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 79 146 http www simpli city eu
119. etplace was accepted quite well by every user Also we can see that the platform is more useful and easy to use on each interaction The average task completion time or Time on Task TOT for task 0 was 4 minutes and 10 seconds while the fastest participant required only 2 minutes and 30 seconds and the slowest required 6 minutes and 8 seconds Further while some users were able to complete that task on their own 50 others needed minor help from the technician In the task 1 the participants were asked to create and assign a new set of tests to the application created in the previous task On average this task was completed in 5 minutes and 47 seconds while the fastest participant needed 4 minutes and 52 seconds and the slowest required 6 minutes and 33 seconds However one of the test user needed a hint to complete the task Tasks 2 3 and 4 were completed successfully by all the test users within around 1 minute on average It can be seen that the complexity of the tasks was lower and the design the usability and the previous experiences of the users helped them with completing these tasks During the debriefing interview at the end of the Usability Evaluation Sessions the test users stressed three features of the SIMPLI CITY system which they ve especially liked 1 The single screen form to create an application 2 The simplicity to approve an application However the test users mentioned also some aspects of the SIMPLI CI
120. euristics Screenshot Severity Filename 11 Increased mobility gt Settings gt Configure the emission 1 3 2 level value gt specify the value gt euro 3 GUI No feedback is given to the input of the emission value 12 The button back in the upper menu close to the logo 1 3 4 S9 2 does not work 13 POIs button is apparently missing D7 1 2 the section S7 S8 2 concerned is 6 1 4 POI Blackspot Management 14 Anomaly information button is apparently missing S7 S8 2 15 Traffic diagnosis button is apparently missing S7 S8 2 16 When the app is open and the user does not interact for 1 3 1 a short time the app sometimes freezes and there s no possibility to go back to the menu Even the selection of one of the options displayed does not produce any effect The app must be restarted 13 2 1 Screenshots Corresponding to the Revealed Usability Issues Sp SIMPLICITY Login 130 83 245 97 p SIMPLI CITY Register as new user Simplicity Passworc Eco assistant Signin nine teilen ea quoe Begister Driving Experience Next to me Please add an account Settings Accounts SIMPLI CITY C c e utm m Figure 23 Screenshot S1 Figure 24 Screenshot S2 Figure 25 Screenshot S3 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 135 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No
121. eveloper studio SDK Use case driven development Must U159 Support for dialogue x Have Must U160 Support for apps x Have Support for data Must UTe1 services 5 Have U162 Support for backend i Must services Have Developer studio SDK Easy to use Should mies environment X Have Low investment Should ues required a Have User friendly U165 developer studio X os ave interface The Service Registry permits to store and provide information about the U166 Identification of the developer of a service However it is not Must developer signature possible to integrate with App Have Marketplace due to client restriction on API 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 36 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report tuifilled Explanation why not or only partially Requirement implemented Priority yes part no Will Not U167 Hot updates X Have For Now Developer studio SDK documentation Provision of source Must ise code examples Have Developers can create the dialogs but U169 Provision of UI T the GUI elements are provided Must templates dynamically through the dialog between Have the User and the MMDI P
122. f the evaluation sessions were communicated to SIMPLI CITY s developers During the remaining time of the project their work was focused on solving these issues and improving the developed software 7 3 Detailed Documentation of the Findings 7 3 1 Usability Metrics 7 3 4 1 Task Completion Success Rate Overall performance of each participant was evaluated depending on his her ability to finish the task on his her own The task success rate score is the number of participants that finished the task successfully divided by the total number of participants We believe that Task 2 and Task 3 have the highest completion rate due to appeared user familiarity with the evaluated system While participants still had difficulties with these tasks they managed to resolve them on their own Task 0 was planned as the introductory one that required only basic operating system usage skills i e Windows 8 However some users did not manage to complete this task either due to lacking Windows usage experience or due to poor documentation Finally Task 1 required users to leverage their experience as developers what caused some of them to yield for help mainly due to the unknown IDE and development environment configuration 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 46 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 3182
123. formed the team which organised and conducted the usability evaluation sessions of SIMPLI CITY s developer package Table 7 Team of Facilitators of the Usability Evaluation Sessions Name Organisation Role within the Facilitators Team Michaela Kargl FGM Moderator guided the test users through the session Philipp H nisch TUV Observer took notes of revealed usability issues User Rostyslav Zabolotnyi Technician ensured proper functioning of the technical TUV equipment software and recordings 7 1 4 Technical Setup for the Usability Evaluation The test user worked on a Laptop running Windows 8 1 as operating system On this computer the IntelliJ IDEA 14 1 3 Community Edition was installed as IDE and it was part of the task of the test user to install the respective SIMPLI CITY IntelliJ Plugins Furthermore also Mozilla Firefox Foxit Reader WinRAR and 7 Zip were installed on this P Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 42 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report computer which the test user utilised for related tasks such as e g checking websites or extracting archives TeamViewer 10 was used for displaying the test computer s screen on the observer s computer and BlueBerry Flash
124. gain tries again Types in navigation no result Types in route Enter does not work test user has to scroll to the right on the page and click on the button test user finds the App Increased mobility test user clicks on the App e test user can t find any comments test user goes back to the App list and clicks on Environmental Awareness Rising but it s marked as installed already test user clicks on the simpli city Logo to go back test user clicks on Increased Mobility App again Clicks on the Install Button and gets a message Installation requested Checks the phone if it has been installed test user finds on the phone a Simpli City App and opens this app but this is not the requested App e 14 47 finished test user says that it s probably not possible to install an app Test User 4 09 09 2015 Timeline e Introduction 15 06 15 15 Explaining example task 15 15 15 19 User Test 15 19 15 35 Debriefing Interview 15 35 15 42 Task 1 e Testuser starts with task 1 e Types in the address and comes to the login page e It s a very long address e 15 21 task was completed the test user says it was very easy Task 2 e 15 21 Test user starts with task 2 e User 4 thinks it s the Simpli City marketplace but to be sure he downloads the help document e He does not know what is a PMA e t s a kind of a marketplace he thinks that he cou d install this on his phone e It was easy to find the help but it pro
125. ge type Should U121 Local storage of data x Have Storage of data in the cloud binary Must Mes semantic and Have structured System scalability U123 Scalability of the x Should cloud infrastructure Have This functionality as such was not covered within the final prototype however the overall Service Runtime Environment has been designed in a way that allows provisioning of scalability features in the future Scalability of the The REST Proxy being the single entry Should U124 x service platform point for all method calls can be Have retrofitted to serve as a load balancer and the built in monitoring and SLA watching modules can spawn new Service Runtime Environment instances should load increase and response times degrade General standard approach Should U125 Open interfaces X Have Openness of the Should uiz6 system Have nn Should U127 Extensibility X Have Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 33 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report tuifilled Explanation why not or only partially Requirement implemented Priority yes part no Business model Finance Apps The system allows U128 app developers to X ied
126. gh the session Markus Schuster FGM Observer took notes of revealed usability issues Andreas Maurer FGM Technician ensured proper functioning of the technical equipment software and recordings 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 110 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 12 1 2 2 Technical Setup for the Usability Evaluation The test user worked on a Laptop running Windows 7 as operating system On this computer the browsers Mozilla Firefox Google Chrome and Internet Explorer were installed which the test users utilised for accessing the SIMPLI CITY Application Marketplace when working on the test tasks TeamViewer 10 was used for displaying the test computer s screen on the observer s computer and CamStudio v2 7 4 was used for screen recording A Logitech 920 HD Pro webcam and the related software were utilised for recording the test users facial expressions and voice during their work on the test tasks and a Panasonic Lumix GX7 camera was used for video and audio recording of the debriefing feedback interviews 12 1 2 3 Usability Evaluation Sessions The Usability Evaluation Sessions took place at FGM s premises Sch naugasse 8a 8010 Graz Austria The pre test was done on Friday 04 09 2015 and 5 usability ses
127. hat there are no extra tasks that have to be done So like it when everything is done in one programme when there s only a button and it works User 3 And liked the easy way how you can publish a service to the marketplace 7 3 3 Main Usability Issues Revealed During the Evaluation 7 3 3 1 Main Usability Issues Mentioned by the Test Users During the debriefing interview at the end of the Usability Evaluation Sessions the test users mentioned several aspects of the SIMPLI CITY system which should be improved 1 itis difficult to find the delete button in the SIMPLI CITY menu of the IntelliJ IDEA User 5 Probably would improve the menu Bundle service and persist that s not really obvious that the delete button is in there User 1 The delete button placement was not intuitive at all Because it was on the menu option that stated that it persists a service and creates the service User 7 The deletion process of the service was hm think there could be some menu integration for the popup for this persist service think the popup is not necessary so did not like that think it would be easier to set up a context pane in IntelliJ User 3 Mixture of the items is not good so that you have a distinct menu for publishing and also included the deleting functionality That should be in a separate menu entry That would be more clear to say manage the market pl
128. hat this x know how SIMPLI CITY works was the right action how it is organised and how to develop services and apps within its framework Use an existing User will know what to x Developers should already One of evaluators suggested adding video example of how to do at this step have some SIMPLI CITY tutorials to the help section of the Application connect to SIMPLI development background i e Design Studio where using a series of CITY services to User will know that this x know how SIMPLI CITY works screencasts several important techniques can how it is organised and how to develop services and apps within its framework be described for instance connecting to an existing service Document Date i Page D7 3_Evaluation_Report_v1 0_For_Approval docx 2015 10 30 Status For Approval 90 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public 9 3 4 Task 4 Put an App on the SIMPLI CITY App Marketplace D7 3 Evaluation Report Action sequence for Yes Success Failure Story Comment Idea for Improvement completion of Task 4 Open IntelliJ IDEA User will know whatto x Developers start their IDEs daily do at this step User will know that this x was the right action Open an existing User will know whatto x Developers should already have SIMPLI CITY App do at this st
129. he team of facilitators watched the test user silently and the observer took notes of the test user s activities and any usability issues that became apparent After finishing the last test task the moderator thanked the test user for her his cooperation and asked the following three questions in the debriefing interview How was it Was there anything that you found to be especially good Was there anything that you found to be especially bad When the test user finished answering these questions the moderator asked her him to fill in the System Usability Scale SUS a standardised questionnaire to assess a user s perception of the overall usability of a system In case the test user had further questions these were answered by the team and after that the session was finished After each session the recordings were saved and the materials and setup was prepared for the next session 8 1 5 Test Users In total 4 test users participated in the Usability Evaluation of the SIMPLI CITY Applications Marketplace All of them answered the introductory questionnaire did the 5 test tasks and provided their feedback in the debriefing interview and feedback questionnaire Table 14 gives an overview of the answers of the Short Introduction Questionnaire stated by each test user 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 65 146 http www simpli ci
130. he view of the Service Marketplace and the search functionality should be improved User 6 think the web view of the Service Marketplace could be a little bit over full if there are many apps in there So this could be improved with an extra view User 4 The counter intuitive search interface puzzled me because normally you would expect that it works case insensitive 4 the feedback from the system should be improved User 1 was not aware first that the IDE is actually communicating and interacting with an external marketplace in the background without me noticing User 0 The worst thing well did not know whether I m done or not with this first stuff with the plugin It was hard like just copied some parts and then did not know whether I m done or not would have expected to get just some kind of confirmation 5 functionalities of the system should be improved User 7 The deployment process itself could probably be simplified cause basically the process is create a jar file and deploy it there s no necessity to have this step in between that you have to compile it with Maven first and then deploy it you could just have a button deploy my thing and the process does it for you User 0 Actually normal plugins are usually installed with some kind of installation menu next next next and then I check some checkboxes to not install some c
131. her App and step back once more the Login Page appears Once found out how the search function has to be handled it works via search button and not search app Every app has a short description and the average rate in stars of previous users is clear Within the app mask some very essential information is provided related to the app e g category version published date Within the app mask some pictures taken by the app are provided This is useful to let users have a better idea of the app itself The following Table 22 Potential Usability Issues includes all potential usability issues that were revealed by the evaluators Related screenshots are provided in Section 11 4 These potential usability problems found in course of the Heuristic Evaluation are ordered according to their severity with the most critical problems on top of the list The severity has been assessed by the evaluators according to the following scale 0 I do not agree that this is a usability problem at all 1 Cosmetic problem is not necessary to be fixed 2 Medium severe usability problem should be fixed 3 High level usability problem imperative to fix this before product can be released Table 22 Potential Usability Issues Reported by Corresponding Evaluator Screenshot s to be found in MB GL mk DW Section 11 4 Description of the Observed Usability Violated Heuristic Severity
132. ht SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public 9 3 1 Task 1 Develop an App Using the SIMPLI CITY System D7 3 Evaluation Report Action Sequence for Yes Success Failure Story Comment Idea for Improvement Completion of Task 1 Install the App Design User will know what to X Installing IDE plugins as While the Application Design Studio can be easily Studio in IntelliJ IDEA do at this step described is common installed from the supplied zip archive it is worth following the install practice checking if the plugin itself can be uploaded to the instructions in the main IntelliJ plugin repository where third party provided ZIP file User will know that this X plugins are located http plugins jetbrains com was the right action then installation would be even easier Open IntelliJ IDEA User will know what to xX Developers start their Rearrange menu system of the IDE put the do at this step IDEs daily SIMPLI CITY menu item before the Help menu item User will know that this X was the right action Create a new SIMPLI User will know what to x This is a usual step when CITY App project do at this step creating a new project following the New independently of the Project wizard under project nature File New Project menu option User will know that this X was the right action Document Date i Page
133. iUatlOniz oh eor la dta E eta iria nd iens 110 12 1 2 Method ADDpIIG uie etian teet eer S i cat o qtd erneute tndsteares 110 12 2 Summary of the Results of the Usability Evaluation ssssssss 113 12 3 Detailed Documentation of the Findings 114 123 1 Usability Melkles euo Eee Rer Pate bo Ee r ci a tere aeta cues ches a aE 114 12 3 2 Main Positive Aspects Mentioned by the Test Users 115 12 3 3 Main Usability Issues Revealed During the Evaluation 116 12 4 Task Sheets for Usability Evaluation of the App Marketplace from road users eje ig ON VIGW TEILEN 121 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 6 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 12 5 Transcription of the Debriefing Interviews essseseeesseeeeeeeeee 123 12 6 Observer Protocol 2 ot mitate rrr fee tei Dena wi eee 126 13 Usability Inspection of the Use Case related SIMPLI CITY App 133 13 1 Approach Taken for the Usability Inspection ssseesseeeeeeess 133 13 2 List of Usability Issues FOL aerei tdt Rete nea eins and eee 133 13 2 1 Screenshots Corresponding to the Revealed Usability Issues
134. ights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Table 33 Participants Time On Task Task 0 Task 1 Task 2 Task 3 Participant GUI GUI VUI GUI User 1 MA 1 5 6 0 4 0 0 5 User 2 EM 1 5 4 0 6 0 0 5 User 3 MB 1 0 8 0 6 0 0 5 User 4 MO 2 0 4 0 7 0 0 5 Average TOT minutes 1 5 5 5 5 75 0 5 14 3 1 4 Rating of the Overall System Usability After the completion of all tasks participants were asked to fill in a standardised feedback questionnaire SUS System Usability Scale As explained in Section 7 3 1 4 SUS yields a single number representing a composite measure of the overall usability of the system SUS scores have a range of 0 not usable at all to 100 perfectly usable system Table 34 Rating of the Increased Mobility application SUS score j User 1 User 2 User3 User4 SUS Questions MA EM MB MO 1 I think that would like to use this system frequently 2 3 5 2 2 found the system unnecessarily complex 4 3 3 3 3 thought the system was easy to use 2 2 3 2 4 would need the support of an experienced person to 2 1 2 4 be able to use this system 5 found the various functions in this system were well 1 2 3 3 integrated 6 I thought there was too much inconsistency in this 4 2 5 3 system 7 would imagine that most users would learn to use this 3 3 2 2 syste
135. in a couple of sentences But as said before think it s just the beginning of this marketplace so everything can follow User 5 Evaluation Session 5 Moderator How was it User 5 I expected a more mature version So do not understand why you already test it at this stage Well can understand can assume why you test it right now But would improve it improve it just improve it improve it Moderator Well that s exactly the purpose of this usability test that you find out at an early stage of development where the flaws and problems are and the way of thinking of the users so that the developers can adjust to that and improve it accordingly User 5 Yea but you could have made it better already now think there are so many basics not done right now that the test just shows you the basics and everything more detailed would be lost Moderator Yes that s true You have to make more usability tests in the course of the development That s completely true So this is the first usability evaluation of this product developing product and of course you need to have much more of these tests in the course of the development p Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 125 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluati
136. in the specification The latter limitation is by the way the major limitations of cloud offerings nowadays Sensor abstraction and interoperability interfaces The Sensor abstraction and interoperability interfaces to access all data from car sensors Data is accessed through OBD connection and stored on the Cloud infrastructure Data is collected dynamically and processed by the data processing component for further analysis and reasoning Interfaces are all well defined and documented The interaction with any car sensors is then easy and straightforward even for non experts the documentation is important though since many APIs are exposed The main limitation which goes beyond the scope of the project is the adaptability to different car manufacturers This is also known as a common limitation of commercial offerings Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 98 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Partner Opinion and Comments WORLD Application Design Studio Partner WORLD developed three apps using the Application Design Studio ADS The ADS is easy to install and comfortable to use with clear instructions in the help system Adding the possibility of debugging code in the future will be very important for the deve
137. ing the evaluation sequence In general the evaluated Application Marketplace was accepted quite well by every user Also we can see that the platform is more useful and easy to use on each interaction During the debriefing interview at the end of the Usability Evaluation Sessions the test users stressed two features of the SIMPLI CITY system which they ve especially liked e The single screen form to create an application e The simplicity to approve an application However the test users mentioned also some aspects of the SIMPLI CITY system which should be improved Create App button functionality and position Confusion between Add binary button and Upload binary button The place of the Submit button on the publication of an application The drag and drop in Tests Management tab is not intuitive Differences between test and group test are not stated clear Publish button was hard to find 4 2 Results of the Usability Inspection by Experts As a second step in the usability evaluation of SIMPLI CITY from developers point of view the Usability Inspection of the SIMPLI CITY system from software developers point of view took place at the TIE headquarters in the Netherlands on 26th of August 2015 As the usability inspection involved evaluation of development tools Application Design Studio and Service Development API plus SIMPLI CITY marketplaces it was taken care that all evaluators inv
138. ion 12 4 includes the task sheets which the test users got during the usability evaluation session 12 2 Summary of the Results of the Usability Evaluation The Usability Evaluation sessions with the test users were quite successful since a number of important usability issues were discovered during the test sessions In addition to important qualitative observations also some quantitative metrics were obtained such as the test user s time on task and the completion rate for each of the test tasks However due to different levels of experiences of the users high variations of the users task completion times in the test tasks 2 3 4 and 5 were noticed Most test users found the test task 2 Find out what this website is about test task 3 Search for an app that helps you to find the best route and test task 5 Install any of the apps listed in the SIMPLI CITY marketplace on the smartphone to be quite difficult During the debriefing interview at the end of the Usability Evaluation sessions the test users stressed three features of the SIMPLI CITY App marketplace which they ve especially liked e The website is very clear and simple e The site is not overloaded e The design of the login page and the logo of the App marketplace are appealing However the test users mentioned also some aspects of the SIMPLI CITY Application Marketplace which should be improved e There should be some introductory text at the top of
139. ion for SIMPLI CITY I would use this interface Test user User 5 Test Session 4 Moderator So User 5 how was it User 5 do not know My experience in service creation is not really impressive but it was easy for a person who sees this for the first time Or let s say quite easy Moderator What was the best thing What do you think was good User 5 The tutorial was good So because even I did not know about what is going on here could go on and perform Moderator Was there anything especially bad User 5 Nothing especially bad Only that forgot how to write an interface Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 61 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report Moderator What would you let s say fix or improve if you would be the software developer who created this stuff User 5 Probably would improve the menu Bundle service and persist that s not really obvious that the delete button is in there That s the only thing Moderator Nothing else to improve User 5 Oh yes it was hard for me to see this yellow message it was not really yellow Test user User 6 Test Session 5 Moderator How was it User 6 Nice and fun Moderator Was ther
140. ion give valuable hints for further improvement of the SIMPLI CITY system 8 1 2 Method Applied For the Usability Evaluation of the SIMPLI CITY Applications Marketplace the well established Usability Evaluation method Thinking Aloud Test was applied A test user is asked to speak out loud all her his thoughts while using the system to complete a given test task By observing the test user s way of working on the task and by analysing the reasons for confusion hesitation or mistakes of the test user valuable hints for improvement of the usability of the system can be obtained 8 1 3 Team of Facilitators Three persons formed the team which organised and conducted the usability evaluation sessions of SIMPLI CITY s Applications Marketplace Table 13 Team of Facilitators of the Usability Evaluation Sessions Name Organisation Role within the Facilitators Team Xavier Cases Camats WORLD Moderator guided the test users through the session Sergi Martinez WORLD Observer took notes of revealed usability issues Xavier Febrer WORLD Technician ensured proper functioning of the technical equipment software and recordings 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 64 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public
141. irmation is received check that the App is installed on the device and execute the App to check for correct installation Test The Usability Inspection was done in the following technical Environment environment e Smartphone LG Nexus 5 operated by the system Android 4 4 4 with browser Chrome installed and PMA installed under an active Wi Fi connection e Smartphone Samsung S3 mini GT 18200 operated by the system Android 4 4 2 with browser Chrome installed and PMA not installed under an active Wi Fi connection e Smartphone Motorola Moto G operated by the system Android 5 0 with browser Chrome installed and PMA not installed under an active Wi Fi connection 11 1 4 Method Applied For the Expert Evaluation of the SIMPLI CITY App Marketplace the well established Usability Inspection method Heuristic Evaluation was applied At the core of the usability inspection method Heuristic Evaluation is the evaluation of a user interface to a checklist of heuristic usability design rules in order to reveal key user interface issues The evaluators examined the SIMPLI CITY App Marketplace and assessed its compliance with the Usability Heuristics for touchscreen based mobile devices with voice based user interfaces listed below The evaluators followed the task based Heuristic Evaluation methodology and examined the App Marketplace by following the user s main tasks as listed in the context information tabl
142. it does not work Task 6 e 15 34 Test user starts with task 6 e Types on the button logout on the desktop e 15 35 Done test user states that this task was easy Test user 5 User 5 09 09 2015 Timeline e Introduction 16 04 16 12 Explaining example task 16 12 16 16 User Test 16 16 16 38 Debriefing Interview 16 39 16 49 Task 1 e 16 16 Test user starts with task 1 e Types in the address that s very long and comes to the login page e Logged in and finished the task e 16 18 task was completed the test user says it was very easy Task 2 e 16 18 Test user starts with task 2 e Read through the Description of the first App Environmental Awareness Rising e User 5 thinks the website is about Eco driving Task 3 e 16 20 Test user starts with task 3 e Clicks into the Search field e types in navigation in the search area clicks the search button no result Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 131 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report e types in route finds Increased Mobility App e test user states that this task was not too difficult Task 4 16 22 Test user starts with task 4 test user states he would like to have a button show all Clicks on the Amnesty App test user states that he would ex
143. ited to the session already had an ample experience with different development tools IDEs integrated development environments and so on which they use in their daily work Thus it was ensured that they could subjectively and partly objectively judge whether they could use the tools being tested in their real life work They could compare the tools being discussed with alternatives they have used in the past and they were able to compare the implementation of standard features ways of interface organisation responsiveness developer friendliness and other aspects The unanimous opinion of the evaluators was positive They liked what the SIMPLI CITY development tools allow them to do for development of SIMPLI CITY services and apps and how these services and apps can be later marketed on the SIMPLI CITY marketplaces Specifically the evaluators noted positively that the development tools are based on one of the best freely available IDEs IntelliJ IDEA which together with Eclipse is one of the most widely used IDEs in the industry 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 17 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report Furthermore the evaluators noted that the extensions provided for SIMPLI CITY specific work are logical and conscious the layo
144. ity Inspection of the SIMPLI CITY system from software developers point of view In Section 9 1 the Usability Inspection approach is described briefly Section 9 2 gives a summary of the results of the Usability Inspection and Section 9 3 outlines the findings in detail 9 1 Approach Taken for the Usability Inspection 9 1 1 Aim of the Usability Inspection This Usability Inspection was done in order to find out how easy it is for developers to learn by exploration to use the SIMPLI CITY tools targeted at software developers The results of this Expert Evaluation give valuable hints for further improvement of the SIMPLI CITY system 9 1 2 Method Applied For Expert Evaluation of the SIMPLI CITY developer package the well established Usability Inspection method Cognitive Walkthrough was applied The usability inspection method Cognitive Walkthrough focuses on the learnability of a system and is designed to find gaps and problems a user might have in determining the next appropriate step for completing a task A team of evaluators examines a system by stepping through the exact sequence of actions which are necessary to complete a user s task For each action the evaluators discuss whether or not a novice user would know what to do at this step and whether or not a novice user would notice that she he is on the right way when she he has done the right action All problems reasons and assumptions revealed during this
145. ity eu 1 5 Document Structure This deliverable is broken down into the following sections Section 1 provides an introduction for this deliverable including a general overview of the project and outlines the purpose scope context status and target audience of this deliverable Section 2 outlines SIMPLI CITY s evaluation approach Sections 3 4 and 5 provide summaries of the results of the Technical Evaluation the Evaluation from developers point of view and the evaluation from road users point of view respectively Sections 6 7 8 9 10 11 12 13 and 14 give a detailed description of the evaluation activities conducted and the results obtained during Technical Evaluation Usability Evaluation of SIMPLI CITY s Developer Tools Usability Evaluation of the developers part of the SIMPLI CITY App Marketplace Usability Inspection from developers point of view Functional Evaluation from developers point of view Usability Inspection of the road users part of the SIMPLI CITY App Marketplace Usability Evaluation of the road users part of the SIMPLI CITY App Marketplace Usability Inspection of the use case related app and Usability Evaluation of the use case 1 related app respectively Section 15 briefly summarises the evaluation results Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 9 146 http www simpli city eu Copyright SIMPLI CITY Pro
146. ject Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 2 SIMPLI CITY s Evaluation Approach The evaluation strategy followed within SIMPLI CITY aimed to reach two main objectives e to evaluate the prototypes developed within SIMPLI CITY against the requirements specified in deliverable D2 3 Requirements Analyses Report e to validate the applicability of the SIMPLI CITY prototypes in a real world setting In order to reach these objectives the evaluation strategy combined elements of technical evaluation elements of usability evaluation and elements of functional evaluation Technical Evaluation looked at the grade of fulfilment of the requirements specified within deliverable D2 3 Since these requirements cover all parts of the SIMPLI CITY system Technical Evaluation was done for the whole SIMPLI CITY system Usability Evaluation and Functional Evaluation looked at the usability and the functionality of the system from users point of view However the SIMPLI CITY system addresses two user groups software developers and road users Hence the parts of SIMPLI CITY that are relevant for software developers and the parts of SIMPLI CITY that are relevant for road users were evaluated separately Table 1 gives an overview of the evaluation approach applied in SIMPLI CITY Table 1 SIMPLI CITY s Evaluation Approach Evaluation Element Applied Objective Technical Eva
147. liarities contact NO NO NO lenses Mobile phone ownership Other Android Android Other smartphone smartphone smartphone smartphone Smartphone usage since gt 1 year gt 1 year gt year gt year Smartphone user expertise High High Medium Medium Android user expertise Medium High Medium Low Usage of Apps High High Medium Medium Usage of voice commands No Seldom Seldom Seldom Car driving expertise Medium Medium High Medium Use smartphone in the car Medium High Medium Medium Experience with routing apps YES YES YES YES What do you know about SIMPLI CITY Worked for Worked for Worked for Heard of Ever participated in a Usability study YES NO NO NO 14 1 5 Usability Evaluation Sessions The Usability Evaluation of the Increased Mobility app took place in SRM s office in Bologna Italy A dedicated usability testing place was arranged in SRM s office and equipped with two cameras recording the session One camera filmed the device screen and the activity of the hands of the test user The other camera pointed at the face of the test user in order to capture her his expression and feeling while using the application With this camera in addition to the video the audio was recorded as well Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 138 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201
148. lic University of Valparaiso Chile and Polytechnic School of Chimborazo Ecuador Source R Inostroza C Rusu S Roncagliolo C Jin nez V Rusu Usability Heuristics for Touchscreen based Mobile Devices in 2012 Ninth International Conference on Information Technology New Generations IEEE DOI 10 1109 ITNG 2012 134 10 Valeria Farinazzo Martins Salvador Andre Luiz S Kawamoto and Joao Soares de Oliveira Neto 2010 An Empirical Approach for the Evaluation of Voice User Interfaces User Interfaces Rita Matrai Ed ISBN 978 953 307 084 1 InTech Available from http www intechopen com books user interfaces an empiricalapproach for the evaluation of voice user interfaces 11 Gerry Gaffney 2001 Information amp Design pty Itd Voice Interaction checklist www infodesign com au Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 103 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Usability Heuristics for Touchscreen based Mobile Devices with Voice based User Interfaces 4 Consistency and standards The application should follow the established conventions the user should be able to do things in a familiar standard and consistent way 5 Error prevention In order to prevent errors non available functionalities should be hidden or disabled
149. lopment process Personal Mobility Assistant PMA Multimodal Dialogue Interface MMDI The PMA and basically MMDI still needs some development for being a mature product as some user interfaces are difficult to create or WORLD had to use some workarounds in order to get the desired result This is a normal process of developers needing elements that were not initially planned and should evolve altogether with developer engagement The MMDI is probably the most complex part to use not only technically but conceptually The learning curve is really high so having a good documentation and samples is the key for this part This last advice is a recommendation for the whole project in order to engage developers to work within the SIMPLI CITY environment a better documentation should be developed WORLD had to invest a lot of time learning how things work and also received a lot of help from other partners but for external developers having clear documentation tutorials samples or similar elements will be really important CRF Personal Mobility Assistant PMA Multimodal Dialogue Interface MMDI Partner CRF developed an App that includes functionalities in two languages Italian and English The PMA was used for building the multimodal interface in both languages The main difficulties were due to the limitations of the GUI that has forced developers to redesign some interactions i e it was impossible to visualize image a
150. lowing sections summarise the results of these evaluation activities More detailed information can be found in the respective Annexes of this document 4 1 Results of the Usability Evaluation by Test Users As a first step in the evaluation of that part of the SIMPLI CITY system which is targeted at software developers usability evaluation sessions with test users were conducted in Vienna and in Barcelona The results of these evaluation sessions were communicated to SIMPLI CITY s developers so that they could immediately start with improving the developed software 4 1 1 Results of the Usability Evaluation of SIMPLI CITY s Developer Tools The Usability Evaluation Sessions took place at the Technical University of Vienna Austria The pre test was done on Tuesday 23 06 2015 and 6 usability sessions were conducted on Wednesday 24 06 2015 The usability evaluation demonstrated that the developed tools already provide a reasonable support for SIMPLI CITY software developers Nevertheless a number of important usability issues were discovered during the evaluation sequence In general the evaluated developer plugin was accepted quite well by every user However due to different levels of experiences and background knowledge from the users such as a non existing familiarity with Windows 8 IntelliJ the used developer IDE and the pre installed unpack tool high variations of the users task completion times were noticed During the
151. loys the service to the SRE When clicking Bundle It do not hide the dialog window but just display a confirmation OK dialog on top after bundling is finished Remove the Select manifest option as it is not really needed When publishing a service to the Service Marketplace offer the developer a possibility to specify username and password to access the marketplace Document Date i Page D7 3_Evaluation_Report_v1 0_For_Approval docx 2015 10 30 Status For Approval 88 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 9 3 3 Task 3 Utilise a Service from the SIMPLI CITY Service Marketplace for Creating an App Action sequence for Yes Success Failure Story Comment Idea for Improvement completion of Task 3 Open the web browser User will know whatto x The URL has been and go to the Service do at this step communicated to the Marketplace web developer interface http simpli city eu 3030 User will know that this x was the right action Login with your User will know what to x Using username anda username and do at this step password for authentication is password common practice Note at the moment there is no open user User will know that this x registration for the was the right action Service Marketplace login credentials are provi
152. luation Evaluate the SIMPLI CITY system against the requirements specified within deliverable D2 3 Usability Evaluation from Evaluate the Usability of SIMPLI CITY s developer Developers Point of View package comprising of Service Marketplace the Application Design Studio the Service Development API and the part of the App Marketplace targeted to developers Functional Evaluation from Evaluate the functionality of SIMPLI CITY s tools Developers Point of View targeted at developers in real world usage Usability Evaluation from Road Evaluate the Usability of SIMPLI CITY s PMA with Users Point of View the Apps developed for the Use Cases in WP7 and WP8 and the part of the App Marketplace targeted to road users Functional Evaluation from Road Evaluate the functionality of SIMPLI CITY s PMA with Users Point of View the Apps developed for the Use Cases in WP7 and WP8 in real world usage conditions 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 10 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report Evaluation in SIMPLI CITY Task T7 3 Task T8 3 Evaluation of SIMPLI CITY s Marketplaces Evaluation of PMA Evaluation of Evaluation of PMA with Use Case SIMPLI CITY s with Use Case II related App Developer Tools related A
153. lugin for IntelliJ IDEA as being rather difficult 2 Jeff Sauro 10 Things to know about the Single Ease Question SEQ online at https www measuringu com blog sed10 php visited 12 06 2015 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 47 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 7 3 1 3 Time on Task TOT The amount of time needed by each participant to finish their tasks depended on their background knowledge about used tools and environment task description length and quality of user interface Task 0 which required participants to install SIMPLI CITY plugin for IntelliJ IDEA required only basic operating system knowledge i e Windows 8 and the ability to follow the manual On average this task took 8 minutes while the fastest participant required only 5 minutes and slowest required 15 minutes Task 1 asked participants to develop a Hello World Service using the SIMPLI CITY system This task required basic programming skills and ability to fulfil critical SIMPLI CITY software development steps On average this task was completed in 16 minutes while the fastest participant needed 10 minutes and the slowest required 22 minutes Task 3 required the users to find the created Backend Service in the Service Marketplace The partici
154. m status Suitable feedback The application should provide feedback to every action of the user the application should inform the user about successful not successful recognition of the user s input the application should keep the user informed about all processes and state changes in a reasonable time if the application takes a longer processing time the user should be informed about the current state and also about the expected duration of this waiting time 2 Match between the system and the real world The application should speak the user s language with words phrases and concepts familiar to the user real world conventions should be followed and information should be displayed in a logical and natural order Voice based UI The dialogue structure should resemble human dialogue and alternative options should be clearly delineated by clear pauses 3 User control and freedom The application should allow the user to undo and redo her his actions and it should provide emergency exits to leave the unwanted state The user should be allowed to walk through the dialogues in her his own pace The user should be able to interrupt at any time and shouldn t be forced to pass through the extended dialogue The application should provide a clear way to return to a starting point or main menu 8 http Awww nngroup com articles ten usability heuristics developed at the School of Informatics Engineering of the pontifical Catho
155. m very quickly 8 found the system very cumbersome to use 4 4 4 4 9 felt very confident using the system 2 3 2 2 10 I needed to learn a lot of things before could get 2 2 3 3 going with this system SUS Score intermediate per test user 35 52 5 45 35 SUS Score overall average 41 9 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 143 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Analysis of the answers of the test users to the feedback questionnaire reveals that the Increased Mobility app reaches a SUS score of 41 9 According to Bangor A et al this SUS score can be translated to users perceiving that the usability of the system is poor Confirming that the system is a prototype this indicates that there s still significant improvement of the system s usability necessary 14 4 Task Sheets for Usability Evaluation of the WP7 related App TASK 0 Use the SIMPLI CITY App to find out the SIMPLI CITY speech server Tell the moderator when you finish this task Then answer the following short question Overall this task was Very Difficult Very Easy TASK 1 INCREASED MOBILITY gt Insert settings related to your car car plate code of permission emission level value power source
156. much inconsistency in this system would imagine that most developers would learn to use this system very quickly found the system very cumbersome to use felt very confident using the system Document Date Page D7 3_Evaluation_Report_v1 0_For_Approval docx 2015 10 30 Status For Approval 48 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 Sal ee O SIMPLI CITY WP7 Public D7 3 Evaluation Report 10 1 needed to learn a lot of things before could get going with this system SUS yields a single number representing a composite measure of the overall usability of the system SUS scores have a range of 0 not usable at all to 100 perfectly usable system Table 12 Participants Rating of the Overall System Usability SUS Score Test Test User s Rating of the 10 Questions in the SUS questionnaire SUS Score User Q1 02 0324 Q5 06 Q7 Q8 QI Q10 User 0 3 2 2 1 2 3 4 3 2 1 57 5 User 1 3 2 4 1 4 3 4 2 3 4 65 0 User 2 5 1 4 2 5 2 5 1 3 1 87 5 User 3 4 1 2 2 4 5 4 2 2 3 57 5 User 4 4 2 4 2 4 2 4 2 4 2 75 0 User 5 4 1 5 3 5 1 5 1 2 1 85 0 User 6 4 2 3 2 3 2 4 2 5 2 72 5 User 7 5 2 4 1 3 3 5 2 4 2 77 5 Analysis of the answers of the test users to the feedback questionnaire reveals that the SIMPLI CITY developer package has got a SUS score of 72 2 Accor
157. n The task success rate score is the number of participants that finished the task successfully divided by the total number of participants Participants had some difficulties with Task 1 GUI and they were not completely able to complete it on their own In Task 2 VUI the test users failed in most cases because of problems with the vocal system However in two cases out of four i e Task 0 and Task 3 all test users were able to complete the task without the intervention of the moderator Task 3 was completed successfully by all the testers as well as the Task 0 which was used as example task to introduce the test users in the session 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 141 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Table 31 Task Completion Success Rate Task 0 Task 1 Task 2 Task 3 Participant GUI GUI VUI GUI User 1 MA v v v v User 2 EM J 7 s s User 3 MB 4 B B 4 User 4 MO J 4 i J Task Success Rate 100 50 50 100 Y successfully completed not completed help needed for completion 14 3 1 2 Ease of Task Ratings After each task the participants rated the ease of completing the task by answering the Single Ease Question SEQ Overall this task was on a 7
158. n a user deletes a service it is not clear whether the service is actually deleted or not In addition it was not clear why it was also deleted from the service marketplace After the Usability Evaluation the described observations were provided to the respective developers and they were actively working on it in order to improve the provided functionality and resolve the detected issues 7 3 8 3 Screenshots Documenting Potential Usability Issues T Ed ew Tools Coments Fome Shaehemnt ip MERUM DOI mS j ae so o 2 ET 42 SIMPLI CITYS 9D 2 BBX RAT 5T te we Extracting CAProgram Files ervices pluginzip foetu Developers sss a Tas EL The SIMPLI CITY Sen simpli city services plui down in case it is alr directly into the plugin M EB oe After unzipping the s ve e IntelliJ IDEA e IntelliJ IDEAV p e IntelliJ IDEAj ph Figure 2 Access Rights Issue during Task 0 see Issue 1 1 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 53 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report EUDGURETIISET VILE L VUEI USET 1 WOLROAGCEOHTIDUI Ly WEMDACRENUSETVILE Reste eric X9ncuniemyava EM resuaCUVelUr Jove Wien FUER 1 19 gt Ble Edt View Nmigste Code Analyze Betactor Buid Rum Took VCS Windo
159. n screen voice X H ave control and non voice control Output interaction Must U42 from system through x H Ul ave Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 26 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report fulfilled Explanation why not or only partially 2 Requirement implemented Priority yes part no Non distracting Should us interaction x Have The built in speech recognizer in f Android is used The recognition quality ee in the automotive environment depends Should U45 UNS acoustic x on the device connectivity and on the Have models for in car use speech recognition provider either built in provided by Google or from a 3rd party The built in speech recognizer in Android is used The recognition quality in the automotive environment depends U46 High speech on the device connectivity and on the Should recognition rate speech recognition provider either built Have in provided by Google or from a 3rd party No evaluation of word error rate WER has been performed Voice interaction through the car audio U47 system microphone amp X noni ave loudspeakers for hands free in car use User Experience Behaviour Interaction with the system App cra
160. n the list reads the description the test user thinks it s a little bit strange test user tries another App He can t find more information in the detailed information than in the overview finished task test user states it was not possible to find more information Task 5 11 36 Test user starts with task 5 User 2 has no idea how to do this He does not have experience with Android phone Login with the username clicks on first App on the list its already installed clicks on another App he has to login once again clicks on another App Gets a message but he finds this message weird confusing he clicks on the app again gets the massage again nothing happens User 2 goes back to the overview and tries again test user states he thinks it does not work he gives up Task 6 11 41 Test user starts with task 6 Types on the button logout on the smart phone Done Types on the button logout on the laptop Done 11 42 finished test user states that it was easy 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 128 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Test User 3 User 3 09 09 2015 Timeline e Introduction 14 04 14 14 Explaining example task 14 14 14 18 User Test 14 18 14 47 Debriefing Interview 14 47
161. nd instructions for use should be easily retrievable Voice based UI The application should take into account that users auditory memory is limited to a few short items the menu structure should be simple with no unnecessary levels there should not be more than five options in any menu 10 Help and documentation Whenever the user needs it at any stage of the dialogue the application should provide easy to retrieve instructions and help centred on the user s current task A list of concrete steps to carry out should be provided 11 Interaction and ergonomics Touch UI User interface elements should be placed in a recognisable position and should fit the natural posture of the hand Voice based UI The application s outputs should be clear with natural intonation and rhythm the application should understand natural user speech and be tolerant of differences in accent and speech patterns the application should recognise male female and child voices equally well the system should be tolerant of noise 12 Privacy Information about how personal data is protected and about content s copyright should be given f Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 104 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 11 2 Summary of the Re
162. nd text in the same screen etc Moreover for the setting up of the dialogue interaction it was necessary to invest more resources than initially planned The way of setting up things to make the whole MMDI interface fully working is quite complex and in CRF case a lot of support from partner TALK s staff was needed Some samples provided were of great support but further documentation and guidelines are necessary for allowing external developers to use this component easily In CRF view the set up needed should be simplified otherwise a lot of files configuration etc needed to be provided Sensor Abstraction CRF also extended sensor abstraction for getting access to the own car sensors The available documentation together with minor support from partner TUDA was sufficient for the completion of this task 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 99 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Partner TALK did not use any of the SIMPLI CITY components directly in its work and therefore abstained from evaluation Since the partners FGM and SRM were not a developing partner FGM and SRM did not participate in this functional evaluation of SIMPLI CITY s developer tools 10 2 Summary of the Results of the Functional Evalua
163. nfusing User 2 This response case is confusing When this little screen pops up and says it is installing do not remember exactly what was written For me it was completely confusing What is this It should be a bit clearer in the wording or not pop up at all and simply start installing Yes User 3 was not sure if it is installed or not on the phone this is confusing was not sure if should do anything else after pressing the install button This is confusing That s all could say 12 3 3 2 Main Usability Issues Noted by the Observer In the following the main usability issues as noticed by the observer during the usability evaluation sessions with the test users are listed Task 1 Login to https devel1 tempos21 com simpli city demo as a user no usability issues encountered login task was straight forward for all test users Task 2 When you ve logged in to https devel1 tempos21 com simpli city demo find out what this website is about 4ll test users had difficulties to find out what this website is about they were reading the texts carefully but could not find a hint about the purpose and features of the website since the only texts available were dummy app descriptions see screenshot Figure 18 only one of the test users had the idea to use the help link for this task but then was a bit irritated at first since the help link opened a word document and the test user stat
164. now what to X Developers should already Add tooltips explaining the difference between these Backend External do at this step have some SIMPLI CITY options However for a developer knowing basics of development background SIMPLI CITY these are self explanatory User will know that this X i e know how SIMPLI was the right action CITY works how it is organised and how to develop services and apps within its framework Document Date Page D7 3_Evaluation_Report_v1 0_For_Approval docx 2015 10 30 Status For Approval 85 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report was the right action Action sequence for Yes Success Failure Story Comment Idea for Improvement completion of Task 2 Write Maven options User will know what to X Since the majority of Add tooltips to the Maven option fields explaining i e group ID artefact do at this step modern projects use what they are However for an experienced ID and version Maven for project developer the fields are self explanatory E ae Add F1 or the Help button which would invoke help 9 p js describing these options likely to be already familiar with Maven User will know that this X was the right action Write project name and User will know what to X Developers should already
165. of SIMPLI CITY and expertise with Thinking Aloud tests After this introductory part the moderator explained to the test user how the Thinking Aloud method works and asked the test user to do one test task after the other and answer the Single Ease Question SEQ immediately after completion of each task While the test user was working on the task she he spoke out loud all her his thoughts and considerations the team of facilitators watched the test user silently and the observer took notes of the test user s activities and any usability issues that became apparent After finishing the last test task the moderator thanked the test user for her his cooperation and asked the following three questions in the debriefing interview How was it Was there anything that you found to be especially good Was there anything that you found to be especially bad When the test user finished answering these questions the moderator asked her him to fill in the System Usability Scale SUS a standardised questionnaire to assess a user s perception of the overall usability of a system In case the test user had further questions these were answered by the team and after that the session was finished After each session the recordings were saved and the materials and setup was prepared for the next session 7 1 6 Test Users In total 7 test users participated in the Usability Evaluation of the SIMPLI CITY developer package
166. of the logos There s no explanation what the stars dedicated to the apps mean 04 05 Rather a remark it is possible to access the App Marketplace without entering a password 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 108 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 11 4 Screenshots Documenting Potential Usability Issues This section displays a digest of the screenshots taken by the evaluators during their assessment in order to document identified potential usability issues T Mobile A O 9 4 8 15 27 devel tempos21 es p o Dn devell tempos21 com 4 c simplicityss consumer Application List Environmental Aware Figure 11 Screenshot 01 devel tempos21 es ED simplicitysess Application List Environmental Awareness Rising Travel 24 Comments ir Figure 14 Screenshot 03 T Mobile A amp i AZ TR BE devel tempos21 es omments fi t fe fr Y mobility for supporting the ever growing need of the society to enhance the driver s Figure 16 Screenshot 07 Di simplicity Figure 12 Screenshot 25 B s devel1 tempos21 com BJ applications marketplace ED simplicity eses gt Increased Mobility WP7 Increased Mobility WP7 P dn Category Traffic
167. ompletion time was 16 minutes with a standard deviation of 4 8 which is still below the allowed maximum time As expected compared to the first 2 tasks the 3 and 4 task were quite simple and straight forward We achieved a completion rate of 100 i e every user was totally able to finish these tasks on their own without any help In both cases the average completion time was about 2 minutes with a minor standard deviation of 1 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 45 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report During the debriefing interview at the end of the Usability Evaluation Sessions the test users stressed two features of the SIMPLI CITY system which they ve especially liked 1 The good integration with IntelliJ IDEA 2 The easy way of publishing services in the Service Marketplace However the test users mentioned also some aspects of the SIMPLI CITY system which should be improved 1 The plugin installation sequence is not straight forward 2 The developer guidance materials shipped along with developed software have to be improved 3 The provided user interface is sometimes counterintuitive and misleading 4 The software development sequence needs to be simplified After the Usability Evaluation the results o
168. on Report User 5 o k Who is responsible for the upkeep So who is the operator I m just thinking cause app stores are already existing so why to invent a new one Moderator Well the App Marketplace is just one component of the SIMPLI CITY system moderator explains the prototypes developed within SIMPLI CITY to the test user User 5 o k Moderator Was there anything that you find good User 5 So first of all SIMPLI CITY logo So the first catch is o k But the rest was just rather unimpressive right now Moderator Was there anything that you find especially bad User 5 Well think everything is based on the user interface So it should be based on graphics and things like this and it s just using text right now it s not Well how to say it You can t say anything right now it s just some text it s like What You need more to say something what is good or bad You know graphics Moderator Anything else What would you improve User 5 need a better search function A search field or so with written search inside Maybe something on top about this app store or app marketplace whatever some background information would like to have categories So do not know what s coming out but there was something like route navigation and something with eco driving Or funny games for eco education something like this So just would look at every o
169. ores by 2 5 to obtain the overall value of the SUS score SUS scores have a range of 0 to 100 5 Adjective ratings related to SUS scores 25 worst imaginable 38 poor 52 acceptable o k 74 good 85 Excellent 100 best imaginable Source Bangor A Kortum P Miller J Determining What Individual SUS Scores Mean Adding an Adjective Rating Scale in Journal of Usability Studies JUS Vol 4 Issue 3 May 2009 pp 114 123 Document Date Page D7 3_Evaluation_Report_v1 0_For_Approval docx 2015 10 30 Status For Approval 49 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 2 the easy way of publishing services in the Service Marketplace User 1 t seemed that the publishing worked pretty straight forward it seems very nice In my experience it can be rather difficult to publish a service in an open marketplace like that So if it stays that easy it would be a very handy thing to have User 2 Well especially good is to have a server side for publishing the service that you develop and then it makes the cycle for the development and publication starting from scratch You just need a couple of installations stuff and then can work with this That is a good point User 6 found it good that IntelliJ automatically uploads the service into the marketplace and t
170. ormation from crashes is U138 Provision of apps statistics rio avaiable Developer studio SDK Identification of the The Service Registry permits to store and provide information about U166 developer sfanat re the developer of a service However it is not possible to integrate p g with App Marketplace due to client restriction on API Developer studio SDK documentation Developers can create the dialogs but the GUI elements are U169 Provision of Ul templates provided dynamically through the dialog between the User and the MMDI Developer studio Statistics Provision of statistics e g U182 Usage traffic This is a manual process by requesting data from the database 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 13 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report Requirement Explanation Regarding Partial Fulfilment Development U195 Proactive user Personal Mobility Assistant PMA can show notifications when notifications asked by Apps MET This functionality has not been covered within T5 2 as prioritization U196 E of of notifications is part of the PMA PMA can show notifications when asked by Apps WORLD Scenario Use Case Topic ll 2
171. pants had to locate the Service Marketplace get familiar with it and locate the service which was deployed there while performing the previous task On average this task took 2 minutes while the shortest time was 1 minute and longest time was 4 minutes Task 4 asked the participants to delete the created service in the Service Marketplace The developers had to locate the functionality to delete the service and verify that the service is deleted On average this task was completed in 2 minutes with a fastest time of 1 minute and a longest time of 4 minutes Table 11 Participants Time On Task Participant Task 0 Task 1 Task 2 Task 3 User 1 6 15 2 2 User 2 15 22 3 2 User 3 9 18 2 4 User 4 5 11 4 1 User 5 7 21 2 2 User 6 8 12 1 2 User 7 6 10 2 1 Average TOT 8 16 2 2 in minutes 7 3 1 4 Rating of the Overall System Usability After the completion of all tasks the participants were asked to fill in a standardised feedback questionnaire SUS System Usability Scale answering the following 10 questions on a 5 point scale ranging from strongly disagree to strongly agree think that would like to use this system frequently found the system unnecessarily complex thought the system was easy to use would need the support of an experienced person to be able to use this system found the various functions in this system were well integrated thought there was too
172. pect to get information about what do need which operating system or WLAN connection and the price pictures clear headline and keywords user rating e test user states that it was a very easy task but that he had not found the expected information Task 5 16 25 Test user starts with task 5 User 5 takes the smartphone and tries to find the app on the Google Play Store Searches for Amnesty App Hint from the Moderator You can t find it in the Play Store User 5 looks on the simpli city marketplace Website if there is any download part Clicks on Help and downloads the Word document Reads through the chapter describing how he can install an App Goes to the Amnesty App and clicks on the Install Button User assumes that it is downloaded on the laptop and he has to copy it onto the smartphone However he cannot find it on the laptop Task 6 e 16 30 Test user starts with task 6 e Clicks on the button logout e 16 32 finished test user states that it was easy Test user wants a second trial of Task 5 e 16 32 Trying task 5 on the smartphone e Login to the SIMPLI CITY marketplace in the smartphone s browser e Test user is surprised that the site does not fit to the screen the user thinks that the font is very small e Test user states that there should be more information about the project on the marketplace website e Test user searches for eco driving App in Google Play Store e Moderator tells the user that
173. pection Bologna Test Device Model LG NEXUS 5 Test Device Operating System Android 4 4 4 Internet Connection wifi Preferences Installation of Apps from unknown sources enabled 13 2 List of Usability Issues Found The following Table 29 includes all potential usability issues that were revealed by the evaluator in the last Usability Inspection end of September 2015 These potential usability problems found in course of the Heuristic Evaluation are ordered according to their severity with the most critical problems on top of the list The severity was assessed according to the following scale 0 I do not agree that this is a usability problem at all 1 Cosmetic problem is not necessary to be fixed 2 Medium severe usability problem should be fixed 3 High level usability problem imperative to fix this before product can be released 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 133 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Table 29 Usability Issues Found by Heuristic Evaluation Corresponding Screenshot Severity Filename Violated Description of the Observed Usability Issue Heuristics 1 SIMPLI CITY Account creation registration There is 7 10 1 S2 S3 3 no possibility to register b
174. point rating scale ranging from Very Difficult 1 to Very Easy 7 Table 10 shows the ratings given to the single tasks by the test users and provides an average of these ratings for each task Table 32 Ease of Task Ratings Task 0 Task 1 Task 2 Task 3 Participant GUI GUI VUI GUI User 1 MA 6 2 4 7 User 2 EM 5 3 3 7 User 3 MB 6 1 4 7 User 4 MO 6 3 3 7 Average Rating 5 75 2 25 3 5 7 Task 3 has been perceived by the users as the easiest one Task 0 has been perceived as pretty much easy while Task 1 and Task 2 have been perceived as the most difficult ones In particular Task 1 is rated as very difficult and most probably some efforts in further developments are required in order to fix the main issues Task 2 is right on the average score and also in this case further development is required in order to enhance the users experience 14 3 1 3 Time on Task TOT The amount of time needed to finish a task varies among the test users However all test users faced some problems with the vocal interface was affected by some system issues 16 Jeff Sauro 10 Things to know about the Single Ease Question SEQ online at https www measuringu com blog sed10 php visited 12 06 2015 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 142 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All R
175. pp 114 123 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 70 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report User 2 think you should place the Submit button in a place more visible accessible probably outside the binaries tab User 3 The button Submit should be placed in a more general place e The drag and drop in Tests Management tab User 3 The drag and drop was anything but intuitive User 1 I figured it out the drag and drop in the Test Management by test and fail it was been hard to find out that you should drag from the name and not from the icon User 4 The zone to drop should be marked once you are trying to drag something this way it will be more intuitive User 2 could not understand why dragging from the icon it did not work Strange e Differences between test and group test User 2 I could not understand why a test should be added to a group User 4 I expected to find the tests created in the page to assign to the application even if did not want to add it to a group User 1 would expect a wizard to create the tests so then will not be confusion between groups and tests and so on e Publish button was hard to find User 1 t was confusing
176. pport of public Could UA transport users as end x Have users Support of i Could U5 pedestrians as end x Have users U6 Support of truck X Could drivers as end users Have Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 22 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report iunea Explanation why not or only partially Requirement implemented Priority yes part no Support Tor dlsanled The Multimodal User Interface allows drivers It should take into account the the support for disabled drivers Will Not U7 interaction between X However the according functionalities Have t e user and the road need to be supported by app For Now h developers information system Multiuser support It should support multiple drivers and Will Not U8 different users with X Have their specific basic For Now data and user preferences User goals Strategic Allow users to become Must U9 X green Have U10 Reduction of the x Through specialized apps i e the Must number of accidents blackspot warning service Have U11 Provision of useful Must information to drivers Have No distraction of the Must ue driver z Have U13 No SIMPLI CITY Big x Should Brother Have Us
177. pps Technical Evaluation of the SIMPLI CITY System Figure 1 Evaluation within the SIMPLI CITY Project As depicted in Figure 1 within the SIMPLI CITY project evaluation was assigned to the two tasks T7 3 and T8 3 responsible for the evaluation of the Use Case and Use Case Il related parts of the SIMPLI CITY system respectively While the evaluation of the Use Case and Use Case II related SIMPLI CITY Apps could easily be assigned to the relevant work packages the evaluation activities related to the other parts of the SIMPLI CITY system were relevant for both Use Cases Therefore the evaluation of SIMPLI CITY s market places and SIMPLI CITY s developer tools as well as the Technical Evaluation of the whole SIMPLI CITY system were done in close cooperation of both evaluation tasks T7 3 and T8 3 The following 3 sections of this document include the results of all these common evaluation activities done within the SIMPLI CITY project as well as the results of the evaluation of the Use Case related SIMPLI CITY App More detailed information regarding these evaluation activities and their results can be found in Sections 6 to 14 of this document For more information regarding the evaluation of the Use Case ll related SIMPLI CITY Apps refer to deliverable D8 3 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 11 146 http www simpli city eu Copyright SIMPLI CITY Project
178. pter presents the results of the Usability Evaluation of the SIMPLI CITY system from software developers point of view In the following Section 7 1 the Usability Evaluation approach is described briefly Section 7 2 gives a summary of the results of the Usability Evaluation and Section 7 3 outlines the findings in detail Furthermore Section 7 4 includes the Task Sheets which were used during the Usability Evaluation sessions and Section 7 5 provides a transcription of the Debriefing Interviews conducted at the end of each evaluation session 7 1 Approach taken for the Usability Evaluation 7 1 1 Aim of the Usability Evaluation This Usability Evaluation was done in order to find out how easy it is for developers to use the SIMPLI CITY tools targeted at software developers The results of this Usability Evaluation give valuable hints for further improvement of the SIMPLI CITY system 7 1 2 Method Applied For the Usability Evaluation of the SIMPLI CITY developer package the well established Usability Evaluation method Thinking Aloud Test was applied A test user is asked to speak out loud all her his thoughts while using the system to complete a given test task By observing the test user s way of working on the task and by analysing the reasons for confusion hesitation or mistakes of the test user valuable hints for improvement of the usability of the system can be obtained 7 1 3 Team of Facilitators Three persons
179. r testing Select tests groups Versions Assign testers and drop over the v d test c J Peg sot imp cea ye version i edd SOE CRB OGD Click on the arrow to see the assignation Drag and drop over the test case group dropped version Fake app tests gt Group test for Environmental Awareness Rising application Group Testing for Rising kam the Driver s Comfort gt Application User Tester Two new Test app group gt new testing category gt ka Test group for Prog3 gt i Simpli city 2014 2015 Figure 10 Assign tests user during Task 1 8 4 Task Sheets for Usability Evaluation of the App Marketplace from Developers Point of View TASK 0 Create add a new application 1 Fill in all needed fields media for a new app 2 Submit the binary and the media needed 3 Submit the application for approval Tell the moderator when you finish this task Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 74 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report TASK 1 Create a new set of tests and assign it to an application 1 Create a new set of tests 2 Assign the tests to the application 3 Assign a user to do the testing 4 Accept it as ready for testing Tell the moderator
180. r the persists hm for this persist service think the popup is not necessary so did not like that think it would be easier to set up a context pane in IntelliJ 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 63 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 8 Usability Evaluation of the Developers Part of the App Marketplace This chapter presents the results of the Usability Evaluation of the SIMPLI CITY Applications Marketplace from developers testers and approvers point of view In Section 8 1 the Usability Evaluation approach is described briefly Section 8 2 gives a summary of the results of the Usability Evaluation and Section 8 3 outlines the findings in detail Furthermore Section 8 4 includes the Task Sheets which were provided to the test users during the evaluation sessions and Section 8 5 provides the transcription of the Debriefing Interviews conducted with the test users at the end of the sessions 8 1 Approach taken for the Usability Evaluation 8 1 1 Aim of the Usability Evaluation This Usability Evaluation was done in order to find out how easy it is for developers to use those parts of the SIMPLI CITY Application Marketplace which are targeted at software developers The results of this Usability Evaluat
181. rNameHere gt BackendService 3 Please ensure that Java SDK is correctly setup a Open an arbitrary java file in order to verify if Java SDK is configured properly hint there is a yellow message on top of the editor field b You can see the standard Maven project structure on the left side src main java 4 Create a helloWorld method in the interface src main java lt Groupld gt lt Artifactld gt Service and in the class located in src main java lt Groupld gt impl lt Artifactld gt Servicelmpl a Ignore any existing errors in the project 5 Return an arbitrary string from the defined method implementation 6 Change Service Registry URL in Settings from SIMPLI CITY menu item a Hint Correct URL is in the task description 7 Open the SIMPLI CITY Development Guide from the SIMPLI CITY Menu a Follow the Bundling a Service instructions b Hint the maven plugin panel is docked to the right side of IDE Maven Projects 8 Ensure Service received a ServicelD Tell the moderator when you finish this task Then answer the following short question Overall this task was Very Difficult Very Easy Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 57 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report T
182. rapware and then usually get the confirmation installation complete and then a checkbox like start it and then see that everything works It would be better to have something like that User 2 You need to support more than just one IDE And also this junction when you generate a project can be richer like provide more functionality mean if want to have a more complex application what shall do You need more detail It should provide more specific things like if want an application for GPS should have a generated code interact with the Google API from someone to generate a map So that s what I want So inside of the 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 51 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report plugin you should have some key to generate code for user for developer that is the user of this system The system should support more than just a skeleton 7 3 3 2 Main Usability Issues Noted by the Observer In the following the main usability issues as noted by the observer during the test are listed in the order of occurrence 1 Task 0 Install the SIMPLI CITY plugin for IntelliJ IDEA 1 1 The IntelliJ IDEA plugin installation sequence has to be simplified Our users had significant p
183. re 5 Marketplace Search Function is Case Sensitive see Issue 3 2 WODOUXETIUETVIUE UVUMITHAUSES I WUT PILE RTT CHE VTRLDREKETIO ST HUI 7 UL DeCKemia ence Tavern ue RPT reser vicem jenes mmis Ld File Eda yew Bavigate Code Analyze Befactor Build Ryn Tools VCS Window Help Ei tevtitackendService TH pomam d e and persat ui jirpenm Ga Side Ot o r mtra Tea Cana to oa Serie W TestServiceimpljava x Maven Projects e 118 G Sette i 3 testBackendService package test um nd Sowa PRO Ses amp D ides Fo Developer s Guide a De import 5 Loge spr f Lifecycle m v mee public cL clean E fiw B D test vehdate s Bites 2 M cei Y zs public String getCityt s is Test eum TestServiceimpt O package b Test ctivator o nse t ver TestService weit g helloterl y return asdf OQ mts Ca resources site E OSGI INF blueprint O depic blueprint smil public void Syster out p tin Test Service started Ca Pug Ett T Dependencies Ope gt D trge mantest ami m po Ran b y Bm wu m pu im D Wes SRun gto LJ SIMPLI CITY development toot oeu SIMPLI CITY Seri C Nu TA 1 24042015 Figure 6 Delete Service Functionality is not Easy to Find see Issue 4 1 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 55 146 http www simpli city eu Copyright SIMPLI CITY Project
184. rketplaces In order to foster the wide usage of those services a holistic framework is needed which structures and bundles potential services that could deliver data from various sources to road user information systems SIMPLI CITY provides such a framework by facilitating the following main project results e Mobility Services Framework A next generation European Wide Service Platform EWSP allowing the creation of mobility related services as well as the creation of corresponding apps This enables third party providers to produce a wide range of interoperable value added services and apps for drivers and other road users e Mobility related Data as a Service The integration of various heterogeneous data sources like sensors cooperative systems telematics open data repositories people centric sensing and media data streams which can be modelled accessed and integrated in a unified way e Personal Mobility Assistant An end user assistant that allows road users to make use of the information provided by apps and to interact with them in a non distracting way based on a speech recognition approach New apps can be integrated into the Personal Mobility Assistant in order to extend its functionalities for individual needs To achieve its goals SIMPLI CITY conducted original research and applied technologies from the fields of Ubiquitous Computing Big Data Media Streaming the Semantic Web the Internet of Things the Internet
185. roblems installing it even if they were following the documentation 2 Task 1 Try to develop a Hello World Service by using the SIMPLI CITY system 2 1 The user interface is suboptimal 2 1 1 Every menu item related to developed plugin is disabled if no SIMPLI CITY project is opened The users have difficulties figuring out what to do to enable them 2 1 2 The Development Guide is unclear and misleading Users had significant problems understanding how to proceed and how to achieve their goal 2 1 3 The manifest xml is being rebuilt on project manipulation bundle and persist etc This may cause a problem editing and extending it 2 1 4 Registry URL should be configurable on project creation Currently a user has to configure it manually in configuration 2 1 5 Error reporting confuses developer Whenever a developer forgets some step or configures something incorrectly SIMPLI CITY system should provide a reasonable message explaining the problem instead of some internal exception 2 1 6 Automatic project packaging is required The developed plugin has to conceal intermediate steps that do not require the user s attention 2 1 7 Connect to a Service menu item is misleading Users often had difficulties understanding if they need it or not 2 1 8 Persist Service bundle dialog is overloaded and hard to use It contains a lot of functionality which are hardly used or usable The users had troubles to navigate
186. rovement completion of Task 5 Open the web browser User will know what to x The URL has been communicated to the and go to the Service do at this step developer Marketplace web interface http simpli User will know that this x city eu 3030 was the right action Login with your User will know what to x Using username and a password for username and do at this step authentication is common practice password User will know that this x was the right action Go to Add Service fill all the required fields and add some media images User will know what to do at this step User will know that this was the right action Currently this option add service is not available in the Service Marketplace Instead the developers can deploy their service to the SIMPLI CITY SRE Service Runtime Environment using the IntelliJ plugin and then the Service Marketplace will list this service Click Create Service then Binaries and Upload Binary and select the previously generated JAR file User will know what to do at this step User will know that this was the right action Currently this option add service is not available in the Service Marketplace Instead the developers can deploy their service to the SIMPLI CITY SRE Service Runtime Environment using the IntelliJ plugin and then the Service Marketplace will list this service Document Date i Page D7 3_Evaluation_Repo
187. rovision of best Must xis practices 5 Have T Must U171 Provision of tutorials X Have Provision of Must ue guidelines Have Provision of Must Mus examples 7 Have Developer studio Mind set Permit to develop an U174 app in less than a X Mod ave day Hide complexity from Should uiis developer a Have Developer should not need to be skilled on TRU NOE UG spoken dialogue nave pes 9 For Now design Developer studio Simulator Provision of a PMA Developers can make use of an Android Wil Nor Ure emulator A Virtual Device to emulate the PMA RAVE For Now Developer studio Functionality description Definition of U178 minimum hardware X iind ave requirements of apps Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 37 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report tuifilled Explanation why not or only partially Requirement implemented Priority yes part no Description of services semantic U179 description x is ave keywords phrasing ontology Developer studio Debugging Easy debugging of Could pie services Have App and service Could Tel crash reports n Have Developer studio Statistics Provision of Ls pe Thi
188. rst is the first time you use the platform some buttons are confusing but l am sure after the first time there will not be problems User 4 Test Session 4 Moderator How was it User 4 I found the platform very useful but think that new prototypes have to be developed in an iterative way to get an optimal user experience or at least better than the actual for me Moderator What was the best thing User 4 What liked most of the platform was the coherence of the spatial distribution and colour We can take the navigational menus as an example on right top that makes it easy to learn In the other and using only two colours with a not overwhelming simple design brings lots of free space Moderator And what was the worst thing User 4 That if you are not familiar with the functional process of the tasks ie if you are a young developer and it is your first time attempting to submit an application the process and or the information of help given is very limited to understand how to do all the steps correctly 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 77 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 9 Usability Inspection from Developers Point of View This chapter presents the results of the Usabil
189. rt_v1 0_For_Approval docx 2015 10 30 Status For Approval 94 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 10 Functional Evaluation of SIMPLI CITY from Developers Point of View In August September 2015 during functional evaluation of SIMPLI CITY the partners software developers were asked to express their subjective opinions about using components developed by other partners in the project This included not only generic thoughts but also concrete suggestions on how to improve existing components and make them more developer friendly and also experiences in working around shortcomings which developers as users of these components had to face Also improvement suggestions which go beyond the specifications of the components within the SIMPLI CITY project were made by the partners The results of this evaluation were communicated to the authors of the described components so that they could get an impression of how useful their components were and how these components could be further improved Some of the suggestions were concrete enough to result in a bug or feature request and could be added directly to the Jira bug tracking system for later fixing 10 1 Detailed Documentation of the Findings Partner Opinion and Comments TUV Cloud Based Information Infrastructure for the Service Registry
190. s are functioning under real world conditions Functional Evaluation activities were carried out On the one hand Functional Evaluation looked at the performance of the SIMPLI CITY components dedicated to support software developers when utilised for developing real services and apps within SIMPLI CITY and on the other hand also the performance of the Use Case related app in a real car in real traffic in Bologna was evaluated The results of these Functional Evaluation activities pointed out that the prototypes of the SIMPLI CITY components are fully functioning in real usage i e when really used for developing services and apps and when used in a real car on the road However of course the results of the Functional Evaluation were also reflecting the prototype state of the SIMPLI CITY system The suggestions of useful additional functionality from a practical point of view which resulted from the Functional Evaluation are valuable and helpful hints for future development of the system Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 146 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201
191. s is a manual process by requesting Must U182 statis tics e g usage x data from the database Have traffic Provision of bug Could meee reports 2 Have Logs can be provided but not step by U184 Provision of crash step debugging as Apps are installed Could reports Bug and statistic information is not Have delivered to the cloud Development Interaction between apps and services Standardised Must U185 messages between X Hav apps U186 Registry of installed x Must SIMPLI CITY apps Have Will Not U187 Composition of apps X Have For Now This requirement was not covered within U188 Composition of the final prototype since the Should services composition of services happens in Have X apps not on the level of services Development Interface U189 Unified interface for x Must accessing sensors Have 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 38 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report tuifilled Explanation why not or only partially Requirement implemented Priority yes part no Unified interface for Must U190 accessing user x H ave centric data Simulation of Must E sensors x Have Development J Must U192 Service deployment x Have Exchange of
192. s of the SIMPLI CITY system several useful hints for improvement of their usability were obtained by the usability evaluation Many of these improvements suggested by the results of the usability evaluation were implemented by the SIMPLI CITY developers immediately Some further improvements of the WP7 related App were even done in the first weeks of October 2015 However of course not all of the usability issues could be solved during the runtime of the project but the remaining usability issues are well documented and constitute valuable contributions for further developments of the SIMPLI CITY prototypes Functional Evaluation confirmed that the SIMPLI CITY prototypes are functioning in real usage As an additional outcome of the Functional Evaluation several suggestions for practically useful enhancements of the functionality of the prototypes were obtained which will be helpful for further development of the SIMPLI CITY system 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 41146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report Table of Contents TINI 31d ojo PE OIG 2 ets a a lets cheer hat RTT rr 8 1 1 SIMPLI CITY Project Overview cccccceeeeeeeeeeeneeeeeeeeeeeeeeeeaeeeeeeeeeeeeeeneeeeeeeeeeess 8 1 2 Deliverable Purpose Scope and Context
193. shes are Should pao minimized Have U49 A crash should not T Should impact other apps Have User Experience Offline access U50 Prefetching of media x Must data amp offline access Have U51 Avoid the download of x Should data from 3G Have U52 Offline access of data x Should used within apps Have Automatically pre fetch data from users cars but no functionalities to detect apps that could make use of such data The Could U53 App recommendations X number of services apps was limited in H E ave the first iterations of the project and recommendation could not be properly tested Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 27 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report ned Explanation why not or only partially Requirement implemented Priority yes part no Should U54 Expiration of data x Have User Experience Feedback Available only from mobile marketplace Should U55 Rating of apps X app not from the marketplace web Have interface Feedback to U56 developers through X ied ave the marketplace Social network U57 functionality with the x Could objective of spreading Have the word User Could moe recommendations Have User Experi
194. siderations the team of facilitators watched the test user silently and the observer took notes of the test user s activities and any usability issues that became apparent After finishing the last test task the moderator thanked the test user for her his cooperation and asked the following three questions in the debriefing interview How was it Was there anything that you found to be especially good Was there anything that you found to be especially bad When the test user finished answering these questions the moderator asked her him to fill in the System Usability Scale SUS a standardised questionnaire to assess a user s perception of the overall usability of a system In case the test user had further questions these were answered by the team and after that the session was finished After each session the recordings were saved and the materials and setup was prepared for the next session Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 111 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 12 1 2 4 Test Users In total 5 test users participated in the Usability Evaluation of the SIMPLI CITY App Marketplace All of them answered the introductory questionnaire did the 6 tasks and provided their feedback in the debriefing in
195. sions were conducted on Wednesday 09 09 2015 The average duration of the Usability Evaluation Sessions was 42 minutes with the longest session lasting for 51 minutes and the shortest session taking 35 minutes All Usability Evaluation Sessions followed the same process First the moderator welcomed the test user introduced the team and explained the purpose of the session The moderator gave an overview of the test procedure and asked the test user for her his permission that the session can be recorded video audio and screen recording and these records can be used for later analysis and reporting purposes After the test user signed the consent form the moderator encouraged the test user to ask questions whenever anything is unclear In a short introductory interview the moderator obtained from the test user some demographic information gender and age and some information regarding relevant skills and knowledge of the test user such as e g smartphone usage experience with app stores and internet browsing in general knowledge of SIMPLI CITY and expertise with Thinking Aloud tests After this introductory part the moderator explained to the test user how the Thinking Aloud method works and asked the test user to do one test task after the other and answer the Single Ease Question SEQ immediately after completion of each task While the test user was working on the task she he spoke out loud all her his thoughts and con
196. stination within Bologna Select an area within the city centre of Bologna or reachable from your current location by passing through the city centre e Task 3 Use the SIMPLI CITY information App to find out the SIMPLI CITY website address Section 14 4 includes the task sheets which the test users got during the usability evaluation session Throughout the sessions the facilitator used a predefined grid structured to simplify data collection After finishing the last test task the facilitator thanked the test user for her his cooperation and asked the following three questions in the debriefing interview How was it Was there anything that you found to be especially good Was there anything that you found to be especially bad When the test user finished answering these questions the Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 139 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report facilitator asked her him to fill in the System Usability Scale SUS a standardised questionnaire to assess a user s perception of the overall usability of a system In case the test user had further questions these were answered by the facilitator and after that the session was finished After each Session the facilitator took a time to control his notes an
197. sults of the Usability Inspection The version of the SIMPLI CITY App Marketplace which was delivered to the evaluators for Usability Inspection still needed some improvements before it can be considered as usable and functioning as expected The potential usability problems which were revealed by the Usability Inspection were forwarded to the respective developer partners so that the SIMPLI CIY App Marketplace can be further improved Table 22 in Section 11 3 presents a list of all the potential usability problems which were revealed by the evaluators The main issues were e The App Marketplace seems to be not designed as a responsive website gt the page view does not adjust automatically to the size of the display and thus the page is only partially displayed on the small screen of the device e g the search button is not visible alongside with the respective communication field where the user enters the search term and the install button is not visible when reading the App description and vice versa e t is not possible to install the App by clicking on the install button and no feedback is provided to the user in case the App cannot be installed e The user is frequently thrown out from the marketplace and has to re login e There is no information about the marketplace not even an introductory text no description of the functionality no help supporting the user Despite the potential usability issues the Usa
198. t better formulated and more intuitively designed Ah yes and the delete button placement was not intuitive at all Because it was on the menu option that stated that it persists a service and creates the service Test user User 3 Test Session 2 Moderator So how was it User 3 Interesting was the first time in an evaluation and testing situation and I m a little bit nervous and thus did not always remember the things that l ve read before that Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 60 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report with the URL for example was a bit lost in the description of the task It s complicated for the first time Moderator Was there anything that you find especially good User 3 The service creation in IntelliJ with a separate plugin or menu structure so that you can access and find the needed methods in an easy way And the easy way how you can publish a service to the marketplace But the other way around it should be more clear how to unpublish or delete again So it was a bit unclear to go again in the menu for the persistence and delete the service there Moderator Was there anything that you found to be bad apart from this issue with the deletion of the service
199. t more information than in the overview and there are no real comments e Size information is empty test user states that it would be good to see a screenshot of the App e test user states that this task was a little bit difficult Task 5 14 33 Test user starts with task 5 clicks on first App but this is already marked as installed so test user clicks on the increased Mobility App User 8 clicks on the install button Gets a message that the installation is requested test user searches for the App on the Computer via the Search function test user can t find it 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 129 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report e test user thinks it is not installed already because it was too fast and he could not find it Task 6 e 14 36 Test user starts with task 6 e Types on the button logout on the desktop Done e test user states that it was easy Test user wants a second trial of Task 5 e 14 38 tries it on the smartphone to install an app Types in the address in the browser Page is not adjusted to the smartphone screen User 3 thinks that s not good because it is hard to navigate test user logs in into the simpli city marketplace website test user clicks on install test user has to login again Login a
200. terview User 1 Evaluation Session 1 Moderator How was it User 1 Interesting It seems to be that it could be a good marketplace to see if this are apps for direct use or guidance for development of apps At this stage I m not able to tell if this is going to be a good user experience since it s not developed that far Moderator Was there anything that you find especially good User 1 The overview of the app list is actually good you see it directly when you start on the front page when you have entered in You have entries that tell you if there is a discussion on an app or an app itself offered This is quite fine That s it Moderator Something especially bad User 1 Hmm it s not especially bad but the search option should react on the enter button guess I m not the only one who is typing and then not using the mouse to click on anything but just firing at the return key And hmm from my experience now it s of course disappointing not to have anything like an app working which you can install or where you can have a look on On this stage it is poor information at a very low level and would not motivate me to go back on the page if wouldn t know that it s a test and it s going to be further developed Moderator Anything else you want to tell us User 1 Maybe the developers should think about where to place the apps or also the app marketplace maybe it s possible
201. terview and feedback questionnaire The 5 test users were persons with different experiences in using App stores We ve had four men and one woman The test users were between 25 and 54 years old Table 14 gives an overview of the relevant skills and knowledge of the test users as stated by them in the introduction questionnaire Table 24 Overview of the Test Users relevant Skills and Knowledge Test User User 1 User 2 User 3 User 4 User 5 Usability Eval Session Session 1 Session 2 Session 3 Session 4 Session 5 Peculiarities glasses glasses N glasses contact lenses contact lenses contact lenses Mobile Phone Ownership Android iOS Android Android iOS Smartphone Usage since gt 1 year gt 1 year gt 1 year gt 1 year gt 1 year Usage of Apps low seldom high a lot medium high a lot high a lot Experience with App Stores low medium medium high high GERE with browsing high mediun high high high the internet what do you know about SIMPLI CITY heard of heard of heard of worked for heard of Ever participated in a ds al a s No Usability Study y y Three of the test users had an Android smartphone two of them had a smartphone operating with iOS All of the test user had used a smartphone for longer than a year Three of the test users stated that their usage of Apps is high one of them said to have medium usage and one said that he uses Apps seldom Four of the test users
202. the description of the app You should get a more detailed description of the app not just the same text as in the overview Moderator Anything else that you would like to tell the developers User 2 Well suspect that this is sort of a test of the surface of the whole thing so think that apps will come in the future right So I won t say much about it Of course there was one thing on the Android on the screen of the mobile it was the install button was not visible You had to scroll so that shouldn t be the case Moderator Anything else User 2 Yes this response case is confusing So it confused me when it said that When this little screen pops up and says it is installing do not remember exactly what was written For me it was completely confusing What is this It should be a bit clearer in the wording or not pop up at all and simply start installing Yes User 3 Evaluation Session 3 Moderator How was it User 3 It was interesting liked it mean did not do something like this before but it was o k enjoyed it Moderator Was there anything of the system that you especially liked or that you find it was implemented quite good User 3 It s very simple to find the apps because you just see them immediately after log in But I did not know what to expect when typed in the link That s why I was a bit confused was not sure whether this is the first pag
203. the end of the sessions and Section 12 6 includes the notes taken by the observer during the usability evaluation sessions with the test users 12 1 Approach taken for the Usability Evaluation 12 1 1 Aim of the Usability Evaluation This Usability Evaluation was done in order to find out how easy it is to use this part of the SIMPLI CITY Application Marketplace that is targeted at road users The results of this Usability Evaluation give valuable hints for further improvement of the SIMPLI CITY system 12 1 2 Method Applied For the Usability Evaluation of the SIMPLI CITY Application Marketplace from road users point of view the well established Usability Evaluation method Thinking Aloud Test was applied A test user is asked to speak out loud all her his thoughts while using the system to complete a given test task By observing the test user s way of working on the task and by analysing the reasons for confusion hesitation or mistakes of the test user valuable hints for improvement of the usability of the system can be obtained 12 1 2 1 Team of Facilitators Three persons formed the team which organised and conducted the usability evaluation sessions of the road users part of the SIMPLI CITY Application Marketplace Table 23 Team of Facilitators of the Usability Evaluation Sessions Name Organisation Role within the Facilitators Team Michaela Kargl FGM Moderator guided the test users throu
204. the page describing what this website is about some sort of Welcome text which explains to the user the features of the marketplace e The search function should be better not only full text search of the app descriptions but also finding suitable apps when typing in more general keywords such as e g navigation or routing e There should be used graphics and not only text in the list of the Apps e There should be more information about costs and size of the Apps e The website should be optimized for smartphones After the Usability Evaluation the results of the evaluation sessions were communicated to the developers of the SIMPLI CITY Application Marketplace Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 113 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report 12 3 Detailed Documentation of the Findings 12 3 1 Usability Metrics 12 3 1 1 Task Completion Success Rate Overall performance of each participant was evaluated depending on his her ability to finish the task on his her own The task success rate score is the number of participants that finished the task successfully divided by the total number of participants Table 25 Task Completion Success Rate Participant Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 User 1
205. ther app store and take the best out of it would also add more information more information about what I need to use it If have to pay for it If there are in app purchases because there are other products for route navigation and other modes of transport where you have to pay something after downloading the app and I do not like this at all but That s it think 12 6 Observer Protocol Test User 1 09 09 2015 Timeline e Introduction 10 10 10 17 Explaining example task 10 17 10 21 User Test 10 17 10 47 Debriefing Interview 10 48 10 52 Task 1 e 10 21 Test user starts with task 1 e Types in the URL comes to the login page e 10 23 task was completed the test user says it was very easy Task 2 10 24 Test user starts with task 2 User reads the application list User chooses the first App in the list It is unclear for the user because there are only test comments The user tries the next App in the list There is no description User types in bicycle in the search area no result User clicks on help a word document is downloaded Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 126 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report e Question from the User reading the Word document What is PMA e After reading the docum
206. timized for smartphones 5 2 Results of the Evaluation of the Use Case 7 related App This section gives an overview of the results of the evaluation of the SIMPLI CITY App covering the Use Cases Routing to a big event and Personalised Traffic Restrictions Detailed information about the Usability Inspection and the Usability Evaluation of the PMA with the Use Case 7 related App can be found in Sections 13 and 14 of this document respectively 5 2 1 Results of the Usability Inspection by Experts Starting from July 2015 a Usability Inspection was conducted by SRM each time when a new release of the WP7 related SIMPLI CITY app Increased Mobility was available The results of each of these Usability Inspections were communicated to the developers immediately after each inspection and the app could be improved accordingly Usability Inspection in September 2015 revealed usability issues mainly related to account creation user registration specifying origin and destination of a trip and displaying the actual position More information about the Usability Inspection of the Increased Mobility app as well as a detailed list of the usability issues found is provided in Section 13 of this document 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 20 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201
207. tion The Ecolndex concept has also been used by Ascora within the course of the eco contest implementation of WP8 The Ecolndex has turned out to be a good gamification like approach to quickly summarize the indication of environmental impacts that a driving behavior has After the project it could be further increased in popularity by open sourcing the algorithm behind it or by providing its calculation as a service Application Marketplace Ascora has used the application marketplaces for experimenting with apps and their listings Findings have shown an interesting way to publish and promote apps which even goes beyond the project and could possibly easily be transferred to other use cases including use case specific app marketplaces such as e g the eHealth sector TIE Service Registry Partner TIE used Service Registry as part of the Service Runtime Environment in different tasks and especially in the implementation of the Service Development API IntelliJ Plugin Application Design Studio and for the REST Proxy The Service Registry component was nice to work with well documented and concisely organised Using modern technologies it was easy to extend when TIE needed extra functionality for its mini project A desired feature would be to have a full text search in the Registry however this has been added as part of the mini project Service Runtime Environment TIE used SRE as the container for hosting real backend services e g
208. tion Multiply the sum of the scores by 2 5 to obtain the overall value of the SUS score SUS scores have a range of 0 to 100 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 69 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report Bangor A et al this SUS score can be translated to users perceiving that the usability of the system is acceptable o k This is quite good for a prototype system but indicates that there s still some improvement of the system s usability necessary 8 3 2 Main Positive Aspects Mentioned by the Test Users During the debriefing interview at the end of the Usability Evaluation Sessions the test users stressed three features of the SIMPLI CITY system which they ve especially liked e The single screen form to create an application User 2 I liked the one screen form creation page It is a pain when you have to scroll for completing fields or you are in the middle of a wizard e Going through the tests User 4 It was like a quiz very easy and very fast e The simplicity to approve an application User 2 Very easy to manage to give the approval to an application would expect a more complex procedure User 1 find how to do it pretty fast so think this is a good think Sometimes yo
209. tion from Developers Point of View In general partners were positive about the SIMPLI CITY components they had to use in the course of development This comes to no surprise as the Functional and Technical specifications produced in the beginning of the project in cooperation with all partners were thought out in the smallest details laying out specifics of interfaces and underlying technologies that partners wanted to be used The majority of comments which were received from partners during this functional evaluation of SIMPLI CITY s developer tools is related to possible future enhancements some of which were considered as optional nice to have features in the earlier stages of the project and were skipped during the implementation phase because the development of other more important must have features occupied time and resources To summarise it can be said that apart from the fact that the documentation and guidance could be improved overall SIMPLI CITY partners software developers were quite satisfied with the way they had to work to develop within the SIMPLI CITY framework The organisation of components and the overall architecture had proven to be up to the expected industry standards 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 100 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No
210. to create the tests set them into a group and assign the group to an application On average this task was completed in 5 minutes and 47 seconds while the fastest participant needed 4 minutes and 52 seconds and the slowest required 6 minutes and 33 seconds 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 68 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report Task 2 required the users to perform the tests created in the previous task On average this task took 1 minute and 28 seconds while the shortest time was 1 minute and 1 second and longest time was 1 minute and 51 seconds Task 3 asked the participants to check the results of the tests and approve the application even some tests were in fail state On average this task was completed in 47 seconds with a fastest time of 44 seconds and a longest time of 1 minute and 5 seconds Task 4 was designed to users find the Publish button On average this task was completed in 48 seconds with a fastest time of 41 seconds and a longest time of 57 seconds Table 4 Participants Time On Task Participant Task 0 Task 1 Task 2 Task 3 Task 4 User 1 230 515 140 44 50 User 2 257 452 151 48 44 User 3 6 8 627 11 32 41 User 4 5 6 633 119 15 57 A
211. to place it in an existing AppStore e g Google or something like that User 2 Evaluation Session 2 Moderator How was it User 2 It was a bit frustrating At the beginning it started nice but then you could not get any further Moderator Was there anything that was good User 2 Ya at the beginning it was in principle fine the log in was fine Also the log out was fine but in the middle it did not work So that was good the beginning and the end Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 123 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report Moderator Was there something especially bad Or lets formulate it in another way What would you improve if you were the developer of this stuff User 2 When you arrive on the website where you have the selection of apps there should be a brief explanation where am That would be the most important thing saying This is SIMPLI CITY and you can download various apps here A brief explanation That would be the most important thing Moderator And the second important User 2 Second important is that for each app you have a very brief just one line so that you can easily see what it is about Then of course by clicking on the app you get
212. traffic X Henn i ave condition Support for querying Must weer diagnosis historic x Have Support for querying Must U205 impact factor on x Have traffic condition SRM Scenario Use Case Topic I 2 Optimization of financial resources Must e g avoiding tolls Have cheaper parking U206 Support suggestions for road trip Must v207 optimization if 5 Have conditions change Possibility to continue a previously Must U208 started trip planning X Hava on the same device or different device 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 40 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report tuifilled Explanation why not or only partially Requirement implemented Priority yes part no WORLD Scenario Use Case Topic ll 2 The information about traffic is taken from STAR system and the sensors provided by PMA and it s real time as Provision of real time far as the information provided by this Must U209 information aboutthe x System is real time However no Have current route updates are sent to the PMA and no real time information is notified to the app without a manual request as this is not supported by PMA Reproduction of Only
213. ty eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Table 14 Overview of the Test Users Introduction Questionnaire Public D7 3 Evaluation Report Study Thinking Aloud Test Test User User 1 User 2 User 3 User 4 Usability Eval Session Session 1 Session 2 Session 3 Session 4 Gender Male Male Male Male Age 20 30 20 30 20 30 20 30 Peculiarities None None None Glasses rons ct lenses Android Android Android Other Mobile Phone Ownership Smartphone Smartphone Smartphone Smartphone Smartphone Usage since gt 1 year gt 1 year gt year gt 1 year Smartphone expertise High High Medium High Android expertise High High High Low Usage of Apps High High Medium High Experience with App Store High Medium Medium High Mna LA Heard of Heard of Nothing Nothing Ever participated in a Usability No Yes Yes No Yes No Three test users self assessed their expertise of Android smartphones usage of apps and experience with App Stores was high One stated that he has a medium expertise All test users are still studying Computer Engineering so their expertise on the core of applications may be limited None of the test users was involved in the development of the SIMPLI CITY Applications Marketplace Two of the test users have heard of SIMPLI CITY before Two of the test users had participated in a Usability Stu
214. ty of apps crashes but a full list of older libraries Have with versions is not maintained on PMA side Backwards U101 compatibility of x ons gt ave services The API version can be changed when U102 Backwards x editing the App Manifest but App Should compatibility of API Design Studio will provide the last library Have available Robustness U103 Fault tolerance x rae ave at Must U104 Stability xX Hava Access to components Full Java libraries are provided for PMA side of the App For MMDI side some examples are provided and the IDE is U105 Access to the dialog capable of highlight properly the source Must system code but not against any library due to Have restrictions when creating projects on IntelllJ that only allow to select one SDK Access to cloud Must mos services x Have Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 31 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report does Explanation why not or only partially Requirement implemented Priority yes part no Access to sensors Access to sensors of the vehicle via car APIs when the U107 functionality is X ibd ave available Types fuel consumption speed acceleration Access to smart Must
215. u need to get use to the platform before do something it does not happened to me doing that 8 3 3 Main Usability Issues Revealed During the Evaluation 8 3 3 1 Main Usability Issues Mentioned by the Test Users During the debriefing interview at the end of the Usability Evaluation Sessions the test users mentioned several aspects of the SIMPLI CITY system which should be improved e Create App button User 1 The button is confusing did not know if should to click it before or after the completion of the fields User 2 The position of the button should be placed in the bottom left corner following the standards e Confusion between Add binary button and Upload binary button User 4 did not know what button push thought that first we should add a binary and then update it but when I pressed Add binary button did not know how to go back User 3 did a bad choice with Add binary button but it should be nice to have a message explaining what is Add binary for e The Submit button on the publication of an application Adjective ratings related to SUS scores 25 worst imaginable 38 poor 52 acceptable o k 74 good 85 Excellent 100 best imaginable Source Bangor A Kortum P Miller J Determining What Individual SUS Scores Mean Adding an Adjective Rating Scale in Journal of Usability Studies JUS Vol 4 Issue 3 May 2009
216. umentation Log out Create new test Fake app test 0 0 Group test for Environmental Awareness Category Fake app tests Rising application Safety and Insurance Test Group Testing for Rising the Driver s Category new testing category Comfort Application Test environment Category Group test for new Test app group Environmental Awareness Test for Accident prevention warning new testing category 9 Category Group Testing for Rising the Driver s C Test for Multimedia guide Test group for Prog3 integration and reproduction Category Group Testing for Rising Test for Notification of proximi points of interest Category Group Testing for Rising the Driver s C Test for Radio streaming access Q Category Group Testing for Rising the Driver s C Test for Route planning and Simpli city 2014 2015 Figure 9 Creation of tests Issue during Task 1 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 73 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report 1 Simpli city Application M x Q fi 3develtempos2Les simpli city demo approver assignTests zul v V a B E e a 4 simplicity aepeorerieing Approve for puealin Tosta Management d User User Approve Approve fo
217. ut an App on the SIMPLI CITY App Marketplace 91 9 3 5 Task 5 Put a Service on the SIMPLI CITY Service Marketplace 93 10 Functional Evaluation of SIMPLI CITY from Developers Point of View 95 10 1 Detailed Documentation of the Findings cccccceeeeeeeeeeeeeeeeeeeeeeeeeeeeeeneeeeeeees 95 10 2 Summary of the Results of the Functional Evaluation from Developers Point of View 100 11 Usability Inspection of the Road Users Part of the App Marketplace 101 11 1 Approach taken for the Usability Inspection sssesssseeeeeeee 101 11 1 1 Aim of the Usability Inspection seseseseeeeeeeeeeennnn 101 11 12 Team ot Evaluators uineam ie ae en D 101 uuu caseras PEE 101 Tes Method Applied ia o RO eoe Coe ioc ce eee roe cen errores 102 11 1 5 List of ICURISIICGS aise cssstsece tate tee sheet rotae Sle Asatte Lela Abas sate LAA 103 11 2 Summary of the Results of the Usability Inspection sssssssss 105 11 8 Detailed Documentation of the Findings 105 11 4 Screenshots Documenting Potential Usability Issues ssss 109 12 Usability Evaluation of the Road Users Part of the App Marketplace 110 12 1 Approach taken for the Usability Evaluation eeeeeeeeeeeeee 110 12 1 T Aim of the Usability Eval
218. ut is straight forward and this makes it easy for developers to use the tools The evaluation went smoothly as all features being evaluated worked as supposed Based on their experience the evaluators suggested several improvements which are listed in detail in the subsequent sections of this document However these suggestions were mostly cosmetic touches and user interface improvements According to the evaluators the overall logic of tooling is absolutely adequate and makes it easier for an experienced service or application developer to achieve her his goals 4 3 Results of the Functional Evaluation from Developers Point of View In order to get a good picture of the functionality of the SIMPLI CITY system from developers point of view in September 2015 the software developers among the SIMPLI CITY partners who had used parts of the SIMPLI CITY system in their work were asked to express their subjective opinions about the practical usefulness and applicability of the SIMPLI CITY components and make suggestions for further improvement In general partners were positive about the SIMPLI CITY components they had to use in the course of development This comes to no surprise as the Functional and Technical specifications produced in the beginning of the project in cooperation with all partners were thought out in the smallest details laying out specifics of interfaces and underlying technologies that partners wanted to be used The
219. ut only to sign in If the account is still to be created there s no way to create it The functionality is foreseen by the account menu but not working The user is re directed to a login mask on HTTP 130 83 245 97 4321 cxf users login but the system says Self registration is not allowed Ask administrator to register you No description on how to ask to administrator and who is the administrator are provided 2 Inthe settings menu there is the Apps Settings menu 1 4 3 that allows users to set Origin and Destination cities However after having set them the PMA Increased mobility gt Navigate asks again the user to select the destination city by VUI 3 Increased mobility gt Navigate Select the destination 1 11 3 city No GUI interaction allowed white screen no list of cities available 4 Increased mobility gt Navigate Select the destination 1 3 5 7 S7 S8 3 city Bologna gt Select the destination street in Bologna Via parigi gt Select the destination street number for Via Parigi in Bologna one the PMA shows a second main menu that is different and longer than the first one displayed as starting menu In this menu there is the possibility to to select time slot that makes sense select transport mode that makes sense and show route that does not work at all blocking the app and forcing the user to restart 5 Increased mobility gt Navigate Select the destination 1 11 3 city Bologna gt Select
220. valuators noted that the extensions provided for SIMPLI CITY specific work are logical and conscious the layout is straight forward and this makes it easy for developers to use the tools The evaluation went smoothly as all features being evaluated worked as supposed Based on their experience the evaluators suggested several improvements which are listed in detail in the subsequent sections of this document However these suggestions were mostly cosmetic touches and user interface improvements According to the evaluators the overall logic of tooling is absolutely adequate and makes it easier for an experienced developer to achieve her his goals 9 3 Detailed Documentation of the Findings The following 5 tasks are typical tasks which a developer would most likely want to complete using the SIMPLI CITY system 1 Develop an App using the SIMPLI CITY system 2 Develop a service using the SIMPLI CITY system 3 Utilise a service from the SIMPLI CITY Service Marketplace for creating an App 4 Put an App on the SIMPLI CITY Application Marketplace 5 Puta Service on the SIMPLI CITY Service Marketplace Therefore the evaluation team examined these tasks using the Cognitive Walkthrough method The following subsections list the findings of this Usability Inspection for each of these tasks 7 Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 80 146 http www simpli city eu Copyrig
221. verage TOT 410 547 128 47 48 8 3 1 3 Rating of the Overall System Usability After the completion of all tasks the participants were asked to fill in a standardised feedback questionnaire SUS System Usability Scale As already explained in Section 7 3 1 4 SUS yields a single number representing a composite measure of the overall usability of the system SUS scores have a range of 0 not usable at all to 100 perfectly usable system Table 16 Participants Rating of the Overall System Usability SUS Score Test Test User s Rating of the 10 Questions in the SUS questionnaire SUS Score User Q1 02 03 34 Q5 Q6 Q7 QB QI Q10 User 1 3 2 3 3 3 1 4 1 3 1 70 User 2 2 0 2 1 3 0 4 2 2 0 82 5 User 3 3 1 4 0 4 0 3 1 3 0 87 5 User 4 2 3 2 1 3 1 3 1 1 3 55 Analysis of the answers of the test users to the feedback questionnaire reveals that the SIMPLI CITY Applications Marketplace has got a SUS score of 73 75 According to According to User 0 Brooke SUS A quick and dirty usability scale http hell meiert org core pdf sus pdf To calculate the SUS score first sum the score contributions from each item Each item s score contribution will range from 0 to 4 For items 1 3 5 7 and 9 the score contribution is the scale position minus 1 For items 2 4 6 8 and 10 the contribution is 5 minus the scale posi
222. vides not much information Task 3 e 15 25 Test user starts with task 3 e Clicks on Search App Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 130 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 8 Public D7 3 Evaluation Report types in navigation in the search area clicks the search button no result types in navigate finds Increased Mobility App it was difficult because there was no result on navigation Task 4 15 27 Test user starts with task 4 going back to the overview Clicks on Environmental Awareness Rising there is not more information than in the overview He expects screenshots information about which rights need the App how many downloads more detailed description than 2 sentences e very easy to find the relevant page but no information provided there Task 5 15 29 Test user starts with task 5 Clicks on navigation App User 4 was logged out logged in again Clicks on the App and then on the Install button He thinks it does not work Clicks on the test app gets a message User 4 thinks he ll maybe get an E Mail for the App on the consumer email address e test user states Maybe it should pair with my account with my smartphone when click Install on the desktop e test user thinks that this task is very difficult because
223. w Help SIMPU CITY Cy testBackendService gt Sc gt main Ove Cotes Testhctvator 4 be aa Some c Oo or mter Teshatem ma Ca testBackentiervice Project SOK is wet defined Seup sox a ahaa package teat o 3 Osx v Omen public class d Oe i tet public void System cut println Viemma Service was started Ja e E engl i E W YesActhator g D TestService i Ca resources Diet D jra E mandet iml M pomar wemphcity properties Bl testBackendService emi Mh Eterna Libranes B Terminal s 1000 EJ Class TestActiestor is never used I qm pom suec Manuslpat Fen teamed E VEU DUXETIUSETYICE PU LIEUE FWOTRSEGUESHITHI UJ Vie DSURETRIDETVILE TECDELRETIUDET VICE YEL ETT Yarra G LEE vmprysrunranyser vi enmps java RERU nut r9 13 ES Ete Edit yew Mavigate Code Analyze Befactor Build Ryn Toots VCS Widow Help SIMPLI CITY Ca teeBackendService C uc C men gt D java Tiong D tee mpl ArhitraryServiceimet L bs a D rja 7 Op Qto miebtey E ArbiraryAcietorjea AbitarySenicejava amp ArhetrerySerceimplaave Maven Project Maven projects need to be imported im 1 Emas Quee Enable Auto Import H Ca teeBackendSenace package crg tee impl SRA PMH I D ides a Dx import org tee Ai ce m Aiivey ys teu he D man 4 Utecycle Die public class ArbitraryServicelmpl implements ArbitreryService clean m veids i E validate E m c String t
224. y the Usability Inspection were forwarded to the respective developer partners so that the SIMPLI CIY App Marketplace can be further improved Several potential usability problems were revealed by the evaluators The main issues were e The App Marketplace seemed to be not designed as a responsive website gt the page view did not adjust automatically to the size of the display and thus the page was only partially displayed on the small screen of the device e g the search button was not visible alongside with the respective communication field where the user was supposed to enter the search term and the install button was not visible when reading the App description and vice versa e t was not possible to install the App by clicking on the install button and no feedback was provided to the user in case the App could not be installed e The user was frequently thrown out from the marketplace and had to re login e There was no information about the marketplace not even an introductory text no description of the functionality no help supporting the user T Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 19 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserved Grant Agreement No 318201 SIMPLI CITY WP7 Public D7 3 Evaluation Report However the Usability Inspection revealed also some positive findings The main positiv
225. ystem is processing information e g via the cloud could help in cases where such processing is slow U19 Minimum manual Could configuration Have Personalization U22 Incremental No configuration by the end user is Could needed Have configuration The development environment is ble of handling strings and E U20 Multilingual x Capa n uL aa Have grammar components for different For Now languages f Will Not U21 Link voice commands x Have 10apps For Now User Experience Look amp Feel of Apps During the project it turned out that design changes should be in the hand of the corresponding app developer in order to be successful in the market The reason is twofold Firstly app developers need to integrate their apps into their company wide look amp feel and Ls secondly apps need to have the Unified Look amp Feel sms Should U23 within the project X possibility to react to new trends without Hav being bound to a Ul restriction of the underlying framework For example Google has released the Material Design guideline which quickly led to several changes for most apps As such SIMPLI CITY recommends the use of the Material Design but does not inforce a unified look amp feel Document Date Page D7 3 Evaluation Report v1 0 For Approval docx 2015 10 30 Status For Approval 24 146 http www simpli city eu Copyright SIMPLI CITY Project Consortium All Rights Reserv

Meeting the Increased Mobility Demand D7.3 - SIMPLI-CITY

Contents

Download Pdf Manuals

Related Search

Related Contents