DiamondHelp: A Graphical User Interface Framework for
Contents
1. di detected Healthy Choice HQ High Quality S VHS E WTPE 28 11 00AM CVBS video EXT2 0 30 total time LP Composite video E Saturday June 21 2004 SP Standard Play LP Long Play EP Extended Play Camcorder Front MITSUBISHI ELECTRIC RESEARCH LABORATORIES Figure 4 DiamondHelp for DVD recorder with several options for the user to choose from We call this the user s things to say bubble At the opening of the con versation there are only four choices What next oops done and help Notice that the last three of these options are specially laid out with icons to the right of the vertical line this is because these three options are always present see Predefined Things to Say section below At this point the user is free either to reply to the system s utterance by selecting one of the four things to say or to inter act with the direct manipulation part of the interface Unlike the case with traditional dialogue boxes the application GUI is never locked This flexibility is a key aspect of how we have combined conversational and direct manipulation interfaces in DiamondHelp In this scenario the user decides to request task guidance from the system by selecting What next in the things to say bubble As we have argued elsewhere 10 every inter active system should be able to answer a What can I do next question A Welcome to DiamondHel2. recognition is in general quite reliable because of the small number of very different utterances being recognized Fur thermore because of the way that the things to say mecha nism is implemented this extension requires no changes in the collaboration or manipulation plug ins for an application A second and much more ambitious approach is to dispense with the things to say bubble and try to recognize anything the user says which of course requires broad underlying speech and natural language understanding capabilities If one has broad natural language understanding another vari ation is to support unrestricted typed input from the user in stead of speech Finally it would be trivial as least from the user interface point of view to revert the conversational interface to its original function as a chat between two humans the user and a remote person This could be useful for example for re mote instruction or troubleshooting The system could even automatically shift between normal and live chat mode Future Work In addition to building DiamondHelp interfaces for several additional applications we also plan to conduct first infor mal and then formal user studies The purpose of the infor mal user studies will be simply to get feedback and sugges tions on our user interface design For the formal studies we would like to compare a DiamondHelp interface with a conventional interface for the same application an
3. incentive or motivation and they supply a cri terion by which the human executor of the instruc tions will know when he has accomplished his task In short instructions directed to computers specify courses instructions directed to human beings spec ify goals 6 The problem with the command view of user interfaces is that it demands too much knowledge on the part of the user Conventional systems attempt to compensate for this prob lem by adding various kinds of ad hoc help facilities tool tips wizards etc which are typically not integrated into any clear paradigm See the Related Work section for specific discussion of the relation of DiamondHelp to wizards and Microsoft s Clippy In contrast collaboration encompasses all of these capabilities within the paradigm Finally notice that Figure 2 does provides a place to include direct manipulation as part of the collaborative paradigm i e one can choose to implement the application GUI us ing direct manipulation This is in fact exactly what we have done in the user interface design for DiamondHelp de scribed in the next section USER INTERFACE DESIGN Our overarching goal for the DiamondHelp user interface design was to signal as strongly as possible a break from the conventional interaction style to the new collaborative paradigm A second major challenge was to accomodate what is inescapably different about each particular applica tion of DiamondHelp th
4. of each new product The DiamondHelp framework consists of three components an interaction paradigm based on task oriented human collaboration a graphical user interface design which combines conversational and direct manipulation interfaces e and a software architecture of reusable Java Beans The first two of these components are oriented towards the user the third component addresses the needs of the software developer Each of these components is described in detail in a section below Figure 1 shows an example of the Diamond Help user interface design for a combination washer dryer The immediate motivation for DiamondHelp has been our re cent attention to the usability crisis in high tech home prod ucts such as DVD recorders combination washer dryers refrigerator ovens programmable thermostats etc How ever DiamondHelp is not limited to home appliances it can be applied to any software interface DiamondHelp grows out of a longstanding research thread on human computer collaboration organized around the Collagen system 9 10 Our work on Collagen how ever has been primarily concerned with the underlying se mantic and pragmatic structures for modeling collaboration whereas DiamondHelp is focused on the appearance the look and feel that a collaborative system presents to the user Although Collagen plays a key role in our implementa tions of DiamondHelp applications see Software Architec ture s
5. tails of these GUI s are not important to the main point of this paper If fact there is nothing in the DiamondHelp frame work that guarantees or relies upon the application GUI be ing well designed or even that it follow the direct manipula tion paradigm though this is recommended The only way to guarantee this would be for DiamondHelp to automati cally generate the application GUI from a task model We have done some research on this approach 2 but it is still far from practical Furthermore to achieve our overall goal of consis tency across applications an industrial grade DiamondHelp would like conventional UI toolkits provide standard color palettes widgets skins etc However this is beyond the scope of a research prototype Two design constraints which are relied upon by the Dia mondHelp framework are 1 the user should be able to use the application GUI at any time and 2 every user action on the application GUI is reported to the system see Manipula tion Plug in section below Finally it is worth noting that for the three applications pre sented here it is possible to perform every function the appli cation supports entirely using the direct manipulation inter face i e totally ignoring the conversational window While this is a pleasant fact we also believe that in more complex applications there may be some overall advantage in relax ing this constraint This is a design issue we are still exp
6. their disposal 13 Our diagnosis of this problem identifies two fundamental un derlying causes the exhaustion of conventional interaction paradigms and the lack of consistency in user interface de sign This paper addresses both of these causes by introduc ing a new framework for building intelligent user interfaces called DiamondHelp The currently dominant human computer interaction paradigm is direct manipulation 11 which can be applied with great effectiveness and elegance to those aspects of an interface which afford natural and intuitive analogies to physical actions such as pointing dragging sliding etc In practice however most interfaces of any complexity also include an ad hoc and bewildering collection of other mechanisms such as tool bars pop up windows menus command lines etc To make matters worse once one goes beyond the most ba sic functions such as open save etc there is very little design consistency between interfaces to different products even from the same manufacturer The result is that it re DiamondHeln Make a new cycle gt Adjust wash agitation gt Help How can I help you Explain wash agitation Explain how to use DiamondHelp What next l Never mind oops done help Enter Cycle Name MITSUBISHI ELECTRIC RESEARCH LABORATORIES Figure 1 DiamondHelp for combination washer dryer quires too large an investment of the typical user s time to learn all the intricacies
7. wash agitation At a deeper level to be true to the spirit of the collaborative paradigm the content of the conversations in DiamondHelp i e both the system and user utterances should concern not only primitive actions Remove a schedule entry but also higher level goals Schedule a vacation and motivation Why When Collagen is used in the implementation of a Diamond Help application all the system utterances and the user s things to say choices are automatically generated from the task model given for the application see Collagen Collab oration Plug in section Without Collagen the appropriate collaborative content is provided by the application devel oper guided by the principles elucidated here An obvious limitation of the things to say approach is that there is only room for a relatively small number of choices inside the user bubble six or eight without some kind of nested scrolling which we would like to avoid Furthermore since we would also like to avoid the visual clutter of drop down choices within a single user utterance each thing to say is a fixed phrase without variables or parameters Given these limitations our design strategy is to use the di rect manipulation interface to enter variable data For ex ample in DiamondHelp for a washer dryer instead of the system asking How long is the dry cycle and generating a things to say bubble containing The dry cycle is 1
8. 0 min utes 15 minutes etc the system says Please set the dry cycle time points to the appropriate area of the direct manipulation interface and expects the user to enter the time via the appropriate graphical widget Figure 5 is a similar ex ample of using the direct manipulation interface to provide variable data in this case to select the thermostat schedule entry to be removed Predefined Things to Say Most of the things to say are specific to an application How ever a key part of the generic DiamondHelp design is the following set of user utterances which should have the same meaning in all applications What next Never mind Oops Done and Help In addition to save space and enhance appearance we have adopted a special layout for the last three of these utterances which are always present What next has already been discussed above Never mind is a way to end a question without answering it see Figure 1 Oops Done and Help each initiate a subdi alogue in which the system tries to determine respectively how to recover from a mistake what level of task goal has been completed or what form of help the user desires When Collagen is used in the implementation of Diamond Help the semantics of the predefined things to say are au tomatically implemented correctly otherwise this is the re sponsibility of the collaboration plug in implem
9. MITSUBISHI ELECTRIC RESEARCH LABORATORIES http www merl com DiamondHelp A Graphical User Interface Framework for Human Computer Collaboration Rich C Sidner C Lesh N Garland A Booth S Chimani M TR2004 114 September 2004 Abstract DiamondHelp is a reusable Java framework for building graphical user interfaces based on the collaborative paradigm of human computer interaction DiamondHelps graphical design com bines a generic conversational interface adapted from online chat programs with an application specific direct manipulation interface DiamondHelp provides 4 things to saymechanism for use without spoken language understanding it also supports extensions to take advantage of speech technology DiamondHelp software architecture factors all application specific content into two modular plug ins one of which includes Collagen and a task model International Workshop on Smart Appliances and Wearable Computing IEEE International Conference on Distributed Computing Systems Workshops This work may not be copied or reproduced in whole or in part for any commercial purpose Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following a notice that such copying is by permission of Mitsubishi Electric Research Laboratories Inc an acknowledgment of the authors and individual contributions to
10. ad range of interaction modes depending on the relative knowledge and initiative of the system versus the user For example tu toring aka intelligent computer aided instruction is a kind of collaboration in which the system has most of the knowl edge and initiative At the other end of the range is intelli gent assistance wherein the user has most of the knowledge and initiative Furthermore a collaborative system can eas ily and incrementally shift within this range depending on the current task context This kind of mixed initiative is the exception rather than the rule in conventional interfaces Second and at a deeper level the primary role of com munication in collaboration is not for the user to tell the system what to do the traditional ccommands but rather to establish and negotiate about goals and how to achieve them Licklider observed this fundamental distinction over 40 years ago in a classic article which is well worth revisit ing 3 Compare instructions ordinarily addressed to intel ligent human beings with instructions ordinarily used with computers The latter specify precisely the in dividual steps to take and the sequence in which to GOALS oer etc O ogi o 2 Q o communication 2 SYSTEM gt USER observe observe 7 action w action Application GUI Figure 2 The collaborative paradigm take them The former present or imply something about
11. an human communication we adopted the scrolling speech bubble metaphor often used in online chat programs which support human human com munication The bubble graphics nicely reflect the collabora tive paradigm s symmetry between the system and the user discussed above This graphical metaphor also naturally ex tends to the use of speech synthesis and recognition technol ogy see Extensions and Variations section below The top half of every DiamondHelp interface is thus a con versation chat between DiamondHelp the system rep resented by the Mitsubishi three diamond logo on the left and the user represented by the human profile on the right All communication between the user and system takes place in these bubbles there are no extra toolbars pop up menus etc The basic operation of the conversational part of the Dia mondHelp interface is as follows Let s start with the open ing of the conversation which is the same for all Diamond Help applications A Welcome to DiamondHelp 3 2 What next oops done help a After the system says its welcome a user bubble appears Make a recording gt Select recording quality Since we are recording from a digital signal suggest changing the recording quality from LP to HQ Change recording quality to HQ Keep recording quality at LP Why bee 2 cops done help Recording Controls Recording Source Chapter Information Recording Quality Cable
12. cate by selec tion in the direct manipulation interface which schedule en try is to be removed Notice that the user s things to say bubble below includes only three choices What next is not included because the system has just told the user what to do next 7 You can 0 Remove a schedule entry 0 A Please select the entry you want to remove 2 tage 2 N oops done help r Although the system is expecting the user next to click or touch in the direct manipulation interface see Figure 5 the user is still free to use the conversational interface e g to ask for help Scrolling History As in chat windows a DiamondHelp user can at any time scroll back to view parts of the conversation that have moved off the screen This is particularly useful for viewing ex planatory text which can be quite long e g suppose the system utterance in Figure 4 was several lines longer The oval beads on each side of the upper half of the screen in Figures 1 3 and 4 are standard scroll bar sliders We also support scrolling by simply dragging anywhere on the lined background of the upper window DiamondHeln lt Thermostat Remove a schedule entry Remove a schedule entry 2 Please select the entry you want to remove Sleep Sleep FEBRUARY 2005 MTITWTFSS 12345 6 7 8 9 10 11 12 13 21 22 23 24 25 26 27 28 Midnight j Trash MITSUBISHI ELECTRIC RESEARCH LABORATORIES Figure 5 Selection
13. covering new devices connected to a network loading the appropriate plug ins etc which are beyond the scope of this paper Manipulation Plug in Notice that the manipulation plug in sticks out of the Di amondHelp framework box in Figure 6 This is because it is directly responsible for managing the application specific portion of the DiamondHelp interface DiamondHelp simply gives this plug in a Swing container object corresponding to the bottom half of the screen We recommend that the manipulation plug in provide a direct manipulation style of interface implemented using the model view architecture as shown by the dotted lines in Fig ure 6 In this architecture all of the semantic state of the interface is stored in the model subcomponent the view sub component handles the application GUI Regardless of how the manipulation plug in is implemented internally it must provide an API with the DiamondHelp framework which includes two event types outgoing events user action observation which report state changes result ing from user GUI actions and incoming events system ac tion which specify desired state changes Furthermore in order to preserve the symmetry of the collaborative paradigm see Figure 2 it is the responsibility of the plug in to update the GUI in response to incoming events so that the user may observe system actions As an optional but recommended feature the manipulation plug in may also p
14. d measure the usual parameters such as task completion time solution quality and user satisfaction One approach for the com parison condition is to use the same direct manipulation in terface as in the DiamondHelp condition but with a written user manual instead of DiamondHelp REFERENCES 1 E DeKoven Help Me Help You Designing Support for Person Product Collaboration PhD thesis Delft Inst of Technology 2004 2 J Eisenstein and C Rich Agents and GUIs from task models In Proc Int Conf on Intelligent User Inter faces pages 47 54 San Francisco CA Jan 2002 3 N Lesh J Marks C Rich and C Sidner Man computer symbiosis revisited Achieving natural com munication and collaboration with computers IEICE Transactions Inf amp Syst E87 D 6 1290 1298 June 10 11 12 13 N Lesh C Rich and C Sidner H Lieberman C Rich and C Sidner 2004 N Lesh C Rich and C Sidner Using plan recognition in human computer collaboration In Proc 7th Int Conf on User Modelling pages 23 32 Banff Canada June 1999 Collaborating with focused and unfocused users under imperfect communi cation In Proc 9th Int Conf on User Modelling pages 64 73 Sonthofen Germany July 2001 Outstanding Paper Award J C R Licklider Man computer symbiosis IRE Trans Human Factors in Electronics HFE 1 4 11 Mar 1960 Integrating user interface agents with conventio
15. day select last day Figure 7 Task model for programmable thermostat the task bar All of the graphical aspects of the conversational window are managed by DiamondHelp framework code It is also an important feature of the DiamondHelp architec ture that the collaboration plug in developer does not need to be concerned with the graphical details of the application GUI The collaboration plug in developer and the manipu lation plug in developer need only to agree on a semantic model of the application state For a very simple application the collaboration plug in may be implemented by a state machine or other ad hoc mecha nisms However in general we expect to use the Collagen version described in the next section Collagen Collaboration Plug in The Collagen version of the collaboration plug in includes an instance of Collagen with a little bit of wrapping code to match the collaboration plug in API Collagen has already been extensively described in the literature 9 10 We will only highlight certain aspects here which relate to Diamond Help The most important reason to use the Collagen collaboration plug in is that the application developer only needs to pro vide one thing a task model All four outputs of the collab oration plug in described above are then automatically gen erated by Collagen A task model is an abstract hierarchical partially ordered representation of the actions typically performed to achi
16. e constructs needed to program a thermostat are very different from those needed to program a washing machine while preserving as much consistency as possible in the collaborative aspects of the interaction If a someone knows how to use one DiamondHelp application they should know how to use any DiamondHelp application In order to address these concerns we divided the screen into two areas as shown in the example DiamondHelp interfaces of Figures 1 3 and 4 Figure 1 has been implemented in Java Thermostat S Add a schedule entry Modify a schedule a a i af 2 Remove a schedule entry Schedule a vacation oops done help Sleep Sleep Sleep Sleep Sleep 6 00am 6 00 am 0 a Home Home Home 0am 9 00am 9 00 az FEBRUARY 2005 MTWTFSS 123s 6423 6 7 8 9 10 11 12 13 21 22 23 24 25 26 27 28 MITSUBISHI ELECTRIC RESEARCH LABORATORIES Figure 3 DiamondHelp for programmable thermostat Figures 3 and 4 are Flash simulations The top half of each of these screens is the same distinctive DiamondHelp con versational interface The bottom half of each screen is an application specific direct manipulation interface Dividing the screen into two areas is of course not new our con tributions are the specific graphical interface design and the reusable software architecture described below Conversational Interface To express and reinforce the human computer collaboration paradigm which is based by hum
17. ection the DiamondHelp framework can also be used independently of Collagen INTERACTION PARADIGM Although the concept of an interaction paradigm is some what abstract and difficult to define formally it is crucial to the overall development of interactive systems In particu lar the consistent expression and reinforcement of the inter action paradigm in the user interface design what the user sees leads to systems that are easier to learn and use The interaction paradigm also provides organizing principles for the underlying software architecture which makes such sys tems easier to build The collaborative paradigm for human computer interaction illustrated in Figure 2 mimics the relationships that typi cally hold when two humans collaborate on a task involving a shared artifact such as two mechanics working on a car engine together or two computer users working on a spread sheet together This paradigm differs from the conventional view of interfaces in two key respects First notice that the diagram in Figure 2 is symmetric be tween the user and the system Collaboration in general in volves both action and communication by both participants and in the case of co present collaboration which is the fo cus of DiamondHelp the actions of each participant are di rectly observed by the other participant One consequence of this symmetry is that the collaborative paradigm spans in a principled fashion a very bro
18. entor see Software Architecture section Task Bar An additional component of the DiamondHelp conversa tional interface is the task bar which is the single line lo cated immediately above the scrollable history and below the DiamondHelp logo in Figures and 4 the task bar in Fig ure 3 happens to be blank at the moment of the screen shot For example in Figure 1 the contents of the task bar reads Make a new cycle gt Adjust wash agitation gt Help The task bar like the scrollable history and the predefined What next utterance is a mechanism aimed towards help ing users when they lose track of their context which is a common problem in complex applications When Collagen is used in the implementation of DiamondHelp the task bar is automatically updated with the path to the currently active node in the task model tree see Figure 7 Otherwise it is the responsibility of the collaboration plug in implementor to update the task bar with appropriate task context informa tion Currently the task bar is for display only However we have considered extending the design to allow users to click touch elements of the displayed path and thereby cause the task focus to move This possible extension is closely related to the scrollable history restart extension discussed above Application GUI The bottom half of each screen in Figures 1 3 and 4 is an application specific direct manipulation interface The de
19. eve goals in the application domain Figure 7 shows a fragment of the task model for the programmable thermostat applica tion in Figure 3 Collagen currently provides a Java exten sion language for defining task models We are also consid ering an XML version of this language System utterances and actions are produced by Collagen s usual response generation mechanisms 8 The semantics of some of the predefined DiamondHelp user utterances such as Oops Done and Help are captured in a generic task model which is used by Collagen in addition to the application specific task model provided Each of these pre defined utterances introduces a generic subgoal with an asso ciated subdialogue using Collagen s discourse stack Other predefined utterances such as What next and Never mind are built into Collagen The third collaborative plug in output produced by Collagen is the user s things to say Basically the things to say are the result of filtering the output of Collagen s existing dis course agenda generation algorithm The first step of fil tering is remove everything except expected user utterances Then if there are still too many choices a set of application independent heuristics are applied based on the type of com municative act proposal question etc Lastly predefined utterances are added as appropriate The further details of this algorithm need to be the topic of a se
20. in the direct manipulation area see small triangular cursor in first column of schedule An interesting DiamondHelp extension to explore is al lowing the user to restart the conversation at an earlier point in the history and move forward again with different choices This could be a good way to support the backtrack ing paradigm for problem solving and is also closely related to our earlier work on history based transformations in Col lagen 9 Things to Say The user s things to say bubble is partly an engineering com promise to compensate for the lack of natural language un derstanding see Extensions and Variations section below It also however partly serves the function of suggesting to the user what she can do at the current point 12 1 In the underlying software architecture each choice in the things to say bubble is associated with the semantic repre sentation of an utterance When the user selects the de sired word or phrase the architecture treats the choice as if the missing natural language understanding system pro duced the associated semantics From a design perspective this means that the wording of the text displayed for each things to say choice should read naturally as an utterance by the user in the context of the ongoing conversation For example in Figure 1 in reply to the system s question How can I help you one of the things to say choices is Explain wash agitation not
21. lor ing SOFTWARE ARCHITECTURE In contrast to most of the discussion of above which fo cuses on the the user s view of DiamondHelp this section addresses the needs of the software developer The overar ching issue in DiamondHelp s software architecture is reuse i e factoring the code so that the developer of a new Dia mondHelp application only has to write what is idiosyncratic to that application while reusing as much generic Diamond Help framework code as possible system bubble task bar user bubble user choice Explain wash agitation What next F se DiamondHelp Never mind oops done hein user manipulation application GUI update DiamondHelp system utterance Collaboration things Plug in to say user utterance user action system action observation Manipulation Plug in Figure 6 DiamondHelp software architecture Figure 6 shows our solution to this challenge which we have implemented in Java Beans and Swing Dotted lines in dicate optional but recommended components All the application specific code is contained in two plug ins which are closely related to the two halves of the user in terface The rest of the code shaded region is generic Dia mondHelp framework code In addition to the issues discussed in this section there are a number of other standard functions of plug and play ar chitectures of which DiamondHelp is an instance such as dis
22. nal applications In Proc Int Conf on Intelli gent User Interfaces pages 39 46 San Francisco CA Jan 1998 C Rich N Lesh J Rickel and G A A plug in archi tecture for generating collaborative agent responses In Proc Ist Int J Conf on Autonomous Agents and Multi agent Systems Bologna Italy July 2002 Collagen A collaboration manager for software interface agents User Modeling and User Adapted Interaction 8 3 4 315 350 1998 Reprinted in S Haller S McRoy and A Kobsa editors Computational Models of Mixed Initiative Interaction Kluwer Academic Norwell MA 1999 pp 149 184 C Rich C Sidner and N Lesh Collagen Applying collaborative discourse theory to human computer inter action AI Magazine 22 4 15 25 2001 Special Issue on Intelligent User Interfaces B Shneiderman and C Plaisant Designing the User Interface Strategies for Effective Human Computer In teraction Addison Wesley Reading MA 2005 C L Sidner and C Forlines Subset languages for con versing with collaborative interface agents In Int Conf on Spoken Language Processing Sept 2002 S H Wildstrom Technology amp you Lessons from a dizzying decade in tech BusinessWeek June 14 2004
23. p What a ds i ie As part of the interface animation whenever the user makes a things to say selection by clicking or touching the de sired word or phrase the not selected items are erased and the enclosing bubble is shrunk to fit only the selected word or phrase Furthermore in preparation for the next step of the conversation the completed system and user bubbles are grayed out and scrolled upward if necessary to make room for the system reply bubble and the user s next things to say bubble 6 Welcome to DiamondHelp e A What next 2 DiamondHelp applications always reply to What next by a You can utterance with the actual options for the pos sible next task goals or actions presented inside the user s things to say bubble 0 What next You can A Add a schedule entry Remove a schedule entry Schedule a vacation Modify a schedule entry 7 oops done help In this scenario from DiamondHelp for a programmable thermostat see Figure 3 the direct manipulation interface shows the family s schedule for the week The entries in this schedule determine the thermostat s temperature settings throughout the day The user chooses the goal of removing a schedule entry A 6 What next 6 You can g A Add a schedule entry Remove a schedule entry Schedule a vacation Modify a schedule entry y gt oops done help The system replies by asking the user to indi
24. parate paper Collagen s powerful plan recognition capabilities 4 5 are used to keep the task bar up to date with respect to the given task model RELATED WORK Although an increasing number of intelligent systems have been built recently using the collaborative paradigm espe cially tutoring systems none have focused specifically on the user interface design issues we discuss here nor do any include a generic software architecture that is comparable with DiamondHelp Use of a textual things to say list together with Collagen and speech recognition has been studied by Sidner and For lines 12 for a personal video recorder interface and by Dekoven 1 for a programmable thermostat In terms of generic software architectures for connecting arbitrary applications with help facilities a major exam ple is Microsofts well known Clippy Clippy s interac tion paradigm might be described as watching over the user s shoulder and jumping in when you have some advice and is in our opinion the main reason for Clippy s well known unpopularity among users In contrast the collabo rative paradigm underlying DiamondHelp emphasizes ongo ing communication between the system and user to maintain shared context The second major example of generic help architectures are wizards such as software installation wizards etc which come in many different forms from many different manufac turers Our feeling about wi
25. rovide the Dia mondHelp framework with the graphical location of each in coming state change event so that DiamondHelp can move a cursor or pointer to that location to help the user observe system actions Note that the content of both incoming and outgoing state change events is specified in semantic terms e g change dry cycle time to 20 min not button click at pixel 100 200 Lieberman 7 further discusses this and other issues related to interfacing between application GUI s and intelligent sys tems Collaboration Plug in Basically the responsibility of the collaboration plug in is to generate the system s responses actions and utterances to the user s actions and utterances Among other things this plug in is therefore responsible for implementing the seman tics of the predefined utterances see next section The collaboration plug in has two inputs observations of user actions received from the manipulation plug in and user utterances resulting from user choices in the things to say bubble It also has four outputs system actions sent to the manipulation plug in system utterances which go into system bubbles things to say which go into user bubbles and the task bar contents Notice that the collaboration plug in is responsible for pro viding only the content of the system and user bubbles and program thermostat add entry remove entry modify entry schedule vacation select first
26. the work and all applicable portions of the copyright notice Copying reproduction or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories Inc All rights reserved Copyright Mitsubishi Electric Research Laboratories Inc 2004 201 Broadway Cambridge Massachusetts 02139 DiamondHelp A Graphical User Interface Framework for Human Computer Collaboration Charles Rich Candy Sidner Neal Lesh Andrew Garland Shane Booth Markus Chimani Mitsubishi Electric Research Laboratories 201 Broadway Cambridge MA 02139 USA rich merl com ABSTRACT DiamondHelp is a reusable Java framework for building graphical user interfaces based on the collaborative paradigm of human computer interaction DiamondHelp s graphical design combines a generic conversational interface adapted from online chat programs with an application specific direct manipulation interface DiamondHelp provides a things to say mechanism for use without spoken language understanding it also supports extensions to take advantage of speech technology DiamondHelp s software architec ture factors all application specific content into two modular plug ins one of which includes Collagen and a task model Keyword Intelligent assistants for complex tasks INTRODUCTION technology remains far too hard for most folks to use and most people can only utilize a tiny fraction of the power at
27. zards is that they embody the right paradigm i e interactively guiding the user but in too rigid a form DiamondHelp subsumes the capabilities of wiz ards but also allows users to take more initiative when they want to CONCLUSION The contributions of this work to intelligent user interfaces are twofold First we have explicated a novel user interface design which expresses and reinforces the human computer collaboration paradigm by combining conversational and di rect manipulation interfaces Second and more concretely we have produced the DiamondHelp software which can be used by others to easily construct such interfaces for new ap plications Extensions and Variations A number of small extensions to DiamondHelp have already been discussed in the body of this paper Another general set of extensions have to do with adding speech and natural language understanding technology Adding text to speech generation to the system bubble is a very easy extension which we have already done its an op tional feature of the software Furthermore we have found that using a system with synthesized speech is surprisingly more pleasant than without even when user input is still by touch or click Adding speech recognition can be done in two ways The first more limited way is to use speech recognition to choose one of the displayed things to say We have already done this another optional feature and have found that the speech
Download Pdf Manuals
Related Search