Home

FrameScript: A Multi-modal Scripting Language

1. 4 4 3 Generic Frames 444 Instance Erames 4 4 5 451 M nu s s kee magma W W Sk a up ue 0 2 Domas cos d l n AA 4 54 nc ig l 2 uo s w w Q diw E la 4 5 5 Conversation GUI 5 MicaBot 5 1 Speech Alternatives 5 2 Simple TextAgentl 5 3 MicaR mnnerl a a of AA BUR Bi yQ Se s A BNE B_ Built in Subroutines 0 1 Operators a owe odd bea a OE SELLA Ded Ea Se ee B 1 23 forall B 1 24 5 1 B 1 25 Precedence Table B 2 General Functions B 2 1 trace B 2 3 atom 2 24 defined B 2 5 undefined B 2 6 set B 2 7 number 2 8 integerl B 2 9 list B 2 10 cons B 2 11 member B 2 12 head B 2 13 tail B 2 14 nthi B 2 15 lengthi 3 2 16 append 3 2 17 delete B 2 18 fixrdr B 2 19 13 2 20 print 3 2 21 error B 2 22 ask B 2 23 eval B 2 24 quotel 3 2 25 load 5 B 2 28 closeoutputi 3 2 29 primt asS text se ash eee bee Pee RUR UR B 3 1 frame B 3 2 generic B 3 3 instance B 3 4 Instances of B 3 5 put B 3 6 replace B 3 7 remove 3 3 8 destroy B 4 1 script
2. C 3 10 formatComment C 3 11 unformatComment C 3 12 unformatComment C 3 13 checkAllReferences D Serialisation List of Figures 1 Agents communicate via MICA s blackboard a a X 11 Match Component Example 13 Topic Fxamplel 2057777 257777 17 Evaluation Statementsi 18 Function Definition Example 19 Function Type Checking Example 20 MaxModule Example 21 Example Multi modal Script 22 Message Dialog 23 Question Dialog 24 RDR Maintenance GUI 25 Frame Browser 26 Instance Frame Browser 27 Function Browser dede Q Q d 29 Domain Browser 30 Script Daemon Browser 31 Abnormal Script Browser 32 Function Browser 33 Conversation GUI 2775 7777771 35 Agent 36 MicaRunner Startup Script Example List of Tables 1 Overview This document describes FrameScript a language for creating multi modal user interfaces FrameScript is a multi paradigm language that enables rule based programming as well as frame representations and simple functional evaluation It
3. sade yoseds TWOJ 3811 1e ua w eyep eweb uonoe baujie wa sabessaw mau uy Bess ul andunxe The text object activates the natural language agent This is a FrameScript program that understands email requests and translates the text into an object that represents 8 command to the email agent The result of the email request goes onto the blackboard and is translated into text by the email to text con verter Finally the text to speech agent creates the audio output that is sent to the output device 1 2 FrameScript Some of the agents that connect to MICA may be generic For example a speech recognition system such as Dragon Naturally Speaking DNS is an off the shelf program that only requires the addition of a stub that allovvs it to be invoked by the communication of objects through MICA Other agents will be application specific An example is a conversational agent that handles dialogues specific to the application such as an in car controller for non critical devices such as the radio or air conditioner FrameScript is a language that allows developers to rapidly script domain specific interactions for particular application MICA and the FrameScript in terpreter are separate programs that may be run completely independently However their usefulness is greatly enhanced when used together Application dependent agents can be written
4. 768 click lt goto demo a_click click put a_click request lt put a click 71 request describe lt put request goto describe get_object click describe house it is a house goto demo road it is a road goto demo gt 1 don t knov what it is goto demo 1 Figure 21 Example Multi modal Seript 36 e That when the user clicks on an object a click event is generated e There exists a function get_object that processes the click event to determine what it was the user selected This example expects two types of events Either the user will say what is this or the user will click on something All other speech and events are ignored If the user says what is this the demo script stores the user s request for 8 description It then checks if the user has clicked on something If the user has clicked on something then we reenter the script using the previous click as the input When a click event occurs preceding clicks are deleted and the script checks if the user has requested a description if they haven t the click is stored in the slot a click If the user has requested a description then the system changes context to the describe context and uses the object clicked on as the input How the get object O function determines what was clicked is irrelevant VVhen the describe context encounters an
5. var var list var list atom var list compound atom arg list arg list expression list arg list 1 rdr expression if expression then expression because case except rdr expression else expression case atom 55 B Built in Subroutines In order for a scripting language to be 01 much use it needs to provide a number of standard functions for performing basic operations on its basic types As such FrameScript provides a set of mathematical and logical operations and functions for interacting with lists and frames While FrameScript is an untyped language its operations and functions often operate on specific types of terms The following function descriptions give the types that each of FrameScript s standard functions expect If they are given parameters of types they are not expecting they will throw an error Below is a list of the parameter types that the standard functions may expect atom an atom boolean either true or false domain a domain filename the name 01 a file it can be in the form of a string or an atom generic a generic frame instance an instance frame unless otherwise specified this includes scripts integer an integer list a list modulename the name of 8 module it can be in the form of a string or an atom number a number list a list pattern a pattern script a script slot
6. Ripple Down Rules RDRs 2 1 are used in FrameScript to provide conditional branching evaluation paths They also provide a basic mechanism for knowledge acquisition A very simple RDR will look like if condition then conclusion This RDR will evaluate and return the conclusion if the condition evaluates to true otherwise it returns nothing An RDR can optionally have a cornerstone case an exception or an alter native An RDR with all 3 will look like if condition then conclusion because cornerstone_case except exception else alternative The cornerstone case is the name of the frame that represents the case that caused the rule to be constructed The exception is a rule constructed to handle cases where the condition is true but due to other conditions require a different conclusion The alternative is an expression to be evaluated and returned if the condition is false NOTE If the condition conclusion alternative of an RDR is an expression list it needs to be bracketed 2 3 11 Functions Functions are used in many programming languages to encapsulate computa tions so that programs can be written regardless of the implementation used 29 to perform the computation and the allow for the code to be reused relatively simply Figure 18 shows the definition of a simple function that finds the maximum of 2 numbers max X Y if X gt Y then X else Y 33 Figure 18 Function Definition Example As FrameScript i
7. 2 3 12 Forall Loops VVhen using lists it is not uncommon to need to perform some evaluation for each element in the list For this reason FrameScript provides the forall loop that will loop through the elements of a list evaluating a given expression for each element The forall loop looks like forall VARIABLE in LIST EXPRESSION The result of evaluating the forall loop is a list where each element is the result of evaluating the given expression for the corresponding element in the list argument 01 the 02811 loop For example the result of evaluating forall in 123 118 2 34 It is possible to nest forall loop The result of evaluating nested forall loops will be a list of lists of NOTE If the body of a forall loop is an expression list it needs to be bracketed 2 3 13 Variables When entering the body of a function or the body of a forall loop FrameScript will define and assign the value to a local variable using the parameter name in the function specification or forall loop On occasion it may be necessary to define other local variables This can be done using the var A var statement can be used in the body of either a function a forall loop or a daemon to declare local variables The use of a var statement necessitates that the body be an expression list where the var statement is the first expres sion in the list If a var statement is not part of an expression list or not the first expression of the lis
8. B 4 4 register 4 5 0 0 B 4 6 current context 3 4 7 previoustopic check framel B 4 8 new_event B49 bot w se ee won won won won B 4 10 match B 4 11 failsafe B 4 12 question B 4 13 return B 5 1 dialog message B 5 2 dialog question B 5 3 frame browser B 5 4 script_browserl B 5 5 fixrdr gui B 6 MicaBot B 6 1_ micabot B 6 2 mica connect B 6 3 mica register B 6 4 mica_unregister B 6 5 mica read mob B 6 6_ mica write mobil B 6 7 mica delete mob 13 0 8 mica_query B 6 9 mica write wait for reply 13 6 10 get mob name B 6 11 current micabot C 1 1 check alternatives C 1 2 check 1 C 1 3 check boolean C 1 4 check compound C 1 6 check exprlist C 1 7 C 1 9 check instance C 1 11 check list C 1 12 check number C 1 14 check rdr C 1 15 check script C 1 17 check string 602 1 6 B 5 GUIs C Utility Functions C 1 Subroutine Argument Type Checking C 2 Files Modules 6 2 0 vloop e www b n a a a ta tata tad C 3 Miscellaneous C 3 1 compress C 3 2 getMessage C 3 3 getName C 3 4 getPattern C 3 5 sortAtomlist C 3 6 IOError C 3 7 isPattern 577 C 3 9 isPatternElement
9. IF a script is selected in the script browser s tree Either by its name or as a topic in a domain That script s details will be displayed on the right The script browser will display any trigger specifically defined for the script the script s failsafe the script s parents the script s slots the script s daemons and the rules defined in the script The script browser can be used to change a script s trigger failsafe parents and slots The rules for the script can be reordered and removed and new rules can be added When a rule is selected the pattern and response for the rule are shown Both the pattern and response can be edited Script Daemons The Daemons tab for a script shows the on_entry and on_exit daemons Script Browser Dx Script Name test script Trigger a D test script 3 functions Parents 2 Emax D max D mayra Failsafe Stots Daemons on_entry Daemon on exit Daemon Rule Comment Pater B C Response y Figure 30 Script Daemon Browser defined for a script and allows them to be edited Abnormal Scripts Some scripts do not use the standard rule response mechanism for processing inputs If such a script is selected the recognition daemon for that script will be displayed If a users selects View Recognition Daemon in the menu the actual recognition daemon that implements the rules for the script will be shown Example
10. gt gt 1 lt lt Table 3 Pattern Examples 2 3 8 Sequences Sequences are a list of patterns When a sequence is evaluated the result is that one of the patterns in the list is evaluated and returned Each subsequent evaluation of the sequence will result in the evaluation of the next pattern 28 in the list When the last pattern is evaluated the sequence returns to the beginning Sequences are found within patterns and are enclosed by and The patterns within the sequence are separated by eg a sequence 1 a pattern another pattern The patterns within a sequence can have conditions When the sequence tries to evaluate a pattern and the pattern has a condition the condition is tested If it is true then the pattern is evaluated if not the sequence tries the next pattern in the list The sequence will continue trying then next pattern in the list until it finds one with no conditions or one whose condition holds true Example condition gt the condition is true not condition gt the condition is false 2 3 9 Alternatives Alternatives are a list of patterns When an alternative is evaluated the re sult is that a randomly selected pattern in the list is evaluated and returned Alternatives are found within patterns and are enclosed by and The patterns within the sequence are separated by eg an alternative a pattern another pattern 2 3 10 Ripple Down Rules
11. the failsafe of the current context the current topic the failsafe of the current topic 5 a p w the failsafe for the domain 22 When an input is compared to the rules of a script the input is first compared to the rules specifically defined by the script If none of these rules match the input is matched against the rules of the script s parents The rules of the scripts are tried in top to bottom order 2 2 10 Daemons FrameScript allows daemons to be defined that are executed when moving from one context to another These are the on entry and on_exit daemons These daemons are attached to scripts and are evaluated when the current context is changed The on entry daemon for a script is run whenever the script becomes the current context The on exit daemon is run when a script stops being the current context The on entry and on_exit daemons run whenever the current context changes this could be because of a trigger firing a goto command or a previous topic command Daemons are inherited normal contezt dictate lt goto dictation script gt do command 0 dictation_context on_entry set_speech_rec_dictate true on_exit set_speech_rec_dictate false finished gt use_saved_text goto normal_context gt save text o Figure 15 Script Daemons Example Figure 115 gives a small example that uses the on_entry and on_exit dae mons With this example the environment is that we have a speech re
12. the name of a slot it must be an atom that will not get evaluated term a term it can be any type of term including atom string number pattern RDR expression ist variable the name of a variable it must be an atom that will not get evaluated B 1 Operators FrameScript provides a number of mathematical and logic operations that can be used within scripts and frames 56 number 10 Standard unary mathematical addition operator number number numerical operand number number2 50 Standard mathematical addition operator number number first numerical operand number2 second numerical operand number 10 Standard unary mathematical subtraction operator number number numerical operand number number2 50 Standard mathematical subtraction operator number number first numerical operand number2 second numerical operand number number2 40 Standard mathematical multiplication operator number number first numerical operand number2 second numerical operand number number2 40 Standard mathematical division operator number number first numerical operand number2 second numerical operand Divide by 0 error if number2 equals 0 57 B 1 1 Usage Precedence Description Returns Parameters Usage Precedence Description Returns Parameters B 1 2 Usage Precedence Description Returns Parameters Usage Precedence Description Returns P
13. Description Writes a mob using micabot Returns atom Parameters micabot the micabot to write the mob mob the mob to be written Error If there are any problems writing the mob B 6 7 mica delete mob Usage mica_delete_mob term Description Uses the current MicaBot to delete the named mob Returns nothing Parameters term the name of the mob to be deleted Error If there are any problems deleting the mob or there is no current MicaBot Usage mica_delete_mob micabot term Description Uses the MicaBot to delete the named mob Returns nothing Parameters micabot the micabot to delete the mob term the name of the mob to be deleted Error If there are any problems deleting the mob B 6 8 mica query Usage mica_query term Description Uses the current MicaBot to get a list of mobs matching a query Returns list Parameters term the query that defines the mobs being re quested Error If there are any problems querying the blackboard or there is no current MicaBot Usage mica_query micabot term Description Uses the MicaBot to get a list of mobs matching a query Returns list Parameters micabot the micabot to query the blackboard term the query that defines the mobs being re quested Error If there are any problems querying the blackboard 77 3 6 9 write wait for reply Usage mica write wait for reply mob integer Description Uses the current MicaBot to write the given mob and waits
14. Usage check_instance String functionName Instance currentObject Term arg int n StackFrame frame Description Function for checking that a argument to a function is a Instance frame This check will also succeed on Scripts Returns Instance Throws FSError if the designated argument is not a Instance frame Parameters functionName name of the function that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n index of the argument to check frame StackFrame that houses variable values C 1 10 check integer Usage check integer String functionName Instance currentObject Term arg int n StackFrame frame Description Function for checking that a argument to a function is an integer Returns FSInteger Throws FSError if the designated argument is not an integer Parameters functionName name of the function that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n index of the argument to check frame StackFrame that houses variable values C 1 11 check list Usage check list String functionName Instance currentObject Term arg int n StackFrame frame Description Function for checking that a argument to a function is a list Returns FSList Throws FSError if the designated argument is not a list Parameters functionName name of the functi
15. term2 second operand term term2 100 Term assignment operator Is right branching term term assignee term2 value being assigned If term cannot be assigned a value boolean and boolean2 75 Logical conjunction operator If boolean evaluates to false then boolean2 isn t evaluated boolean boolean first operand boolean2 second operand boolean or boolean2 80 Logical disjunction operator If boolean evaluates to true then boolean2 isn t evaluated boolean boolean first operand boolean2 second operand 59 Usage Precedence Description Returns Parameters B 1 11 1 Usage Precedence Description Returns Parameters B 1 12 Usage Precedence Description Returns Parameters Error B 1 13 and Usage Precedence Description Returns Parameters B 1 14 or Usage Precedence Description Returns Parameters not boolean 72 Logical negation operator boolean boolean operand slot of instance 5 Gets the value of a slot from an instance frame or a script Is right branching term slot the name of the slot must be hard coded as it isn t evaluated instance the instance frame script whose slot value you 8 looking for term in list 70 List membership check operator boolean term the term that is a possible member list the list that term is a possible member of nev generic 50 Generic frame instantiation operator Create
16. Returns Parameters Error B 2 15 Usage Description Returns Parameters B 2 16 append append list list2 Creates a new list where list2 is appended to the end of list list list list whose elements are to be the start of the new list list2 list whose elements are to be the tail of the new list delete term list Creates a list with the same elements as list except the first occurrence of term list term the term to be removed from list list the list term is being removed from fixrdr Used to generate the construction of an exception to a Ripple Down Rule Ideally should be placed inside the if replaced daemon of the slot whose value was replaced with the correct value RDR If no RDR has been evaluated or the slot has no value or no conditions are given to explain the exception rdr generic slot Loops through the instances of generic and gets the value of their slot slot Then it asks the user to verify the value evaluated If it is wrong the user is asked for the correct value and the value of the slot slot is replaced for that instance nothing generic a generic frame whose instances slots are to be tested slot the name of the slot where the ripple down rule is being tested Possible TOExceptions SyntaxErrors when communicating with the user 65 Usage Description Returns Parameters B 2 17 delete Usage Description Returns Parameters B 2 18 fixrdr Usage Des
17. a generic frame they can provide a basic dynamic element to scripts as long as instances of the generic frame can be dynamically created modified and destroyed Similarly because FrameScript allows not just text to be given as input but also events where the event is represented by an instance frame it can respond not only to user actions but also to system changes For example FrameSeript could tell a user if they receive an email or if their car was exceeding the speed limit This could give a system a proactive feel as the system can initiate a conversation with a user rather than always waiting for the user to initiate a conversation It also allows FrameScript to respond to non verbal inputs such as the use of a touch pad or mouse and so allow the construction of multi modal scripts Allowing daemons to be attached to scripts makes it possible for the system to alter environmental variables as it moves between contexts For example the on_entry daemon for a script could be used to tell a speech recogniser which grammar to use for a specific context Alternative if the system enters a watching movie context it could turn off the lights and turn on the TV then when it leaves it could turn the TV off and the lights back on For all the techniques employed in FrameScript to simplify the task of writing of scripts writing scripts is still not simple The script author has to take into account the ways a conversation can diverge and the myria
18. and e mail Once connected to a blackboard agents can read write and query objects on the blackboard In addition they can register for new objects written on the blackboard When a new object of interest to the agent is written to the blackboard the agent is informed of its arrival 3 MICA Objects or mobs for short are the basic unit of information in MICA Mobs are the things that agents actually read and write from the blackboard Mobs are very similar to objects in an object oriented framework but also share some characteristics with frames Each mob has a type defined by the user similar to a class in an object oriented language In addition mobs have slots which are similar to fields in object oriented programming each slot has a name and one or more values Currently only string values are supported Figure 1 illustrates how agents interact through the blackboard In this example a user is requesting a listing of his emails The audio input device in this case a phone posts an audio object to the blackboard The speech recognition agent has registered its interest in audio objects so it is notified of the arrival of the new object The speech recognition system reads and processes the audio object posting its output a text object back to the blackboard 10 preoqy viq S VOIIN 814 qeorunururoo squo3y oandif Juaby jueby Buiss ooiq eben6ue7 jeinjen Y H AUOO X8 0 ewa E
19. both security and privacy measures e Actas an interpersonal communication mechanism as well as allowing all of one person s communication tools to communicate with each other MICA s is based on the blackboard architecture 3 Groups of agents use the blackboard to share knowledge and communicate When something is written to the blackboard each agent examines it and sees if it can make a contribution to solving a problem In MICA we update the idea of a blackboard by support ing distributed execution and network communication using an object oriented data representation and adding security and privacy mechanisms There are three entities used in the MICA design 1 The blackboard is the core of MICA It is similar to a telephone switch with memory All interactions between agents flow through the blackboard in much the same way that all phone calls in a town go through a switch Much of the power of the blackboard comes from its ability to allow many agents to share information It is expected that in use there would be one blackboard for each person or environment such as a car or house 2 Agents are entities that access and contribute to the information on the blackboard The two main types of agents are interaction agents that convey information to and from the user such as GUls proxies for tele phone connections and devices in the environment and computational agents that provide services such as speech recognition web access
20. e File e New e New Domain Creates a new domain e New Script Creates a new script e New Function Creates a new function 42 Load File Loads a FrameScript file into memory e Load Module Load a FrameScript module into memory e Save Scripts Writes out the scripts in the symbol table to a file Serialise Serialises FrameScript s symbol table to a file e Unserialise Loads the symbol table stored in a file e View View Recognition Daemon Switches the view of a script to the abnormal script view so that normal scripts can be made abnormal e Start Conversation Opens the conversation GUI see section 4 5 5 using a given domain e Help e Set Look amp Feel Changes the UI look and feel e About Describes the script browser 4 5 2 Domains Script Browser File View Help Domain Name fred Current Context test script Current Topic test script Failsafe Topics Topic I Trigger ma yo test script Figure 29 Domain Browser When a domain is selected in the tree the details of that domain are shown in the right side of the script browser See figure 29 Shown are the domain s current context current topic failsafe and the list 01 topics registered for the domain The domain s current context current topic and failsafe can all be changed using comboboxes and its topics can be added removed or reordered 43 4 5 3 Scripts
21. end of the pattern Pattern Elements A pattern element can be any type of term but when a pattern is parsed only specific term types are parsed The types read in by the parser are alternatives sequences atoms strings numbers commands and commands All other types can be found as pattern elements however as a result of evaluation and Commands When parsing pattern elements and are used as escape characters to allow expressions to be embedded within patterns The character indicates that when the pattern is evaluated the following expression should be evaluated but its result should not be returned in the result of evaluating the pattern The character however says the following expression should be evaluated and in serted into the result of evaluating the pattern The command however has one exception when the expression following is an integer then it gets the asso ciated match component for the pattern to a rule This is fine when the pattern being evaluated is part of the response to that rule but at other times it will result in an error Pattern Examples The best way to demonstrate patterns is with some examples See Table Example Evaluated Result lt lt ab c gt gt gt gt ab 6 lt lt gt gt 1 2 lt lt gt gt 1 2 lt lt lt lt 7 1 2 lt lt gt gt 3 lt lt gt gt 1 2 lt lt gt gt lt lt a 1 gt gt a lt lt gt gt 8 lt lt a 1 gt gt 78 lt lt
22. for a reply If no reply is received within the given time period in milliseconds then it returns nothing Returns mob null if a timeout occurs Parameters mob the mob to be vritten integer the number of miliseconds to wait for a reply Error If there are any problems writing or receiving the mobs Usage mica write wait for reply micabot mob integer Description Uses the MicaBot to write the given mob and waits for a reply If no reply is received within the given time period in milliseconds then it returns nothing Returns mob null if a timeout occurs Parameters micabot the micabot to write the mob and wait for a reply mob the mob to be written integer the number of miliseconds to wait for a reply Error If there are any problems writing or receiving the mobs B 6 10 get mob name Usage get mob name mob Description Gets the name of a mob Returns atom Parameters mob the mob whose name is being reguested B 6 11 current micabot Usage current micabot Description Gets the current MicaBot Returns micabot Error If there is no current MicaBot 78 C Utility Functions TODO C 1 Subroutine Argument Type Checking TODO C 1 1 check alternatives check alternatives String functionName Instance currentObject Term arg int n StackFrame frame Function for checking that a argument to a function is an Alternatives Alternatives FSError if the designated argument is not an Alternati
23. function that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n index of the argument to check frame StackFrame that houses variable values Usage Description Returns Throws Parameters C 1 7 check frame check frame String functionName Instance currentObject Term arg int n StackFrame frame Function for checking that a argument to a function is a Frame Frame FSError if the designated argument is not a Frame functionName name of the function that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n index of the argument to check frame StackFrame that houses variable values Usage Description Returns Throws Parameters C 1 8 check generic check generic String functionName Instance currentObject Term arg int n StackFrame frame Function for checking that a argument to a function is a Generic frame Generic FSError if the designated argument is not a Generic frame functionName name of the function that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n index of the argument to check frame StackFrame that houses variable values 81 Usage Description Returns Throws Parameters C 1 9 check instance
24. input it checks if it is a type of house or road f it is neither it claims ignorance If the input is a type of house the script says it is a house if it s a road it says it s a road All of the rules in the describe context switch the context back to the demo context for the next user input 4 GUI The standard FrameScript distribution includes the module sitcrc framescript GUl which contains several subroutines that open GUIs that can be used to facilitate interactions with the user The subroutines currently included in the module are e dialog message Opens a simple message dialog See section 4 1 e dialog_question Opens a simple input dialog See section 4 2 e fix_rdr_gui Opens a dialog for providing a description of the differ ence between two cases when maintaining an RDR See section 4 3 e frame_browser Opens a GUI for creating and manipulating frames See section script_browser Opens a GUI for creating and manipulating domains and scripts See section 4 5 4 1 Message Dialog The subroutine dialog message expects a single argument It will open a simple dialog window to display the given argument 11 the argument is a string or a pattern the surrounding quotation characters will not be displayed The result of calling dialog_message Hello world is given in figure 22 37 FrameScript message Hello world Figure 22 Message Dialog FrameScript question Are mice big or smal
25. is derived from Probot 7 a language intended solely for text based con versational agents FrameScript extends Probot in providing mechanisms for interaction through a variety of devices FrameScript also allows inheritance of scripts FrameScript has a companion program called MICA that coordinates com munication between agents FrameScript s design is better understood if the reader has some knowledge of the context in which it is intended to be uased Therefore the following sections give overviews of both programs before fo cussing on the details of FrameScript Our goal is to provide easy to use and intuitive mechanisms for interacting with intelligent environments Two major software systems have been developed for this purpose FrameScript is a language for writing scripts that implement multimodal user interactions It incorporates rules for describing natural lan guage conversations as well as interpreting events in other modalities including touch screens and gestures MICA is a system that provides a simple but powerful method for software agents to communicate with each other These agents may control devices in an intelligent environment or they may provide services such as speech recognition natural language processing or access to resources on the internet The main advantage of these programs is that they make the development of applications involving multi modal interfaces relatively quick and easy 1 1 MICA MICA Mul
26. slots in scripts and slots in instance frames is that slots in scripts don t have daemons attached to them This is because scripts inherit from scripts not from generic frames However scripts do inherit slots from their parents Effectively the means that a slot in a script s parents acts like an if needed daemon except it is Just returned not evaluated In Figure although peter does not have a defined age he inherits age from person so in response to the input hovv old are you peter would return i was born years ago 2 2 8 Failsafes It is possible to define failsafes for scripts A failsafe is another script whose rules should be used if an input fails to match any of the rules for a script Failsafes may be inherited Domains can also be given a failsafe by using the failsafe subroutine Domain failsafes vvill only be used if an input fails to match any of the rules for the current context the current topic and their associated failsafes Failsafes can be chained together This means that when failsafes are being checked against an input and a failsafe and its parents doesn t match against the input the failsafe s failsafe will be tried 2 2 9 Rule Ordering VVhen an input is received by a domain that domain is responsible for deciding which rules the input should be matched against The order in which domains attempt to find a match is 1 triggers of the topics excluding the current topic 2 the current context
27. term failsafe domain script Sets a script to be the global failsafe for a domain script domain the domain that for which the global failsafe is being defined script the global failsafe for the domain Usage Description Returns Parameters Error B 4 10 match Usage Description Returns Parameters Error Usage Description Returns Parameters B 4 11 failsafe Usage Description Returns Parameters B 4 12 question question domain term Initiates a conversation with the user that continues until 8 value is returned term domain domain that provides the systems side of the conversation term an input to be used to initiate the conversa tion 72 Usage Description Returns Parameters 2 4 15 return Usage return term Description Is used to provide a return value for question domain term If used outside of a question domain term call it does nothing Returns nothing Parameters term the value to be returned by question domain term Error If there is no current domain B 5 GUIs In the standard FrameScript distribution is the module sitcrc framescript GUI This is a small module that implements GUIs for communicating with the user using message and input dialogs and more complex GUIs for maintaining RDRs frames and scripts These GUIs are not active by default but can be enabled by calling load module sitcrc framescript GUl B 5 1 dialog message U
28. tested 63 Usage Description Returns Parameters B 2 6 set Usage Description Returns Parameters B 2 7 mumber Usage Description Returns Parameters B 2 8 integer Usage Description Returns Parameters B 2 9 list Usage Description Returns Parameters cons term list Constructs a list with term at its head followed by the values of list list term term to be the head of the list list the tail of the list B 2 10 Usage Description cons Returns Parameters B 2 11 member member list term Tests if term is a member of list boolean list list whose membership is being tested term term whose membership of list is being tested head list Gets the first value of a list term nothing if list is empty list list of values tail list Gets the tail of a list ie all values except the first list list list whose tail we want nth integer list Gets the nth element of a list term integer index of the element indices go from 0 to length list 1 list list whose value is being retrieved If integer is not a valid index of list length list Gets the number of elements in a list integer list list whose elements are to be counted 64 length Usage Description Returns Parameters B 2 12 head Usage Description Returns Parameters B 2 13 Usage Description Returns Parameters tail B 2 14 nth Usage Description
29. using FrameScript and these connect through MICA to invoke other agents such as DNS or to control devices The Frame Script language provides a very rich set of tools for representing knowledge and interacting with users and external devices Its strength is that new agents can be created quickly and easily In this section we briefly survey only the most important features of FrameScript Later sections describe the language in detail Scripts provide FrameScript with rule based processing for controlling in teractions with users and devices The rule system in FrameScript is derived from Sammut s Probot 7 The basic unit of a FrameScript program is the pattern response rule pattern gt response In stand alone mode rules are used to match input from the user When coupled with MICA the left hand side of a rule contains a pattern than may match objects on the blackboard Thus if another agent posts a text object a FrameScript agent that has registered interest in text objects can read the object and perform natural language processing to produce a response Patterns can also match arbitrary objects For example a pointing gesture may be recorded by a location and direction in space as well as a time stamp A FrameScript agent that has registered interest in these kinds of objects will read the gesture information and match that against its rules to produce a response Patterns can also be mixed allowing multi modal interactions W
30. 12 2 1 1 Slots Slots are the attributes of a frame They hold the attributes of instances frames and provide points for attaching procedures to the attributes in generic frames through daemons Slots can be defined as being multivalued and cache Multivalued slots can hold multiple values Cached slots store the value for an instance s slot when the slot s value is first computed in either an if_new or if_needed daemon otherwise the slot s value is recalculated every time is is accessed 2 1 2 Generic Frames Generic frames define values and behaviours that are applicable to all instances of the type as such they act like classes in the standard OOP paradigm person ako object with name if needed Anonymous age if new 0 cache true Figure 2 Example Generic Frame 2 1 3 Daemons Daemons are procedures that are attached to slots in generic frames They are executed when a frame s slot is accessed in a given way FrameScript currently allows 9 daemons to be attached to a slot These daemons are e if added The if added daemon is run whenever a value is placed in a slot e if destroyed The if destroyed daemon is executed when an instance frame is destroyed if needed The if needed daemon computes the value of a slot if the value isn t currently known when it is accessed if new The if new daemon is run when the instance frame is first created to give the slot an initial value e if removed The if rem
31. A sc a M 6 8 28 Rules 2 2 s sa ik ke e ke Q RAD w Af Patterns 24444 k 222 Raesponsesl 2 2 3 Pattern Component Numbering 2 2 4 2 2 5 Domains Topics and Triggers Abnormal Scripts 2 2 12 Pattern Matching Algorithm 2 2 6 Inheritance 2 2 7 Slots 2 2 8 Failsafes 2 2 9 Rule Ordering 2 2 10 Daemons 2 2 11 2 3 Expression Language 2 3 1 Statements oia a asa ee ak 2 3 2 Numbers sas 2 5 wa boa kek 5512121777 7 2 2 4 a A ss e as A 2 3 6 Expression Lists 2 3 Pattermsi s 5 kt azda AZAL ae BALA Q Q Q 4 n 77 s 2 3 9 Alternatives 2 3 11 Hhuneti ns w ui a a s G wd 23 13 Variablesi asias e ae A ZA A W W W Q A 2 4 1 Subroutines w w oua ao w cosida 064 444 0000000006 3 Maulti modal Interaction 321 Multi modal Example 4 GUI 4 1 Message Dialogg 4 2 Question Dialogl 4 3 RDR Maintenance GUI 44 Frame Browser 4 4 2 Transcript a a d 4 bee s
32. FrameScript A Multi modal Scripting Language M McGill C Sammut mmcegill cse unsw edu au claude cse unsw edu au J Westendorp W Kadous jhw cse unsw edu au waleedQcse unsw edu au Abstract This document is a user manual for the FrameScript scripting lan guage FrameScript combines rule based scripts with frames 6 and an untyped expression language It has been developed for the purpose of rapid prototyping of conversational speech and multi modal interfaces FrameScript uses a number of techniques to allow scripts to be modu larized and so simplify the task of writing them It has also been designed in such a way that it is easily extensible and so can be used in a variety of applications and with a variety of technologies This user manual also describes MicaBot a FrameScript extension that allows FrameScript to be used with the Mica agent architecture MicaBot allows Mica agents to be written using FrameScript 00 12 12 13 13 13 14 14 16 16 16 17 18 19 21 22 22 22 23 24 24 26 26 26 27 27 27 27 28 29 29 29 31 32 32 34 34 Contents 11 MICA 1 2 FrameScript 2 FrameScript w a QU R ua la g d G Ya RO ula da 2 1 1 LOGS A 2 12 Generic Pramesl 21 3 Daemons wot ww xs eae ww ee Q WOW a debat 2 1 4 Instance Frames A de ie mama BA Q Q 0 0 cds s e A
33. The rules in FrameScript consist of patterns and responses where each pattern matches a user s input and the response determines the systems output When a pattern is found that matches the user s input the associated response is returned Rules are of the form pattern gt response 2 2 2 Patterns Patterns can be written to match a variety of sentences The patterns used in FrameScript are similar to context free grammars Wild cards FrameScript allows the use of 2 wild cards characters in patterns They are and will match to 0 or more words terms while will match precisely 1 word term In addition can be embedded into a word to match 0 or more characters Alternatives Alternatives may be used if more than one word can be accepted at a point in a pattern eg synonyms Alternatives are constructed with the form alternative 1 alternative 2 Non terminals Non terminals provide a way for commonly used patterns to be reused To use a non terminal write the name of the non terminal surrounded by lt and gt eg I am lt number gt years old While it is technically possible for any script to be a non terminal they are often declared as a list of alternatives followed by Example number 11121314151617181 9 1 FrameScript uses a greedy matching algorithm so some patterns may not work as ex pected 16 and Some patterns may be context sensitive so FrameScript allows condi
34. ackFrame frame Function for checking that a argument to a function is an RDR RDR FSError if the designated argument is not an RDR functionName name of the function that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n index of the argument to check frame StackFrame that houses variable values 83 Usage Description Returns Throws Parameters C 1 15 check script check script String functionName Instance currentObject Term arg int n StackFrame frame Function for checking that a argument to a function is a Script Script FSError if the designated argument is not a Script functionName name of the function that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n index of the argument to check frame StackFrame that houses variable values Usage Description Returns Throws Parameters C 1 16 check sequence check sequence String functionName Instance currentObject Term arg int n StackFrame frame Function for checking that a argument to a function is a Sequence Sequence FSError if the designated argument is not a Sequence functionName name of the function that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n i
35. arameters B 1 3 Usage Precedence Description Returns Parameters B 1 4 Usage Precedence Description Returns Parameters Error integer 06 2 40 Modulo operator integer integer first integral operand integer2 second integral operand Divide by 0 error 11 integer2 equals 0 number lt number2 70 Less than comparison operator boolean number first numerical operand number2 second numerical operand number gt number2 10 Less than or equal to comparison operator boolean number first numerical operand number2 second numerical operand number 5 number2 10 Greater than comparison operator boolean number first numerical operand number2 second numerical operand number lt number2 10 Greater than or equal to comparison operator boolean number first numerical operand number2 second numerical operand 58 B 1 5 mod Usage Precedence Description Returns Parameters Error B 1 6 lt Usage Precedence Description Returns Parameters B 1 7 lt Usage Precedence Description Returns Parameters B 1 8 gt Usage Precedence Description Returns Parameters B 1 9 gt Usage Precedence Description Returns Parameters term term2 70 Term equality check operator boolean term first operand term2 second operand term term2 70 Term inequality check operator boolean term first operand
36. at script is telling the domain that no matter what the current script is when certain inputs are received the topic is to become the current script and 19 20 4sxon 3 szeo lt 1equnu gt 189 e ere ST x 410019 pez ANS x se3enBuer umouy gt UI gt oaersueznl o durexq Surr qumN Jueuodurog I 91q8L anof ere 9901 Ied e SI 91943 enTq sr Ays eq ur UT uted UT rt qeTsued4 oqur no AOT T elersuezi ndug syonz1 anoj T oze z n INOJ ere 91911 rev e lt lt reo ST ezemI red e ST 01903 anTa SI aya st xs n URUIOS uted ur we p lt lt lt lt lt lt lt lt lt lt nok eAoT T S HNIS HNIS Ama doc uz xd 415 lt squouodmo yaje ueuz 3 ur uted UT we t qelIsueaq oqur no t 910 foul_language_filter domain example trigger 1 swear word k lt swear word gt swearing is not tolerated Figure 13 Topic Example process the input When a script is registered as a topic the domain uses the script s trigger to determine whether or not an input activates that topic If a topic doesn t
37. cogniser that can either use rule grammars or use dictation mode Usually the recog niser would use the rule grammars but on occasion it may need to use dictation In the example the current context would be normal context which would re spond to the recognised speech from the rule grammars But when the user says dictate the current context switches to dictation context and its on entry daemon is run which tells the recogniser to switch to dictation mode Then all of the user s speech would be saved until the user says finished at which point the saved text is used for whatever purpose and the current context switches back to normal context In the course of switching back to normal context dictation context s on exit daemon is run which switches the speech recog niser back to rule grammar mode NOTE If the on entry or exit daemon is followed by a pattern that begins with or any other operator then the pattern should be preceded by _ so that the parser doesn t confuse it with a multiplication symbol Using just says that at the start of the pattern there is an empty space so in effect it does nothing 23 2 2 11 Abnormal Scripts It is possible to define the rules for a script using the expression language used in defining daemons but for most applications it is simpler to use the standard rule response syntax Using the expression language however allows more complex rules to be written For instance using the expre
38. cription Returns Error 2 19 rdr Usage Description Returns Parameters Error print term Prints its arguments to the output stream term term term to be written to the output stream error term Throws an error Uses its arguments to construct a message for the error term message to be used for the error An error with term as its message ask term VVrites term to the output stream and waits for the user to respond with a term term term request to be vritten to the output stream Possible TOExceptions SyntaxErrors when communicating with the user eval term Gets the value of term and evaluates it term term the value to be reevaluated quote term Returns term without evaluating it term term the value to be given load filename Reads in FrameScript code from a file nothing filename name of the file to be read in H file not found or problems reading from the file 66 B 2 20 print Usage Description Returns Parameters B 2 21 Usage Description error Parameters Error B 2 22 ask Usage Description Returns Parameters Error B 2 23 Usage Description Returns Parameters eval B 2 24 quote Usage Description Returns Parameters B 2 25 load Usage Description Returns Parameters Error 13 2 26 load module Usage load module modulename Description Loads a module into memory Returns nothing Parameters f
39. d of ways that people can say the same thing Also they need to understand how the same sentence can take on different meanings depending on the current context of the conversation The script writer must also take into account the very real possibility that the system may have to interact with hostile users 49 FrameScript provides script writers with tools for creating and editing scripts but the interfaces they provide are very simple There are probably better ways of visualising and editing scripts that make it easier for script writers to understand and manipulate the progression of a conversations through the contexts the scripts embody It may also be possible to create a library of scripts that provide the be haviours necessary for implementing the computer s side of common interac tions that users have with their systems Such interaction could include getting a person s address or phone number Such a library could then be used across a number of possible systems and so allow script writers to focus on those in teractions that are specific to their system One way to ease the burden on script writers would be to enable the scripts adapt themselves in response to interactions with users Doing so could result in conversational agents that resemble Turing s Child Machine S in that they can learn by examples A simple method of adding such a feature is the use of rules that construct modify other rules Such rules are feasible for lim
40. e frame the value of the slot is being put into slot the slot the value is being put into term the value to be added to the slot Error If slot of instance already has a value and is not multival ued or an error is thrown by a daemon 68 replace slot term Replaces the value of the slot slot of current_object with term term slot the slot whose value is being replace term the value to be put into the slot If there is no current object or an error is throvn by a daemon replace instance slot term Replaces the value of the slot slot 01 instance with term term instance the instance frame whose slot value is being replaced slot the slot whose value is being replace term the value to be put into the slot If an error is thrown by a daemon remove slot Removes the given slot from current object term slot the slot to remove from current object If there is no current object or an error is throvn by a daemon remove instance slot Removes the given slot from instance term instance the instance frame the slot is being removed from slot the slot to remove If an error is thrown by a daemon remove instance slot term Removes a specific value from a multivalued slot in the given instance frame term instance the instance frame the slot value is being re moved from slot the slot whose value is being remove term the value to be removed from the slot If slot of instance is not multivalued or an error is th
41. e is no current domain or an error is thrown by a daemon 70 other It also has B 4 1 script Usage Description Returns Parameters B 4 2 domain Usage Description Returns Parameters B 4 3 pattern Usage Description Returns Parameters B 4 4 register Usage Description Returns Parameters B 4 5 goto Usage Description Returns Parameters Error goto script term Sets the given script as the current context of the current domain Then gives term as an input to the domain script script the script to be the current context term a term to be used as an input to the domain If there is no current domain or an error is thrown by a daemon current context current contezt Gets the current contezt of the current domain script If there is no current domain Usage Description Returns Parameters Error B 4 6 Usage Description Returns Error B 4 7 previous topic previous topic 0 Returns to the previous topic Sets the topic as the current context nothing If there is no current domain or an error is thrown by a daemon previous topic term Returns to the previous topic Sets the topic as the current context Then gives term as an input to the domain nothing term a term to be used as an input to the domain If there is no current domain or an error is throvn by a daemon Usage Description Returns Error Usage Description Ret
42. e usually want the application of rules to be confined to particular contexts For example if the user says to an in car control system turn it up this can mean different things depending on whether the user had previously said turn on the radio or turn on the air conditioner Contexts are handled by grouping rules into different scripts For example radio_script 11 trigger radio x turn up x 1 11 turn the radio up a little 4180610 volume 10 The trigger is a pattern that is used to determine if the user had changed the context of the conversation If the previous topic ha been the air conditioner and the user now mentions the radio this trigger will cause a context switch The appearing in the patterns is a wild card match When the user says turn it up in the radio context the rule above is triggered and the system invokes an action from the radio the symbol precedes an action and informs the user that the volume has been turned up The radio volume function posts to the blackboard an object that is intended for the radio controller agent In this way a FrameScript agent can act as a device controller or intermediary to external information sources When writing complex scripts that have similar behaviours it is possible to use inheritance to enable rules to be shared between scripts A script can also inherit variables that store information about the state of the interact
43. ey still correspond to the new function specification 4 5 5 Conversation GUI Conversation GUI Figure 33 Conversation GUI If a user selects Start Conversation from the script browser s menu they will be asked to select a domain to manage the conversation When a domain is selected the conversation GUI is opened using that domain to manage a conversation with the user The conversation shows the conversation s history and at the bottom has a text field the user can use to enter their side of the conversation The user can use the arrow buttons of their keyboard to flip through previously entered statements 5 MicaBot FrameScript is a language for scripting of verbal and multi modal interactions In order for it be used in an application it needs to be able to receive speech and events from somewhere and to be able to return its responses The simplest way to manage this is to extend FrameScript and in the extension to provide an interface to whichever technology architecture is being used to handle the device agent system communications MicaBot is a FrameSeript extension that acts as a bridge between Frame Script and the Mica agent architecture 3Mica 4 5 is a middle ware layer for pervasive computing that aims to simplify commu nications betvveen devices and agents and facilitate the separation of applications from their interfaces 46 Among the extensions provided by MicaBot are a number of subr
44. h increments the value of a by 1 which in this case makes a 2 Then the third final expression a is evaluated which retrieves the value of a this final value then is the result of evaluating the list a 1 a a 1 a When writing expression lists common practice should entail enclosing the entire list in brackets This is because by enclosing the list in bracket the expres sion list is being designated as a single expression eg a 1 b a 1 b 2 2 3 7 Patterns Patterns are a data type used primarily in defining and processing scripts But they can be used virtually anywhere within FrameScript Patterns are a sort 27 of hybrid type that results when a string crossbreeds with a list A pattern is a sequence of zero or more pattern elements enclosed by lt lt and gt gt eg gt gt gt gt lt lt hello world gt gt When writing scripts the lt lt and gt gt are implied for readability When a pattern is evaluated each pattern element is evaluated sequentially an a resultant pattern is returned As each element in the pattern is evaluated it is first checked whether or not it is an atom If it is an atom it is appended to the resultant pattern as is without evaluation otherwise it is evaluated If the evaluated value is nothing the value nothing is done with it if it is a pattern the evaluated pattern is appended to the end of the resulting pattern if it is any other type of term it is simply appended to the
45. have a trigger any input will activate it Triggers can be inherited Domains keep track of the current script context and a history of the trig gered topics When a topic s trigger matches the input it becomes the current context and the current topic 2 2 6 Imheritance person age 0 who are you 1 don t know who i am how old are you 1 am age years old 1 john inherits person name lt lt john smith gt gt age 4 who are you i am john peter inherits person name gt gt peter piper lt lt how old are you 1 vas born age years ago 1 Figure 14 Script Inheritance Example When writing complex scripts where scripts have similar behaviours it is 21 possible to use inheritance to enable rules to be shared between scripts Not just rules are inherited a script also inherits slots and daemons from it s parents as well The inheriting script is able to define new responses to patterns located in the inherited script if needed and new values for slots 2 2 7 Slots Scripts can have slots defined for them This allows scripts to store and manip ulate information about the state of the script and the state of the conversation In most respects slots in scripts behave in much the same way as slots in in stance frames In fact many of the subroutines for accessing and manipulating slots such as put remove and replace will work with scripts as well The main difference between
46. his doesn t include subroutines 41 On the left of the Functions tab is a tree showing all of the currently active functions when a function is selected the parameters used in the function are listed and the body of the function is displayed It is possible to add parameters remove svvap the parameters of a function NOTE Calls to the function will need to be updated separately or they will break Tt is also possible to edit the function s body The New button below the known functions can be used to define new functions 4 5 Script Browser Script Browser File View Help Name test script Trigger Failsafe Parents Rule Comment Pattern BC Response y Figure 28 Script Browser The script browser see figurel28 which can be opened by script brovser is for the viewing and manipulation of scripts It can be used to examine Frame Script s state as a conversation progresses and to create and edit scripts On the left side of the script browser is a tree that displays all of the known domains and scripts named in the symbol table and the active functions Also shown in the tree are the registered topics of a domain listed with the topics triggers 4 5 1 Menu The script browser s menu allows a number of operation to be performed A summary of the operations available through the menu is
47. if added if destroyed if new if needed if removed if replaced default range help slot type multivalued cache slot values slot value slot values slot value atom expression script atom script header seript rules expression list script header script modifier script header 52 script modifier inherits atom instanceof Tl atom domain atom topicTl atom trigger pattern element on_entry expression list on exit expression list failsafe atom atom factor script rules pattern seript rule seript rules script rule pattern gt pattern element pattern pattern element pattern pattern element sequence alternative non terminal 77 factor factor atom number string sequence pattern list alternative 1 pattern list Pp non terminal 7 atom pattern list conditional pattern pattern list 1 conditional pattern pattern gt pattern expression list expression expression list expression assignment expression used for regression purposes 511800 for regression purposes 6Except E 1 gt gt l gt gt 1 53 assignment expression disjunction expression lassignmen
48. ilename name of the module to be loaded Error Number of possible error reading the Java class file the mod ule is implemented in B 2 27 output to file Usage load filename Description Sets the output stream to go to a given file Overwrites the file Returns nothing Parameters filename name of the file to be output to Error If problems writing to the file Usage load filename boolean Description Sets the output stream to go to a given file Returns nothing Parameters filename name of the file to be output to boolean whether or not to append to the file Error If problems writing to the file 3 2 28 close output Usage close output Description If the output stream is going to a file it is closed and the previous output stream is set to be the output stream Returns nothing Usage close output filename Description If the output stream is going to the file filename it is closed and the previous output stream is set to be the output stream Returns nothing Parameters filename name of the file to stop outputting to B 2 29 print as text Usage print as text term Description Prints a term to the output stream 11 term is either a frame or script it is written in a textual format that as much as possible can be read back in Returns term Parameters term term to be written to the output stream B 3 Frame Subroutines As FrameScript uses frames to provide structure to data it needs functio
49. in the form of a using a touch pad or a making a recognised gesture This allows FrameScript to be used not just for conversations but also for scripting multi modal interactions between the system and users 3 1 Multi modal Example A simple multi modal script is given in Figure 21 The problem domain in which the example is set is a simple street mapping program that shows houses and roads This example is built using a few assumptions These assumptions are 34 package sitcrc framescript examples import sitcrc framescript public class MaxModule public static void init throws FSError new Subr max public Term apply Instance currentDbject Term args StackFrame frame throws FSError if args length 1 1 throw new FSError max No arguments given FSNumber max Utils check number max currentObject args 1 frame for int 1 2 1 args length 1 FSNumber x Utils check number max currentObject args i frame if max 1t x max x l return max l nev Subr max2 2 1 public Term apply Instance currentObject Term args StackFrame frame throws FSError FSNumber x Utils check number max2 currentObject args 1 frame FSNumber y Utils check number max2 currentObject args 2 frame if x le y return y return x Figure 20 MaxModule Example 35 demo domain example request a_click what is this put request describe
50. ion VVhile the seript is the main construct that is used to define interactions there is a complete programming language underlying the seript mechanism Knowledge structures called frames provide an object oriented programming paradigm Frames were originally developed as a knowledge representation sys tem in Artificial Intelligence Simple scripts will simply be a collection of rules like the ones above Interactions that require more complex behaviour may use the full power of the frame representation system For simplicity we have also only described very simple rules However the rule language can describe complex grammars The remainder of this document describes the details of the language 2 FrameScript FrameScript is a scripting language that combines a rule based system with a frame implementation for data storage and retrieval both of which are built upon an expression language 2 1 Frames Frames 6 provide a way of representing and manipulating complex objects There are two main features of frame systems Procedural Attachment Inference in a frame system is performed on an ad hoc basis by attaching procedures to the attributes of an object These procedures are triggered depending on how the attribute is accessed Inheritance Like all all object oriented systems instance frames can inherit properties from generic frames in a hierarchy In FrameScript frames can inherit from multiple generic frames
51. ited domains where the patterns and responses for the new rule can be easily de fined using some form of template More generalised learning is a much more daunting problem especially when it has to take into consideration the possible development of new contexts and sources of information Much of the recent work with FrameScript has been looking at using it to build speech interfaces to devices and programs Very little however has been done to examine how it can be used in multi modal interactions While it can accept virtually any type of event a system can produce including events related to any input modality vvork is still needed to identify the best methods for processing multi modal input Presently work is exploring the use of FrameScript in multi modal environ ments and for reporting events to a user It is also looking at a variety of ways of representing and manipulating the current state of a conversation References 1 P Compton G Edwards B Kang L Lazarus R Malor T Menzies P Pre ston A Srinivasan and C Sammut Ripple down rules possibilities and limitations In 6th Banff AAAI Knowlede Acquisition for Knowledge Based Systems Workshop 1991 2 Paul Compton and R Jansen A philosophical basis for knowledge acquisi tion In 3rd European Knowledge Acquisition for Knowledge Based Systems Workshop pages 75 89 1989 3 L D Erman F Hayes Roth V R Lesser and D R Reddy The hearsar oo speech unders
52. l Cancel big Figure 23 Question Dialog 4 2 Question Dialog The subroutine dialog_question expects a single argument It will open a simple dialog window that will request an input from the user If the argument is a string or a pattern the surrounding quotation characters will not be displayed The result of calling dialog_question Are mice big or smal17 is given in figure The input dialog will accept simple expression from the user such as numbers atoms strings list patterns function specs calls and will return the first such expression it finds in the input 43 RDR Maintenance GUI As RDR maintenance is a difficult task the fix rdr gui subroutine was implemented to somewhat simplify the process The fix_rdr_gui subroutine is basically the same as the fix_rdr subroutine except it uses a GUI to interact with the user rather than the console Both subroutines are designed to be placed in the if_replaced daemon of a generic frame and to be called when an incorrect computed value of an instance frame is replaced by a correct value and both accept no arguments An example of the GUI opened by fix gui is shown in figure This GUl is roughly divided into 4 components 1 Cases The case that was used in defining the rule that gave the incorrect value is given on the left On the right is the new case whose value has been corrected All slots and their values for each of the cases is given 2 Possib
53. le Conditions A number of possible conditions that could ex plain the differences between the 2 case are listed with check boxes As these conditions a selected deselected the condition to be used to for the 38 LOX Old Case t1 New Case t2 Slot Value slot A slot2 Some Possible Conditions slot2 2 slot2 3 Exception Condition slot2 3 Figure 24 RDR Maintenance GUI new rule will update These conditions do not have to be used they are only given as a way indicating differences between the 2 cases 3 Exception Condition The condition for the exception rule This condition should be an expression that evaluates to true or false that can be used to identify exceptional cases that should take the new value 4 Accept Button When pressed this button will perform some basic syntax checks on the exception condition and if it s fine close the GUI and create the new exception rule If the GUI is closed without a new rule being created fix_rdr_gui will throw an error 4 4 Frame Browser The frame browser see figure B25 which can be opened by frame browser is a GUI specifically designed to enable a user developer to create new and manipulate existing frames 4 4 1 Menu The frame browser s menu allows a number of operation to be performed A summary of the operations available through the menu is e File e Load Loads a FrameScript file into memory e Load Module Load a FrameScri
54. le Example Figurel20lshovvs a fairly simple FrameScript module All this module does is de fine 2 subroutines that can then be used in FrameScript code These subroutines are max and max2 2 The max2 subroutine expects 2 arguments The first thing that it does is to check that both of the arguments are numbers If either of the arguments isn t a number the Utils check number function will throw an exception Then max2 finds the largest number and returns it The constructor used to define the max2 subroutine will throw an exception if when the module is loaded there already exists a max2 subroutine that take an undefined number of arguments This exception can safely be thrown by the surrounding init method The max subroutine doesn t define how many arguments it expects in its declaration so it can be called with any number of arguments The first thing it does is to find out how many arguments it has If there are none it throws an exception Otherwise it checks that the first argument is a number Then it loops through all of its remaining arguments checking that they are numbers and keeping track of the largest seen so far Once all arguments have been checked the largest is returned 3 Multi modal Interaction One of FrameScript s strengths is its ability to use frames to represent events The events that can be represented could range from system events such as receiving an email or a program finishing or a user s input
55. match component when a pattern matches These components are segments of the input possibly transformed 18 hello i m lt number gt years old really 72 is really old 1 i m x gt hi i J what do you mean by 70 Figure 11 Match Component Example that can be referred to in a response using Pattern elements that pro duce match components are wild cards 7 alternatives non terminals and possibly conditions depending on the evaluation result eg generic frames do Component Numbering Example Table L shovs the match components generated for a number of inputs when some inputs are given These examples use some non terminals which are given in figure known languages english french german italian spanish number one lt 1 two lt 2 three lt 3 four lt 4 five lt 5 six lt 6 seven gt 7 eight lt 8 nine lt 9 zero lt 0 O 00 OT IB Q N Figure 12 Component Numbering Example Non terminals 2 2 5 Domains Topics and Triggers When an input is received from a user it is given to a domain The domain is then responsible for ensuring the input is matched against the correct scripts It is the domain that keeps track of which scripts are currently active Scripts can be registered as topics in a domain By registering a script as a topic th
56. mber var list of varsl Not really an operator just a reserved atom used to define local variables in daemons functions forall statements list of vars comma separated list of variable names If it is not at the top of the daemon function forall state ment forall variable in list term Not really an operator just a reserved atom used to loop through a list It returns the values of evaluating term for each value in 118 list variable name of a variable that takes its value from the elements in the list list a list of values for the variable term a term to be evaluated for each value in list If it is not at the top of the daemon function forall state ment 1 B 1 20 Usage Precedence Description Returns Parameters B 1 21 to Usage Precedence Description Returns Parameters Error B 1 22 Usage Description var Parameters Error B 1 23 forall Usage Description Returns Parameters Error 2 1 24 gt Usage pattern gt pattern2 Precedence 90 Description An operator that allows conditional responses to rules Returns pattern Parameters pattern condition for production of the response should evaluate to either gt gt true lt lt or gt gt false gt gt pattern2 response to use if pattern evaluates to gt gt true lt lt 13 1 25 Precedence Table Precedence Operator 100 90 gt 80 or 75 and 72 not 70 lt lt gt g
57. n of a term FrameScript encounters a function call it looks up the appropriate function subroutine to handle the call If the appro priate handler is a subroutine FrameScript will call the Term apply Instance Term StackFrame method defined for that subroutine The app1y O method has 3 parameters These are Instance currentObject If the function call being processed is in the body of a daemon this is the instance frame seript which caused the daemon to run otherwise this is null Term args This is an array that holds the arguments being passed to the subroutine The 1st value in the array is the name of the subroutine being called The actual arguments begin at index 1 eg if the subroutine is called with 2 arguments args length will be 3 StackFrame frame This is a reference to the memory in which the local vari ables are being stored The arguments in args are stored in their unevaluated form To get the actual values for the arguments they need to be evaluated To evaluate an argument you need to call the 6081 6 method on the argument and to pass it currentObject and frame eg args 1 eval currentObject frame For the vast majority of subroutines the currentObject and frame argu ments will only be used in the evaluation of arguments and serve no other purpose If there are any problems encountered during the execution of a subroutine s apply O method the subroutine can elect to throw a sitcrc framescript FSError excepti
58. ndex of the argument to check frame StackFrame that houses variable values Usage Description Returns Throws Parameters C 1 17 check string check string String functionName Instance currentObject Term arg int n StackFrame frame Function for checking that a argument to a function is a string FSString FSError if the designated argument is not a string functionName name of the function that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n index of the argument to check frame StackFrame that houses variable values 84 Usage Description Returns Throws Parameters C 2 Files Modules TODO 2 1 loadFile TODO C 2 2 loadModule TODO C 2 3 setOutput TODO C 2 4 closeOutput TODO C 2 5 FileNotFound TODO C 2 6 evloop TODO C 3 Miscellaneous TODO C 3 1 compress TODO C 3 2 getMessage TODO C 3 3 getName TODO C 3 4 getPattern TODO 5 sortAtomList TODO 85 86 C 3 6 IOError TODO C 3 7 isPattern TODO C 3 8 isUnaryPattern TODO C 3 9 isPatternElement TODO C 3 10 formatComment TODO C 3 11 unformatComment TODO C 3 12 unformatComment TODO C 3 13 checkAllReferences TODO 87 D Serialisation TODO
59. ns that enable frames to be manipulated Several of the functions below require 67 current_object which means they are intended for use in daemons and apply to the daemon whose access manipulation resulted in the running of the daemon B 3 1 frame Usage frame term Description Tests whether or not a term is a frame Returns boolean Parameters term term being tested B 3 2 generic Usage generic term Description Tests whether or not a term is a generic frame Returns boolean Parameters term term being tested B 3 3 instance Usage instance term Description Tests whether or not a term is an instance frame NOTE As scripts are a subtype of instance frames this will return true if term is a script Returns boolean Parameters term term being tested B 3 4 instances of Usage instances of generic Description Gets all the instances of a generic frame Returns list Parameters generic generic frame whose instances are desired B 3 5 put Usage put slot term Description Puts the value term into the slot slot of current object Returns term Parameters slot the slot the value is being put into term the value to be added to the slot Error If there is no current_object slot already has a value and is not multivalued or an error is thrown by a daemon Usage put instance slot term Description Puts the value term into the slot slot of instance Returns term Parameters instance the instanc
60. ns for displaying GUls for communicating with a user Details on the siterc framescript UI module can be found in section 4 Implementationally speaking a module is simply a Java class with a public static void init method When the module is loaded the class is loaded into memory and the init method is executed Commonly found within the body of the init O methods of modules are subroutine declarations see An example of a module is given in section 2 4 2 2 4 1 Subroutines Subroutines are FrameScript functions whose implementations have been writ ten in Java rather than FrameScript The most common way of adding sub routines to FrameScript is by creating a module that declares and defines the subroutines desired To create a new subroutine in FrameScript an instance of the sitcrc framescript Subr class needs to be created for each subroutine being added The class Subr is abstract so it cannot be instantiated directly but must first be extended A concrete class that extends Subr needs an implementation of the public Term apply Instance Term StackFrame throws FSError method Probably the simplest way of implementing a subroutine is in the init O method of a module instantiating an anonymous class for each subroutine being created Each anonymous class would then implement the subroutine in its definition of the applyO method The MaxModule example see is written in this manner Constructors The sitcrc framescript Subr cla
61. on Argument Type Checking If your subroutines requires its arguments of a specific type there are some argument type checking methods in the sitcrc framescript Utils class that can help These methods were developed to check argument types and to provide standard error messages if the arguments are not of the correct type If the argument is not the correct type an FSError exception is thrown To make thing simpler and the error messages more detailed these argument checking methods will evaluate the arguments for you If the subroutine my_subroutine 33 expected an integer as its 1st argument the check integer method could be used the code would then look something like FSInteger i Utils check_integer my_subroutine currentObject args 1 frame There is a method in sitcrc framescript Utils to check the argument type for virtually all of possible types of term in FrameScript More details on the argument type checking methods can be found in appendix C Returning Values VVhen returning values in the apply O method any subclass of sitcrc framescript Term can be returned If you have a numerical value being returned then FSNumber getNumber can be used to create an appropriate FrameScript numerical type If a sub routine isn t really designed to return a value it is preferred that it returns sitcrc framescript Atom null rather than just null as it makes Frame Script less prone to unrecoverable crashes 2 4 2 MaxModu
62. on Creates a new MicaBot agent that uses the specified domain The agent transport is configured using the current micabot Returns micabot Parameters domain the domain the new MicaBot will use to talk to FrameScript Error If there are any problems creating the new MicaBot or there is no current MicaBot Usage micabot domain term Description Creates a new MicaBot agent that uses the specified domain The agent transport is configured using the current micabot Then it initialises the conversation with a given input Returns micabot Parameters domain the domain the new MicaBot will use to talk to FrameScript term an input used to initialise the conversation Error If there are any problems creating the new MicaBot or there is no current MicaBot 74 Usage micabot domain host integer Description Creates a new MicaBot agent that uses the specified domain The new MicaBot connects to the blackboard identified by the host and port parameters Returns micabot Parameters domain the domain the new MicaBot will use to talk to FrameScript host the host the blackboard resides on integer the port on the host the blackboard has open Error If there are any problems creating the new MicaBot Usage micabot domain term host integer Description Creates a new MicaBot agent that uses the specified domain The new MicaBot connects to the blackboard identified by the host and port parameters Then it initialises the con ve
63. on that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n index of the argument to check frame StackFrame that houses variable values 82 C 1 12 check number check number String functionName Instance currentObject Term arg int n StackFrame frame Function for checking that a argument to a function is a number FSNumber FSError if the designated argument is not a number functionName name of the function that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n index of the argument to check frame StackFrame that houses variable values Usage Description Returns Throws Parameters C 1 13 check pattern check pattern String functionName Instance currentObject Term arg int n StackFrame frame Function for checking that a argument to a function is a pattern Pattern FSError if the designated argument is not a pattern functionName name of the function that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n index of the argument to check frame StackFrame that houses variable values Usage Description Returns Throws Parameters C 1 14 check rdr check rdr String functionName Instance currentObject Term arg int n St
64. outines that allow the standard Mica function calls writeMob readMob register etc to be accessible in FrameScript so that they can be used within responses It also has the responsibility of turning Mica mobs into utterances and events to be parsed processed by FrameScript VVhen a MicaBot agent is first initialised it uses a TypeManager to explore the mob inheritance hierarchy It then recreates this hierarchy using generic frames in FrameScript On initialisation MicaBot registers for textFromUser mobs from which it uses the utterance slot to provide the speech input to FrameScript 5 1 Speech Alternatives Speech recognition systems don t just give one result when recognising speech but instead give a list of possible statements in the order of their likely probabil ities MicaBot allows these possible alternatives to be checked when processing the response to a user s statement In the textForUser mob the utterance slot holds the most likely alternative If there are other possible alternatives then they are placed in a list in the alternatives slot of the textFromUser mob failsafe NOMOREALTS lt Sorry 1 didn t understand that 1 gt 7 Figure 34 Speech Alternative Example The most common way to access the alternatives from within FrameScript would then be to use a failsafe similar to Figure In this example RECO GALT is a simple response that tells MicaBot to use the next alterna
65. oved daemon is run whenever a value is removed from a frame e if replaced The if replaced daemon is executed when the value of a slot is replaced with a new value e range The range daemon is used to restrict the values allowed in a slot It is used every time a slot s value is added or replaced 13 e help The help daemon is run whenever an attempt to add a value to a slot that is disallovved by the range daemon default The default isn t really a daemon but a non evaluated default value used in the GUI for viewing manipulating frames 2 1 4 Imstance Frames Instance frames describe instances 01 generic frames as such they are like objects in OOP When a slot cannot be found within an instance frame FrameScript will search through the inheritance hierarchy for a procedure to compute the appropriate value for the slot john isa person with name John Doe age 35 Figure 3 Example Instance Frame Defining Slots There are four ways of giving an instance frame a value for a slot They are e Define the slot as cached and provide an if nev daemon Define an if needed daemon for a generic frame from which the instance inherits This defines the slot for all instances of the type e Define a value for the slot in the instance s declaration eg definition of name in Figure 3 e Put a value in the slot after it is instantiated Adding Values to Slots There are two ways to put a value in a slot for an ins
66. own way to get around this situation NOTE inside alternatives should be avoided where possible especially at the end of an alternative 2 1is c not 6 bc 24 25 yore UILA Z A 228 6 gt 9 L n EA ki q qx e qqe qoe qe q q e q olo ofa o d old 692 SA SOX Sox 692 SA 692 SA Sox ON 692 SA ON Sox Sox ON SOX UIIEIA uIPj1eZ4 SAA SOX SOX ON SOX SOX ON ON A SOX ye W uru roSsTy 2 3 Expression Language In order to attach procedures to objects a language must be defined in which those procedures can be written FrameScript uses a fairly simple untyped procedural language that includes some programmatical elements inspired by machine learning techniques FrameScript s untyped nature means that when we declare a variable we don t declare what type it is The variable can can take any type at anytime it is only when we come to use the variable s value that the type is taken into consideration 2 3 1 Statements FrameScript code is written as a sequence of statements that are evaluated in turn There are 2 types of statements currently available in FrameScript definition statements and expression statements Definition statements a
67. pression evaluates an atom to get its value The atom s value is evaluated in a rather specific order First if the atom is the name of a declared variable the value of the variable is used second if the expression being evaluated is part of a daemon attached to a frame and the frame has a slot whose name is the atom the value of the slot is used third if the atom is then name of an object the object is used otherwise the atom itself is used as the value Some examples of atoms are red hello X X23 my_frame 2 3 5 Lists Lists are a compound data type that allow collections of terms to be grouped Lists can contain any type of term including numbers atoms lists frames When a list is evaluated the result is a list whose elements are the evaluation results of the corresponding element in the original list eg 1 1 evaluates to 2 Some examples of lists are 1 a 1 231 a 1 1 1 1 2 3 6 Expression Lists Expression lists are a sequence of terms which are evaluated sequentially The expressions in the list are separated by The resultant value when evaluating an expression list is the result of the last expression in the list being evaluated eg 1 2 3 evaluates to 3 The following expression list shows the sequential nature of evaluating an expression list When it is evaluated the first expression a 1 is performed setting the valued of a to 1 Then the second expression a a 1 is evaluated whic
68. pt module into memory e Serialise Serialises FrameScript s symbol table to a file e Unserialise Loads the symbol table stored in a file e Help 39 Frame Browser File Help Generic Frames Instance Frames Functions Generic Frames Generic Frame test Name test2 kest2 Comment test3 Daemon Comment 1 Lee Transcript 192 3 Figure 25 Frame Browser Set Look amp Feel Changes the UI look and feel e About Describes the frame brovvser 4 4 2 Transcript In the lower section of the frame browser is the transcript The transcript 8 essentially a block of FrameScript code to be evaluated When the user clicks on the Eval button the frame browser will parse statements written in the transcript and evaluate them The results of the evaluation will be appended to the end of the transcript The purpose of the transcript is to allow sections of FrameSeript code to be written and evaluated without having to resort to console input 4 4 3 Generic Frames The upper section of the frame browser has a series of tabs The Generic Frames tab shows the generic frames that are currently named in the symbol table When a generic frame is selected from the list on the left its parents and slots will be displayed The frame browser can be used to add and remove parents from a generic frame or to rearrange the order in which parents are inherited FrameScript use
69. re used to declare and define generic frames instance frames and scripts Evaluation statements are used to define an expression that needs to be evaluated All statements whether definition or evaluation end with Figures 2 Bland Blare examples of definition statements while some expres sion statements can be found in figure 17 1 2 Hello vorld print Hello vorld head 1 2 3 foo bar bar 1 foo 2 II Figure 17 Evaluation Statements 2 3 2 Numbers Numbers in FrameScript can be either integers or real numbers When per forming mathematical operations the numerical type returned depends on the product of the operation not on the types of the operands eg 3 2 1 5 1 5 2 5 2 2 3 3 Strings A string is just a sequence of characters enclosed between s eg the grass is green roses are red 26 2 3 4 Atoms Atoms are a fairly simple type that provide FrameScript s expression language with a lot of flexibility and possible complexity Atoms are essentially sequences of non whitespace characters There are some restriction on the characters that comprise an atom imposed by the parser such as atoms cannot begin with any of the characters 0 9 and some characters such as refuse to associate with others Atoms are used as the names of objects such as frames or scripts the names 01 slots the names of variables and a values in their own right When an ex
70. rown by a daemon 69 replace remove B 3 6 Usage Description Returns Parameters Error Usage Description Returns Parameters Error B 3 7 Usage Description Returns Parameters Error Usage Description Returns Parameters Error Usage Description Returns Parameters Error destroy instance Destroys an instance frame Runs if_destroyed daemons and removes the instance from the instance lists of all it s parent generic frames Does not work on scripts nothing instance the instance frame to be destroyed If instance is a script or an error is thrown by a daemon B 3 8 destroy Usage Description Returns Parameters Error B 4 Seript Routines FrameScript provides a number of functions for moving from one script to an some functions to initiate conversations using domains script term Tests whether or not a term is a script boolean term term being tested domain term Tests whether or not a term is a domain boolean term term being tested pattern term Tests whether or not a term is a pattern boolean term term being tested register domain script Registers script as the dominant topic of domain nothing domain domain the script is a topic of script script that is being registered as a topic goto script Sets the given script as the current context of the current domain script script the script to be the current context If ther
71. rsation with a given input Returns micabot Parameters domain the domain the new MicaBot will use to talk to FrameScript term an input used to initialise the conversation host the host the blackboard resides on integer the port on the host the blackboard has open Error If there are any problems creating the new MicaBot B 6 2 mica connect Usage mica connect term Description Disconnects the current MicaBot and reconnects with the given name It then registers for textFromUser mobs Returns atom Parameters term the agent name to use when reconnecting Error If there are any problems reconnecting or there is no current MicaBot Usage mica_connect micabot term Description Disconnects micabot and reconnects with the given name It then registers for textFromUser mobs Returns atom Parameters micabot the micabot to disconnect and reconnect term the agent name to use when reconnecting Error If there are any problems reconnecting B 6 3 mica register Usage mica_register term Description Registers the current MicaBot for the given mob type Returns atom Parameters term the type of mob the current MicaBot is in terested in Error If there are any problems registering or there is no current MicaBot 75 Usage mica_register micabot term Description Registers micabot for the given mob type Returns atom Parameters micabot the micabot that is interested in the mob type term the type of mob
72. s an instance frame that inherits from the given type instance generic the generic frame being instantiated term 20 If the unevaluated term is an integer it gets the associated match component otherwise it just returns the evaluated value of term It s main usages are for retrieving sections of input for analysis and for dynamically generating responses term term either the index for a match component or term to be evaluated 60 B 1 15 not Usage Precedence Description Returns Parameters B 1 16 of Usage Precedence Description Returns Parameters B 1 17 in Usage Precedence Description Returns Parameters B 1 18 new Usage Precedence Description Returns Parameters B 1 19 Usage Precedence Description Returns Parameters term 20 Evaluates term and returns nothing Is is mainly used to insert FrameScript code for execution in responses that pro duce no visible response It also is used in the pattern for a rule to provide a means of including the state of the system in the conditions of the rule nothing term term to be evaluated number to number2 70 An operator to ensure that a value inserted into a slot falls within a given range It only works in daemons that define new value eg range if added if replaced boolean number inclusive lower bound of the range number2 inclusive upper bound of the range If new_value is not a nu
73. s an untyped language the arguments to the function can be of any type This means that if a function requires specific types for its parameters it must either explicitly check the types of its arguments or assume the types that the arguments posses Figure 19 shows the same maximising function as Figure 18 except it explicitly checks that the parameters for X and Y are numbers and if either isn t a number it throws an error max X Y if number X then if number Y then if X gt Y then X else Y else error Y is not a number else error X is not a number Figure 19 Function Type Checking Example FrameScript is implemented im Java and for some functions it may more efficient necessary for them to be written in Java rather than FrameScript Subroutines are functions that can be called like any other function from within FrameScript but are instead written in Java Detailed information about writing subroutines can be found in section 2 4 1 NOTE If the body of a function is an expression list it needs to be brack eted Functions in FrameScript are defined by giving the function specification and assigning it an expression that is to be evaluated whenever the function is called The most common way of defining a function is as a single expression in a statement The code that defines the function is treated by frame script as any other expression so it can be placed in the body of a daemon or another function 30
74. s depth first searches for inheritance so this can change a frame s behaviour Slots can be added to or removed from a generic frame and the daemons attached to a slot in a generic frame can be added or removed The New button below the list of generic frames can be used to create a new generic frame 4 4 4 Instance Frames The mstance Frames tab in the frame browser gives access to all the instance frames for the known generic frames 40 Frame Browser Instance Frame Name t1 Comment Parents test Remove Move Up Move Down Transcript 1 3 Figure 26 Instance Frame Browser When a generic frame is select all of the instance frames that inherit from it will be listed When one of these instances is selected its parents and slots will be displayed The frame browser can be used to add or remove the parents and slots of instance frames and the reorder the parents of an Instance frame The New button below the list of instance frames can be used to create a new instance frame that inherits from the currently selected generic frame 4 4 5 Functions Frame Browser Instance Frames T Move Up pi Move Down if lt Y then x else Y Transcript 1 3 Figure 27 Function Browser The Functions tab of the frame browser allows users to view create func tions in FrameScript NOTE t
75. sage dialog_message term Description Opens a dialog box with a message for the user Returns nothing Parameters term a term used to provide the message for the user B 5 2 dialog question Usage dialog_question term Description Opens a dialog box requesting an input from the user Returns term Parameters term aterm used to provide the input request mes sage Error If there is an error parsing the user s input B 5 3 frame browser Usage frame browser Description Opens the frame browser user interface Returns nothing B 5 4 script browser Usage script browser Description Opens the script browser user interface Returns nothing 73 B 5 5 fix rdr_gui Usage fix gui 0 Description Opens a graphical RDR maintenance interface Returns nothing Error If there is no current object RDR has been evaluated no conditions are given to explain the difference no value in the slot of current object B 6 MicaBot When using MicaBot several subroutines are loaded into FrameScript to allow access to the mobs on Mica s blackboard Along with some new functions Mi caBot defines some new term types Many of the new functions use these new types The new types are listed below host the name of a machine hosting a blackboard will usually be a string micabot a MicaBot mob an instance frame that inherits from the mob generic frame B 6 1 micabot Usage micabot domain Descripti
76. scription Returns Throws Parameters C 1 4 check compound check compound String functionName Instance currentObject Term arg int n StackFrame frame Function for checking that a argument to a function is a Compound Compound FSError if the designated argument is not a Compound functionName name of the function that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n index of the argument to check frame StackFrame that houses variable values Usage Description Returns Throws Parameters C 1 5 check domain check domain String functionName Instance currentObject Term arg int n StackFrame frame Function for checking that a argument to a function is a Domain Domain FSError if the designated argument is not a Domain functionName name of the function that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n index of the argument to check frame StackFrame that houses variable values 80 Usage Description Returns Throws Parameters C 1 6 check exprlist check exprlist String functionName Instance currentObject Term arg int n StackFrame frame Function for checking that a argument to a function is an Exprlist ExprList FSError if the designated argument is not a ExprList functionName name of the
77. shown in figure 31 It is recommended that if a script is using the rule response mechanism for its rules that the rules are added edited using the standard rule interface and not using the recognition daemon interface as whenever a script is selected the script browser checks the recognition daemon to see whether or not the normal pattern response rule interface can be used and writing recognition daemons that can be displayed though this interface is not a simple task 44 Script Browser D script EZ functions D maroc n maya A hen lt lt a gt gt else if match lt lt B l 6 then gt gt 5 else if match lt lt gt gt then Figure 31 Abnormal Script Browser Script Browser Y test_script J functions maxv Dau Definition 1808806 Y 2 Figure 32 Function Browser 45 4 5 4 Functions If a user selects a function from the tree then that function s details are shown Both the function s parameters and body are shown and can be manipulated Beware when using this interface to add remove reorder parameters for a function as all calls to the function will need to be checked to ensure that th
78. ss defines 2 constructors that can be used in instantiating a subroutine They are e public Subr String str e public Subr String str int nArgs throws FSError The code that links the subroutine into FrameScript is located inside these constructors so one of them needs to be used when creating the subroutine Both constructors take a String argument this is the name of the function being created If the my function subroutine were being implemented the argument should be my function Optionally the number of arguments the function expects can be given to the constructor FrameScript will then ensure that the subroutine is only run when the function is called with the expected 32 number of arguments If no expected number of arguments is given or it is less than 0 the subroutine will be run whenever the function is called regardless of the number of arguments provided by the function call If when a subroutine is instantiated with a given number of arguments there already exists a sub routine with the same name that handles an undefined number of arguments an exception will be thrown If a subroutine can expect a variable number of arguments it is possible to instantiate a subroutine with the same name for each possible number of arguments or to instantiate a single subroutine that takes an undefined number of arguments and in the body of the subroutine check the number of arguments given apply Method When during the evaluatio
79. ssion language it is possible to create a script that responds with the instance frame whose slot matches a user s input contact_lookup var rval forall C in instances_of contacts if undefined rval and match lt lt name of C gt gt then rval C rval send mail Send the email to lt contact_lookup gt send_email email of 71 the_email Figure 16 Abnormal Seript Example Abnormal Script Example Figure 16 gives an example of an abnormal script contact lookup that loops through the list of all contacts to find a contact whose name matches the input Any matching contact found is then returned Then if send mail is the current context and the user says Send the email to John the rule will match as long as there exists a contact with the name John It will then send the email to John 2 2 12 Pattern Matching Algorithm By default FrameScript uses a greedy matching algorithm without backtracking which means some patterns may not work as expected For many patterns that do not work as expressed it is possible to rewrite the pattern so that it will work Table 2 shows some examples of patterns that do and don t work A second fully back tracked matching algorithm has been implemented for those applications that require it but it needs to be activated NOTE Due to the greedy nature of the algorithm there is an implicit at the end of every pattern except when used as a non terminal Currently there is no kn
80. t to in 50 infix infix new 40 mod 20 10 prefix prefix 5 of Table 4 Operator precedence B 2 General Functions FrameScript has a number of standard functions for testing the types of terms and to manipulate list It also provides mechanisms for interacting with the user B 2 1 trace Usage trace boolean Description Enables disables trace reporting Returns boolean Parameters boolean true to enable trace reporting false to disable B 2 2 verbose Usage verbose boolean Description Enables disables verbose output Returns boolean Parameters boolean true to enable verbose output false to disable 62 atom term Tests if term is an atom boolean term term to be tested defined term Tests if term has a defined value boolean term term to be tested B 2 3 atom Usage Description Returns Parameters B 2 4 defined Usage Description Returns Parameters B 2 5 undefined undefined term Tests if term does not have a defined value boolean term term to be tested set atom term Tests if term does not have a defined value term atom name of a variable to be given a global value term global value to be given to atom number term Tests if term is a number boolean term term to be tested integer term Tests if term is an integer boolean term term to be tested list term Tests if term is a list boolean term term to be
81. t expression disjunction expression disjunction expression conjunction expression disjunction expression or conjunction expression conjunction expression relational expression conjunction expression and relational expression relational expression additive expression comparison expression comparison operator additive expression relational operator lt p gt ES 5 jt to in additive expression multiplicative expression ladditive expression multiplicative expression ladditive expression multiplicative expression multiplicative expression unary expression multiplicative expression unary expression multiplicative expression unary expression multiplicative expression mod unary expression unary expression slot retrieval unary expression unary expression nevv unary expression not relational expression unary expression unary expression slot retrieval factor of unary expression factor atom number string list expression list lt lt pattern gt gt rdr expression forall expression variable declaration compound 54 list P list values list values expression list list values forall expression forall atom in expression expression variable declaration
82. t its evaluation will result in an error A var statement is var followed by a non zero comma separated list of variable names The variable names must be atoms Some valid var statements are var X var X Y When the body of the function fora11 loop daemon is finished being eval uated the memory used for storing the values for the local variables is freed and the values lost A simple example using local variables is example X Y var Z Z X mod Y Y mod 2 Ji When FrameScript is evaluating a daemon is declares some common variables that may be used in the body of the daemon These variables are current object This is the instance frame script which caused the daemon to be run current 810 This is the name of the slot which the daemon is assigned to new value When a value is being added to a slot this is the value being added In the on entry and on_exit daemons this is the context being switched to l old value When a value is being added to a slot this is the previous value of the slot VVhen a value is being removed from a slot this is the value being removed In the on entry and on exit daemons this is the context being switched from 2 4 Modules Modules a a way in which FrameScript can be extended to make it more flexible Modules can be loaded into FrameScript using the load module function A standard module provided within the FrameScript distribution is sitcrc framescript GUl which provides some functio
83. tance They are e put instance slot value eg put john age 7 e slot of instance value eg age of john 7 However be aware that both of these methods with fail if there is already a value in the slot and it isn t multivalued 2 1 5 Multiple Inheritance FrameScript allows multiple inheritance 1 0 a frame may inherit attributes from more than one generic frame When FrameScript looks up a value for an item the inheritance hierarchy is searched in a depth first left to right manner 14 simon isa person rabbit with name Simon age 4 fur brown Figure 4 Multiple Inheritance Example greeting domain example init hello hello hi 7 i m hi 1 hello hi hi how are you goodbye bye bye see you soon Figure 5 Script Example 15 2 2 Scripts Scripts are used to provide FrameScript with rule based processing for control of conversational interactions The rule system used in FrameScript is heavily influenced by the system used in ProBot 7 Each script contains a list of rules that are matched against a user s input and used to determine the appropriate response In addition to their rules scripts can also make use of inheritance and slots to allow reuse of scripts and retaining contextual information Because rules are used in a first come first served basis more specific rules should be placed before more generic rules in a script 2 2 1 Rules
84. tanding system Integrating knowledge to resolve uncertainty ACM Computing Surveys 12 2 213 253 1980 4 Mohammed Waleed Kadous and Claude Sammut The MICA Manual 0 1 edition 2003 5 Mohammed Waleed Kadous and Claude Sammut Mica Pervasive mid dleware for learning sharing and talking In PerCom Workshops pages 176 180 IEEE Computer Society 2004 50 Marvin Minsky A framework for representing knowledge Technical report Massachusetts Institute of Technology Cambridge MA USA 1974 Claude Sammut Managing context in a conversational agent Electronic Transactions on Artificial Intelligence 5 B 189 202 2001 Alan M Turing Computing machinery and intelligence Mind 59 236 433 460 October 1950 51 A BNF Below is a BNF grammar defining the syntax for FrameScript The terminal symbols atom number and string have been left undefined statement frame declaration lexpression list frame declaration generic frame linstance frame seript generic frame atom ako parent list with slot definitions instance frame atom isa parent list with slot values parent list atom parent list slot definitions atom daemon definitions daemon definitions daemon definition daemon definitions 1 daemon definition daemon name expression list slot type true slot type false daemon name
85. the MicaBot is interested in Error I there are any problems registering B 6 4 mica unregister Usage mica_unregister term Description Unregisters the current MicaBot for the given mob type Returns atom Parameters term the type of mob the current MicaBot is no longer interested in Error If there are any problems unregistering or there is no current MicaBot Usage mica unregister micabot term Description Unregisters micabot for the given mob type Returns atom Parameters micabot the micabot that is no longer interested in the mob type term the type of mob the MicaBot is no longer interested in Error If there are any problems unregistering B 6 5 mica read mob Usage mica read mob term Description Uses the current MicaBot to get the named mob Returns mob or nothing Parameters term the name of the mob to be read Error If there are any problems reading the mob or there is no current MicaBot Usage mica read mob micabot term Description Uses the MicaBot to get the named mob Returns mob or nothing Parameters micabot the micabot to read the mob term the name of the mob to be read Error If there are any problems reading the mob B 6 6 mica write mob Usage mica_write_mob mob Description Writes a mob using the current MicaBot Returns atom Parameters mob the mob to be written Error If there are any problems writing the mob or there is no current MicaBot 76 Usage mica_write_mob micabot mob
86. timodal Inter agent Communication Architecture is a middleware layer for pervasive computing that is especially well suited to sharing of infor mation between users learning user s preferences and interacting with the user through many devices and modalities It is based on the idea of a blackboard a global shared memory which acts as both a communication and storage mech anism It uses an extremely simple API that is easy to program but still small enough to fit on PDAs and mobile phones In designing MICA the design goals were to e Allow applications to be built in such a way that the interface to the ap plication and the application themselves are clearly separated to support multimodal interaction e Operate in heterogeneous environments in particular it should be able to run on low power and mobile devices This also means that it should support low level programming languages such as C e Allow users to use multiple input and output modalities and devices such as speech gesture audio and video at the same time allowing both mul timodal input fusion and multimodal output generation For example the user should be able to begin answering an e mail on her PDA using speech then when she gets to her 01166 continues working on the same e mail on her desktop from where she left off Support learning of user s preferences and patterns of usage e Use a simple abstraction that developers can learn and use quickly e Support
87. tions to be included in patterns There are two types of conditions that FrameScript allows and conditions are evaluated and if they don t evaluate to true the pattern doesn t match conditions are evaluated and the result of the evaluation is matched against the current point of the input greeting have_met_user_before hello hello its good to see you again not have_met_user_before hello hi its good to meet nev people Figure 6 Rule Condition Example One interesting application of the condition test is that if the condition evaluates to a generic frame then any instance frame that inherits from the given generic frame will match This allows scripts to be written that perform actions in response to events such as use of a touch pad or arrival of an email as long as the event causes an appropriate instance frame to be created and used as an input email ako object with from if_needed Anonymous new_email email you have a nev email from from of 71 1 Figure 7 Rule Evaluation Condition Example 2 2 3 Responses Responses tell the system how it is to respond to a user s input Sequences To avoid writing scripts that users may find repetitive responses can include a sequence of responses where each response is given in turn every time the pattern is matched After the last response is given it returns to the first response in the sequence Alterna
88. tive If there are no more alternatives to try MicaBot will give the input NOMORE ALTS 5 2 SimpleText Agent SimpleText Agent is a fairly simple Mica interface agent that writes textForUser mobs to the blackboard and displays textForUser and textFromUser mobs as they arrive 5 3 MicaRunner If you wish to use MicaRunner to start stop MicaBot then there are some parameters that can be passed to the MicaBot agent These arguments are domain the name of the FrameScript domain that the MicaBot uses to define its responses file the name of a script file to be loaded init a string used to initialise the conversation interaction 47 SimpleTextAgent Welcome to today s demonstration play a DVD where is john Sorry didnt understand that i wantto watch ABC tv Figure 30 SimpleText Agent lt runner host localhost port 8500 gt lt blackboard gt lt restore value false gt lt debug level information gt lt blackboard gt lt agent class sitcrc framescript SimpleTextAgent gt lt agent class sitcrc framescript MicaBot gt lt arg param file value example frs gt lt arg param domain value example gt lt arg param init value init gt lt agent gt lt runner gt Figure 36 MicaRunner Startup Script Example 48 transient either true or false it tells MicaBot whether or not to make its textForUser responses transient defaults
89. tives Alternatives are similar to sequences except instead of responding with the 17 first response second response third response Figure 8 Sequence Example next response in the sequence the response is selected randomly each time from the list of all responses a response another response yet another response Figure 9 Alternatives Example Sometimes a response requires that the system do something not just say something If a 27 is found in the response the following expression is evaluated and the result discarded This allows a response to alter the system s state while not burdening the user with unnecessary dialog A common use of is goto a_script which moves a conversation interaction from one context to another In a response is similar to except that when the following expression is evaluated it is inserted into the response not thrown away The other difference is that if is followed by an integer then the numbered pattern component see 2 2 4 associated with that integer is placed in the response Conditional Responses Some responses may be dependent on some condition holding true Condi tional responses can be constructed in the form condition gt response if true condition gt response if false Figure 10 Conditional Response Example 2 2 4 Pattern Component Numbering Some pattern elements create a numbered
90. to true The parameters should be used in this order transient file domain init A simple example startup script for MicaRunner that starts MicaBot and SimpleText Agent is given in Figure This script will start MicaBot with the file example frs loaded using the example domain It will then initialise MicaBot by giving it init as its first input 6 Discussion While it will never be a simple matter to write scripts for use in conversational agents it is possible to simplify the task FrameScript s use of script inheritance is such a technique Using script it is possible to define a set of basic behaviours in a script 16 18 then is a simple task to inherit these behaviours in subsequent scripts so that these behaviours can be used or altered This allows common behaviours to defined only once rather than for each context in which they occur This greatly simplifies editing scripts as the scripter only has to apply modification at one point and not have to search for each implementation of the behaviour Abnormal scripts are also a useful tool for scripting because they allow for the look up of objects frames whose characteristics match a given portion of an input text Not only do abnormal scripts allow the look up of objects they can do so in a way that means the look up is performed as part of the function that compares the rule to the input Because an abnormal script can be used to match against the characteristics of all instances of
91. urns Parameters Error B 4 8 new event new_event domain term Gives a new input to a domain to be responded to pattern domain the domain that is being given a nev input term a term to be used as an input to the domain If an error is thrown during processing of the response bot domain Starts a simple console conversational interface using domain as the domain for the conversation atom domain the domain that is being used to determine the system s responses If an error is thrown during the processing of an input 71 Usage Description Returns Parameters Error B 4 9 bot Usage Description Returns Parameters Error bot domain term Starts a simple console conversational interface using domain as the domain for the conversation Starts the conversation using term as the first input atom domain the domain that is being used to determine the system s responses term a term used to initialise the conversation If an error is thrown during the processing of an input match term Comparison to see if term is a valid pattern to match the current position in the current input boolean term a term to be used as a pattern to be matched against the current input If there is no current input match term term2 Comparison to see if term is a valid pattern that matches term2 boolean term aterm to be used as a pattern to be matched against the term2 term2 aterm to matched against the pattern
92. ves name of the function that is checking the ar gument enclosing object for the code which called the subroutine array of arguments to the function index of the argument to check StackFrame that houses variable values functionName currentObject arg n frame Usage Description Returns Throws Parameters C 1 2 check atom check atom String functionName Instance currentObject Term arg int n StackFrame frame Function for checking that a argument to a function is an Atom Atom FSError if the designated argument is not an Atom name of the function that is checking the ar gument enclosing object for the code which called the subroutine array of arguments to the function index of the argument to check StackFrame that houses variable values 79 functionName currentObject arg n frame Usage Description Returns Throws Parameters C 1 3 check boolean check boolean String functionName Instance currentObject Term arg int n StackFrame frame Function for checking that a argument to a function is either true or false Atom true or Atom false FSError if the designated argument is not true or false functionName name of the function that is checking the ar gument currentObject enclosing object for the code which called the subroutine arg array of arguments to the function n index of the argument to check frame StackFrame that houses variable values Usage De

FrameScript: A Multi-modal Scripting Language

Contents

Download Pdf Manuals

Related Search

Related Contents