Home

Context-aware augmented communication

image

Contents

1. 0032 Referring now to FIG 1B a first exemplary step 152 in accordance with the present automated method corre sponds to electronically gathering one or more data elements for use in determining a communication context Exemplary data elements may correspond to such items as user specifi cation speaker voice identification facial recognition speech content GPS compass data and or geolocation infor mation One or more software modules 101 as shown in FIG 1A may be configured for accomplishing the data gathering step 152 Exemplary data gathering software modules may include without limitation a speaker identification i e voice recognition module 102 facial recognition module 104 GPS data module 106 compass data module 108 geolo cation information module 110 speech recognition speech content determination module 112 bar code data module 113 and user specifications module 114 Each of these information gathering modules will be described more fully below 0033 One or more of the data gathering modules 101 generally may be used for communicator identification including but not limited to the speaker identification module 102 and or the facial recognition module 104 and or the speech recognition module 112 It should be appreciated that the data gathering modules described above may be useful for identifying communicators including not only the user of an AAC device but additionally or alternatively one or more com
2. A still further data gathering module 101 in accor dance with the presently disclosed technology more particu larly concerns a bar code data module 113 Bar code data module 113 may correspond to the software interfacing fea tures and resultant data provided when an AAC device includes an integrated bar code reader or when a bar code reader is attached as a peripheral device to an AAC device e g using a bar code reader scanner as peripheral device 507 in FIG 5 Bar codes readable by such a bar code reader scanner may be placed within a user s environment and be associated with one or more identifying items including but not limited to people things places events and the like Each bar code may then either store additional information associ ated with its identifying item or may contain information about an electronic link e g website URL RF transmission connection information etc to such additional information Bar code input information may particularly correspond to information used for communicator identification and or location identification aspects associated with identifying a communication context 0040 For example each friend or family member of an AAC device user may have a bar code associated therewith such that the AAC device user can scan a communicator s associated barcode when the AAC device user is interacting with such communicator This provides the AAC device user and the user s AAC device with an
3. FIG 1B provides a flow chart 150 of exemplary steps in such a method while FIG 1A provides a schematic overview 100 of exemplary software modules that can combine to implement selected of the steps such as those shown in FIG 1B and those otherwise disclosed in the present application In general the software modules of FIG 1A are categorized in one of three general areas data gathering modules 101 a communication context data structure 111 and a data processing module 121 Various embodiments of the presently disclosed technology may include some or all of May 31 2012 the modules provided in FIG 1A Similarly the steps pro vided in FIG 1B may be performed in the order shown in such figure or may be modified in part for example to exclude optional steps or to perform steps in a different order than shown in FIG 1B 0031 The modules shown in FIG 1 and the steps shown in FIG 1B illustrate various aspects of an electronically implemented computer based process Computerized pro cessing of electronic data in a manner as set forth in FIG 1B may be performed by a special purpose machine correspond ing to some computer processing device configured to imple ment such electronically implemented process For example a hardware embodiment is shown in FIG 5 which may be used to implement the subject process particularly where the modules shown in FIG 1A are stored in one or more of the memory media devices shown in FIG 5
4. affirmative identification of the communicator and in some cases an identification that is even more reliable than other identification means such as voice recognition speech recognition and the like Under standing that bar codes may not be available for every person or place one of ordinary skill in the art will appreciate that multiple identification modules in addition to barcode input modules may also be employed in an AAC device of the presently disclosed technology In addition to identifying the communicator each bar code read by a bar code reader scanner associated with an AAC device may thus provide variety of information associated with that individual For example a bar code may provide not only the name of an individual communicator but also information such as that person s birthday the names of his family members his hob bies address and the like The AAC device user thus has ready access to important information about such person and can then use that information in communicating with that person This information may be encoded directly within the optical parameters of a barcode Or alternatively each bar code provides information to a communication link e g an item specific URL where information about a communicator or other item can be stored and continually updated 0041 The types of bar codes and encoding used in accor dance with bar code data module 113 and any associated US 2012 0137254 Al re
5. drive that easily withstands shocks for example that may occur if the electronic device 500 is dropped Although FIG 5 shows three separate memory media devices 504a 5046 and 504c the content dedicated to such devices may actually be stored in one memory media device or in multiple devices Any such possible variations and other variations of data storage will be appreciated by one of ordi nary skill in the art 0064 Inone particular embodiment of the present subject matter memory media device 504b is configured to store input data received from a user such as but not limited to audio video multimedia files for analysis and vocabulary extraction in accordance with the presently disclosed tech nology Such input data may be received from one or more integrated or peripheral input devices 510a 5102 associated with electronic device 500 including but not limited to a keyboard joystick switch touch screen microphone eye tracker camera or other device Memory device 504a includes computer executable software instructions that can be read and executed by processor s 502 to act on the data stored in memory media device 5045 to create new output data e g audio signals display signals RF communication signals and the like for temporary or permanent storage in memory e g in memory media device 504c Such output data may be communicated to integrated and or peripheral US 2012 0137254 Al output devices such as a monitor or o
6. input devices e g peripheral device 510 may include but are not limited to a peripheral keyboard peripheral touch screen monitor peripheral microphone mouse and the like A camera 519 such as but not limited to an optical sensor e g a charged coupled device CCD or a complementary metal oxide semiconductor CMOS optical sensor or other device can be utilized to facilitate camera functions such as recording pho tographs and video clips and as such may function as another input device Hardware components of SGD 500 also may include one or more integrated output devices such as but not limited to display 512 and or speakers 514 0067 Display device 512 may correspond to one or more substrates outfitted for providing images to a user Display device 512 may employ one or more of liquid crystal display LCD technology light emitting polymer display LPD technology light emitting diode LED organic light emit ting diode OLED and or transparent organic light emitting diode TOLED or some other display technology In one exemplary embodiment a display device 512 and touch screen 506 are integrated together as a touch sensitive display that implements one or more of the above referenced display technologies e g LCD LPD LED OLED TOLED etc or others 0068 Speakers 514 may generally correspond to any com pact high power audio output device Speakers 514 may func tion as an audible interface for the speech generat
7. is operating in the determined commu nication context and provide to a user the message item suggestions as an array of selectable output items on the at least one electronic output device 9 The electronic device of claim 8 wherein the message item suggestions comprise one or more of vocabulary words phrases and symbols 10 The electronic device of claim 8 wherein said elec tronic device comprises a speech generation device and wherein said speech generation device further comprises a speaker for providing audio output of messages composed while using the speech generation device 11 The electronic device of claim 8 wherein the data elements comprise one or more communicator specific data elements defining a user or communication partner of the electronic device said communicator specific data elements comprising one or more of user specification data speech content data voice identification data and facial recognition data 12 The electronic device of claim 8 wherein the data elements comprise one or more location specific data ele ments defining a current or previous location of the electronic device said location specific data elements comprising one or more of GPS data compass data and geolocation data 13 The electronic device of claim 8 wherein said at least one electronic input device comprises a touchscreen or key board 14 The electronic device of claim 8 wherein said at least one electronic input device co
8. or external switch activation 0070 Referring still to FIG 5 SGD hardware components also may include various communication devices and or modules such as but not limited to an antenna 515 cellular phone or RF device 516 and wireless network adapter 518 For example antenna 515 may be provided to facilitate wire less communications among the components of SGD 500 and or between SGD 500 and other devices e g a secondary computer in accordance with one or more of a variety of RF communication protocols including but not limited to Blue tooth WiFi 802 11 b g n and ZigBee wireless commu nication protocols A cellular phone or other RF device 516 may be provided to enable the user to make phone calls directly and speak during the phone conversation using the SGD thereby eliminating the need for a separate telephone device A wireless network adapter 518 may be provided to enable access to a network such as but not limited to a dial in network a local area network LAN wide area network WAN public switched telephone network PSTN the Internet intranet or ethernet type networks or others Addi tional communication modules such as but not limited to an infrared IR transceiver may be provided to function as a universal remote control for the SGD that can operate devices in the user s environment for example including TV DVD player and CD player When different wireless communica tion devices are included within an
9. previously filed U S Provisional Patent Application entitled CONTEXT AWARE AUGMENTED COMMUNICA TION assigned U S Ser No 61 417 596 filed on Nov 29 2010 and which is fully incorporated herein by reference for all purposes BACKGROUND 0004 The presently disclosed technology generally per tains to systems and methods for providing alternative and augmentative communications AAC steps and features such as may be available in a speech generation device or other electronic device 0005 Electronic devices such as speech generation devices SGDs or Alternative and Augmentative Communi cation AAC devices can include a variety of features to assist with a user s communication Such devices are becom ing increasingly advantageous for use by people suffering from various debilitating physical conditions whether result ing from disease or injuries that may prevent or inhibit an afflicted person from audibly communicating For example many individuals may experience speech and learning chal lenges as a result of pre existing or developed conditions such as autism ALS cerebral palsy stroke brain injury and others In addition accidents or injuries suffered during armed com bat whether by domestic police officers or by soldiers engaged in battle zones in foreign theaters are swelling the population of potential users Persons lacking the ability to communicate audibly can compensate for this deficiency by the use of
10. s 502 Other memory media devices e g memory media devices 504b and or 504c are used to store data which will also be accessible by the processor s 502 and which will be acted on per the software instructions stored in memory media device 504a Computing processing device s 502 may be adapted to operate as a special purpose machine by executing the software instructions rendered in computer readable form stored in memory media element 504a When software is used any suitable programming scripting or other type of language or combinations of lan guages may be used to implement the teachings contained herein In other embodiments the methods disclosed herein may alternatively be implemented by hard wired logic or other circuitry including but not limited to application spe cific integrated circuits 0063 The various memory media devices of FIG 5 may be provided as a single portion or multiple portions of one or more varieties of computer readable media such as but not limited to any combination of volatile memory e g random access memory RAM such as DRAM SRAM etc and nonvolatile memory e g ROM flash hard drives magnetic tapes CD ROM DVD ROM etc or any other memory devices including diskettes drives other magnetic based storage media optical storage media and others In some embodiments at least one memory device corresponds to an electromechanical hard drive and or or a solid state drive e g a flash
11. the lawn in front of the Bilt more House 0046 It should be appreciated that additional information may also be collected that is pertinent to the context in which an AAC device user may find himself that may also be used in conjunction with the present technology For example the network communication interface 520 FIG 5 may be oper ated in conjunction with either the GPS data 106 or triangu lation information based on cellular tower locations to obtain a local weather report so that relevant context aware vocabu lary suggestions regarding for example an approaching thunderstorm may also be presented to the AAC device user 0047 Referring again to FIGS 1A and 1B once a com munication context has been identified in module 120 and any additional information has been gathered in modules 116 and 118 some or all of such data is provided to data process ing module 121 In data processing module 121 of FIG 1A step 158 of FIG 1B is implemented Step 158 involves elec tronically processing information identifying the communi cation context and or stored speech output and or stored navi gation location information to make language or other message item suggestions for potential use in the determined communication context Software instructions and rules for processing data may be stored in the process data module 122 of FIG 1A and the generated language suggestions may be stored in module 124 As previously described the proces
12. the use of GPS vs triangulation via cellular towers to obtain location information The context determination features of the pres ently disclosed technology may track the above exemplary operational features of an AAC device and other operational features to help analyze and determine the most appropriate communications context for a user 0044 Regardless of the sources of information including the ones mentioned above as well as other sources as may become apparent to those of ordinary skill in the art from a reading of the present disclosure these information sources all provide data to a communication context data structure 111 as shown in FIG 1A It is within the confines of the communication context data structure 111 that selected gath ered data elements are analyzed per step 154 of FIG 1B to May 31 2012 manually or automatically determine a communication con text The determined communication context may then be stored as a separate data variable represented by the commu nication context identification information 120 within com munication context data structure 111 In general communi cation context identification information 111 provides a profile of a user and or one or more of the user s communi cation partners and or one or more of the locations speech device specifications or other related aspects associated with device use The specifics of the profile are then used to develop communicator specific and or loc
13. E NAVIGATION LOCATIONS o MADE WHILE OPERATING IN THE DETERMINED COMMUNICATION CONTEXT ELECTRONICALLY PROCESS INEORMATION 158 IDENTIFYING THE COMMUNICATION CONTEXT Z AND OR STORED SPEECH OUTPUT AND OR SOFTWARE NAVIGATION LOCATION INFORMATION TO MAKE LANGUAGE SUGGESTIONS FOR POTENTIAL USE IN SUCH COMMUNICATION CONTEXT Fig 1B Patent Application Publication INTERNET Browso Folders GONTEXT AWARE ON OFF May 31 2012 Sheet 3 of 4 PICTURES CALENDAR US 2012 0137254 A1 CONTEXT KEYBOARD GEO LOCATION Patent Application Publication May 31 2012 Sheet 4 of 4 US 2012 0137254 402 Composed Message 406 404 SUGGESTED VOCABULARY DISPLAY CONTEXT VOCAB BILTMORE HOUSE GARDENS WINERY STABLES ASHEVILLE NORTH CAROLINA AMERICA S LARGEST PRIVATE HOME FRENCH BROAD RIVER 408 KEYBOARD 410 412 Fig 4 506 510 510b GPS Device 508 7 500 501 Cellular Phone or RF Device 515 Network Commun Interface s 516 Peripheral Device 507 Wireless Control Network 509 Adapter 518 Fig 5 512 US 2012 0137254 Al CONTEXT AWARE AUGMENTED COMMUNICATION CROSS REFERENCE TO RELATED APPLICATIONS 0001 N A STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 0002 N A PRIORITY CLAIM 0003 This application claims the benefit of priority of
14. SGD a dedicated com US 2012 0137254 Al munications interface module 520 may be provided within central computing device 501 to provide a software interface from the processing components of computer 501 to the com munication device s 0071 While the present subject matter has been described in detail with respect to specific embodiments thereof it will be appreciated that those skilled in the art upon attaining an understanding of the foregoing may readily produce alter ations to variations of and equivalents to such embodiments Accordingly the scope of the present disclosure is by way of example rather than by way of limitation and the subject disclosure does not preclude inclusion of such modifications variations and or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art What is claimed is 1 A method of creating context aware message item sug gestions for inclusion in composing messages for an elec tronic device said method comprising electronically gathering one or more data elements char acterizing one or more of a communicator and location associated with an electronic device electronically analyzing selected ones of the one or more data elements to determine a communication context wherein said communication context comprises a data structure that provides a profile defined by the analyzed data elements electronically processing information identifying th
15. US 20120137254A1 as United States a2 Patent Application Publication Pub No US 2012 0137254 Al CUNNINGHAM et al 43 Pub Date May 31 2012 54 CONTEXT AWARE AUGMENTED Publication Classification COMMUNICATION 51 Int Cl G06F 3 048 2006 01 75 Inventors BOB CUNNINGHAM Pittsburgh 52 US CL 715 835 US DAVID EDWARD LEE Pittsburgh PA US Ga ABSIBACT Systems and methods of providing electronic features for creating context aware vocabulary suggestions for an elec 73 Assignee DYNAVOX SYSTEMS LLC tronic include roy dina user interface Pittsburgh US design area having a plurality of display elements An elec tronic device user may be provided automated context aware 21 Appl No 13 304 022 analysis from information from plural sources including GPS compass speaker identification voice recogni tion facial identification speech content determination user 22 Filed Nov 23 2011 specifications speech output monitoring and software navi gation monitoring to provide a selectable display of sug Related U S Application Data gested vocabulary previously stored words and phrases or a keyboard as input options to create messages for text display 60 Provisional application No 61 417 596 filed on Nov and or speech generation The user may optionally manually 29 2010 Browse Folders specify a context PICTURES RETURN a Patent Ap
16. Xpress device Tango device device and or DynaWrite products a mobile computing device a handheld computer a tablet computer e g Apple s iPad tablet a mobile phone a cellu lar phone a VoIP phone a smart phone a personal digital assistant PDA a BLACKBERRY device a DROID a TREO an iPhone an iPod Touch a media player a navigation device an e mail device a game console or other portable electronic device a combination of any two or more of the above or other electronic devices or any other suitable component adapted with the features and functionality dis closed herein 0060 When electronic device 500 corresponds to a speech generation device the electronic components of device 500 enable the device to transmit and receive messages to assist a user in communicating with others For example electronic device 500 may correspond to a particular special purpose electronic device that permits a user to communicate with others by producing digitized or synthesized speech based on configured messages Such messages may be preconfigured and or selected and or composed by a user within a message window provided as part of the speech generation device user interface As will be described in more detail below a variety of physical input devices and software interface features may be provided to facilitate the capture of user input to define what information should be displayed in a message window a
17. ader scanner hardware may be in accordance with a variety of known standards or standards as developed in the future that provide a suitable optical machine readable representa tion of data that is specific to each coded item Two dimen sional 2D or matrix barcode technology may be particularly applicable for use with the disclosed technology since such bar codes generally have a higher data representation capa bility than one dimensional 1D barcodes although 1D bar codes are not excluded Non limiting examples of matrix 2D barcodes for use with the disclosed technology include QR codes stacked barcodes multi segment barcodes high capacity color barcodes and the like 0042 Further with respect to step 152 of FIG 1B addi tional information that may be gathered for use in subse quently determining a communication context may include a user specifications module 114 In some instances the user specifications module 114 may include data corresponding to a user s manual specification of a particular context within which the user wants to operate For example a user s AAC device may be adapted with several different preconfigured communication contexts based on different people with whom the user interacts e g spouse caregiver friends etc or different places e g home work school AAC device can be provided with selectable software features for the user to manually select a communications context for thes
18. ation specific mes sage items for display to a user for selectable inclusion in messages being composed by the user on an AAC device 0045 Referring still to FIGS 1A and 1B it should be appreciated that additional information may be gathered once a communication context has been determined More particu larly step 156 of FIG 1B indicates that speech output and or software navigation locations made while operating in a given communication context as determined in step 154 may also be electronically stored Such data is indicated in FIG 1A as the modules for monitoring the speech output 116 of the AAC device as well as the software navigation data 118 1 navigation steps a user has followed during the operation of the AAC device Both of these features may be used to provide input that may be used to further expand vocabulary suggestions offered to the AAC device user For example if the user has caused the AAC device to ask a question of her conversation partner about the winery associated with the Biltmore estate vocabulary suggestions listing different types of wine or wine related terms may be included in a vocabulary suggestions list In like manner if a user has used the AAC device software to specify cellular tower location determination as opposed to more accurate GPS location vocabulary suggestions may be expanded to cover more dis tant locations for example downtown Asheville as opposed to the more precise location of
19. e com munication context to make message item suggestions for potential use while the electronic device is operating in the determined communication context and electronically providing to a user the message item sugges tions as an array of selectable output items on a display component of the electronic device 2 The method of claim 1 wherein the message item sug gestions comprise one or more of vocabulary words phrases and symbols 3 The method of claim 1 wherein said electronic device comprises a speech generation device 4 The method of claim 1 wherein said data elements comprise or more communicator specific data elements defining a user or communication partner of the electronic device said communicator specific data elements compris ing one or more of user specification data speech content data voice identification data and facial recognition data 5 The method of claim 1 wherein said data elements comprise one or more location specific data elements defin ing acurrent or previous location of the electronic device said location specific data elements comprising one or more of GPS data compass data and geolocation data 6 The method of claim 1 wherein said step of electroni cally gathering one or more data elements comprises scan ning a bar code placed within a user s environment said bar code identifying selected data elements associated with a communicator or location associated with the electronic d
20. e given operational environments The user specifications module 114 can then receive such user selected context infor mation and utilize it to automatically toggle a preconfigured communication context as opposed to automatically deter mining the best context based on other analyzed information e g communicator identification information and or loca tion information 0043 With further respect to user specifications module 114 the user specifications module may track the operational features of an AAC device selected by a user It should be appreciated that an AAC device user may select certain opera tional features and the way those features are configured may indicate something about the user For example a user may choose to operate his AAC device such that messages are composed with text only with symbols only or with a com bination of text and symbols In another example a user may choose to operate his AAC device with one of many different input options such as but not limited to the Touch Enter Touch Exit Touch Auto Zoom Scanning Joystick Auditory Touch Mouse Pause Headtrackers Morse Code and or Eye Tracking access modes still further example the previously mentioned camera input may be altered to permit selection of an external camera by way of a peripheral device 507 FIG 5 USB connection to the AAC device Other selection options may include selecting
21. ection button 202 Upon touching the CONTEXT selection button 202 graphical user interface area 200 is changed to display aCONTEXT graphical user interface area 300 0053 Withreference to FIG 3 there is illustrated a second exemplary embodiment ofa graphical user interface area 300 with a plurality of display elements in accordance with aspects of the presently disclosed technology More particu larly graphical user interface area 300 provides an enlarged screen area 302 on which may be viewed a live or recorded presentation from which vocabulary may be extracted in accordance with present technology Exemplary operational selection buttons are provided to for example select from buttons for a GPS 304 COMPASS 306 GEOLOCATION 308 AUDIO VIDEO INPUT 310 LIVE VIEW 312 and to activate or disable the context aware vocabulary process via CONTEXT AWARE ON OFF 302 As is evident from an inspection of FIG 3 a number of other exemplary options also available and provide other relevant operational options Those of ordinary skill in the art will appreciate that other or additional options may also be provided 0054 Selection of button 310 for and AUDIO VIDEO INPUT will enable inputs from a peripheral device e g peripheral device 507 illustrated in FIG 5 as will be further discussed later Alternatively selection of button 312 for LIVE VIEW may activate camera 519 and or microphone 508 also shown in FIG 5 0055 Upon selecti
22. eech sample can be used to determine speech content which may also be used to identify a user and or communicator For example speech recognition module 112 can use speech to text conversion software to convert a user s speech into resultant text to iden tify the speaker based on the speaker s conversation content In an exemplary non limiting implementation Dragon Naturally Speaking software by Nuance Communications Inc may be employed to provide speech to text conversion to provide text usable in a search engine to identify the speaker Similar text to speech conversion software may be used in the speech output monitor data module 116 which is described later in more detail 0035 In any instance of communicator identification fur ther processing of an obtained identification of a user and or communication partner such as by search of online or local databases will provide the user with relevant communicator specific message item choices as an aid to message compo sition Local databases could be stored for example in one of memory devices 504a 504b and or 504c FIG 5 and online databases may correspond to those provided by an online search engine for example without limitation Google Bing Snap Yahoo and Lycos that may be accessed via the Internet using onboard Network Communication Interface 520 FIG 5 of AAC device 500 0036 To appreciate the types of communicator specific language elements or re
23. evice 7 The method of claim 1 further comprising a step of electronically storing speech output and or software naviga tion locations made while operating in a determined commu nication context for subsequent use in making communica tor specific and or location specific message item suggestions for use when operating in the determined com munication context May 31 2012 8 An electronic device comprising at least one electronic output device configured to display a user interface area for composing messages as visual output to a user at least one electronic input device configured to receive electronic input defining one or more data elements characterizing one or more of a communicator and loca tion associated with the electronic device at least one processing device at least one memory comprising computer readable instructions for execution by said at least one processing device wherein said at least one processing device is configured to receive the one or more data elements characterizing the communicator and or location asso ciated with the electronic device analyze selected ones of the one or more data elements to determine a com munication context wherein said communication con text comprises a data structure that provides a profile defined by the analyzed data elements process informa tion identifying the communication context to make message item suggestions for potential use while the electronic device
24. f an electronic device configured to perform various steps as outlined above For example one exemplary embodiment concerns a tangible computer read able medium embodying computer readable and executable instructions configured to control a processing device to implement the various steps described above or other combi nations of steps as described herein 0016 Inone particular exemplary embodiment a tangible computer readable medium includes computer readable and executable instructions configured to control a processing device to analyze faces and or speech to recognize individual communicators i e the device user and or communication partners with whom the user is communicating and to sug gest language or other message items appropriate to the iden tified individual In further embodiments the executable instructions are configured to cause the display of identified context specific words and phrases in a scrollable selectable format on a display screen In certain embodiments the executable instructions are configured to employ identified context specific terms as search terms in a database and to display the results of such search terms as additional select able words and phrases In selected embodiments the com puter readable medium includes computer readable and executable instructions configured to apply facial recognition and voice identification algorithms to previously recorded and or real time data to identify individua
25. facial recognition speech content GPS compass data and or geolocation information One or more data gathering soft ware modules such as a speaker identification 1 voice recognition module facial recognition module GPS data module compass data module geolocation information module speech recognition speech content determina tion module bar code data module and user specifications module may be used to for communicator identification and or location identification 0012 Selected pieces of the gathered data elements are then electronically analyzed either to determine that a user has manually specified a communications context e g by selecting a preconfigured context within the user specifica tions module or to implement the automatic determination of a communication context based on the gathered data ele ments In general the manually or automatically determined communication context provides a profile of a user and or one or more of the user s communication partners and or one or more of the locations speech device specifications or other related aspects associated with device use 0013 The specifics of the profile are then used to develop communicator specific and or location specific message items e g words phrases symbols pictures and other lan guage items for display to a user for selectable inclusion in messages being composed by the user on an AAC device Additional message items or other lang
26. hat was selected is automatically enlarged for better visual recognition by a user In Scanning mode highlight ing is used in a specific pattern so that individuals can use a switch or other device to make a selection when the desired object is highlighted Selection can be made with a variety of customization options such as a 1 switch autoscan 2 switch directed scan 2 switch directed scan 1 switch directed scan with dwell inverse scanning and auditory scanning In a Joystick mode selection is made with a button on the joystick which is used as a pointer and moved around the touch screen Users can receive audio feedback while navi gating with the joystick In an Auditory Touch mode the speed of directed selection is combined with auditory cues used in the Scanning mode In the Mouse Pause Head trackers mode selection is made by pausing on an object for a specified amount of time with a computer mouse or track ball that moves the cursor on the touch screen An external switch exists for individuals who have the physical ability to direct a cursor with a mouse but cannot press down on the mouse button to make selections A Morse Code option is used to support one or two switches with visual and audio feedback In Eye Tracking modes selections are made simply by gazing at the device screen when outfitted with eye controller features and implementing selection based on dwell time eye blinking
27. home so that vocabulary sugges tions relative to that previous visit may be presented possibly based on optional settings selected by user specifications 114 0051 The present technology also may be equally applied in other context aware situations such as file or document management For example static or interactive files or docu ments may include elements susceptible of association with a present or past context Exemplary elements may include but are not limited to graphic audio video multi media word processing database or other files documents or elements within such items Such provision is well within the scope of the present technology and is well suited to situations where an AAC device user would wish to discuss a related visit or a planned future visit to a new location May 31 2012 0052 With reference now to FIG 2 there is illustrated a first exemplary embodiment of a graphical user interface area 200 with a plurality of display elements in accordance with aspects of the presently disclosed technology As may be seen graphical user interface area 200 may correspond to an initial interface area as presented upon power up of an AAC device constructed in accordance with present technology Upon power up graphical user interface area 200 provides a number of selection item buttons including EMAIL INTER NET MUSIC PICTURES CALENDAR GAMES etc and in accordance with the present subject matter a CON TEXT sel
28. ion device when computer processor s 502 utilize text to speech func tionality Speakers can be used to speak the messages com posed in a message window as described herein as well as to provide audio output for telephone calls speaking e mails reading e books and other functions Speech output may be generated in accordance with one or more preconfigured text to speech generation tools in male or female and adult or child voices such as but not limited to such products as offered for sale by Cepstral HQ Voices offered by Acapela Flexvoice offered by Mindmaker DECtalk offered by Fonix Loquendo products VoiceText offered by NeoSpeech prod May 31 2012 ucts by AT amp T s Natural Voices offered by Wizzard Microsoft Voices digitized voice digitally recorded voice clips or others A volume control module 522 may be con trolled by one or more scrolling switches or touch screen buttons 0069 The various input output and or peripheral devices incorporated with SGD 500 may work together to provide one or more access modes or methods of interfacing with the SGD In a Touch Enter access method selection is made upon contact with the touch screen with highlight and bold options to visually indicate selection In a Touch Exit method selection is made upon release as a user moves from selection to selection by dragging a finger as a stylus across the screen In a Touch Auto Zoom method a portion of the screen t
29. lated message items e g pictures symbols phrases and the like that may be developed in accordance with the disclosed technology consider the iden tification of a communication partner as a particular friend or acquaintance of the AAC device user A search ofa previously generated local database may result in presenting the user with a communicator specific message item list including such as the identified communicator s spouse s name chil dren s names pet s name home town job title hobbies or other related information Symbols and or phrases or other language elements or message items related to these commu nicator specific vocabulary choices may also be provided 0037 Referring still to FIG 1A some other data gathering modules 101 may generally include software features for identifying location information and include such examples as a GPS data module 106 compass data module 108 and geolocation information module 110 Such modules may be used individually and or collectively to provide information regarding a user s current or previously visited locations Location information can also be obtained by triangulation methods using cellular telephone tower locations using cel lular phone device 510 FIG 5 In one example if a GPS receiver associated with an AAC device provided location information of 35 32 25 56 N 82 38 06 46 W and a compass for example a fluxgate magnetometer compass also associated with
30. ls 0017 still further example another embodiment of the disclosed technology concerns an electronic device such as but not limited to a speech generation device including such hardware components as at least one electronic input device at least one electronic output device at least one processing device and at least one memory The at least one electronic output device can be configured to display a plu rality of graphical user interface design areas to a user wherein a plurality of display elements are placed within the graphical user interface design areas The at least one elec tronic input device can be configured to receive electronic input from a user corresponding to data for selecting one or more ofanumber of display element types to be placed within the graphical user interface area The at least one memory may comprise computer readable instructions for execution by said at least one processing device wherein said at least one processing device is configured to receive the electronic May 31 2012 input defining the various features of the graphical user inter face and to initiate a graphical user interface having such features 0018 In more particular exemplary embodiments of an electronic device the electronic device may comprise a speech generation device that comprises at least one input device e g touchscreen eye tracker mouse keyboard joy stick switch or the like by which an AAC device user may
31. ly disclosed technology DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 0027 Reference now will be made in detail to the pres ently preferred embodiments of the disclosed technology one or more examples of which are illustrated in the accompany ing drawings Each example is provided by way of explana tion of the technology which is not restricted to the specifics of the examples In fact it will be apparent to those skilled in the art that various modifications and variations can be made in the present subject matter without departing from the scope or spirit thereof For instance features illustrated or described as part of one embodiment can be used on another embodi ment to yield a still further embodiment Thus it is intended that the presently disclosed technology cover such modifica tions and variations as may be practiced by one of ordinary skill in the art after evaluating the present disclosure The same numerals are assigned to the same or similar compo nents throughout the drawings and description 0028 Referring now to the drawings various aspects of a system and method of providing electronic features for cre ating context aware message item suggestions for inclusion in composing messages for an electronic device are disclosed In general the subject technology provides features by which a user can be provided with a context aware library of com municator specific and or location specific message items such a
32. mprises a bar code scanner 15 The electronic device of claim 8 wherein said at least one processing device is further configured to store speech output and or software navigation locations made while oper ating in a determined communication context for subsequent use in making communicator specific and or location spe cific message item suggestions for use when operating in that determined communication context 16 A computer readable medium comprising computer readable and executable instructions configured to control a processing device to implement acts of electronically gathering one or more data elements char acterizing one or more of a communicator and location associated with an electronic device US 2012 0137254 Al electronically analyzing selected ones of the one or more data elements to determine a communication context wherein said communication context comprises a data structure that provides a profile defined by the analyzed data elements electronically processing information identifying the com munication context to make message item suggestions for potential use while the electronic device is operating in the determined communication context and electronically providing to a user the message item sugges tions as an array of selectable output items on a display component of the electronic device 17 The computer readable medium of claim 16 wherein the message item suggestions comprise one or more of vocabular
33. munication partners with whom a user is communicat ing For example speaker voice recognition speech recogni tion and or facial recognition can be variously used to identify just the user just the communication partner s or both par ties to a conversation This versatility can help provide a broader range of customization in accordance with the dis closed context specific communications options by creating a communication context that is dependent on one or more ofa variety of individuals whom are party to an electronically tracked conversation using an AAC device 0034 With more particular reference to the data gathering modules 101 that may be used for communicator identifica tion speaker identification module 102 can be used to iden tify a user and or communication partner via voice recogni tion techniques Such module 102 may correspond to an audio speaker identification program via voice recognition software analysis of audio received by for example micro phone 508 FIG 5 Speaker identification via voice recog nition can be implemented for example by comparing gath ered voice samples to a prerecorded library of known samples Identification of a user and or communication part ner may also be made via facial recognition module 104 in US 2012 0137254 Al conjunction with facial recognition software and an input from for example camera 519 FIG 5 Still further analysis of the words phrases etc contained in a sp
34. nd ultimately communicated to others as spoken output text message phone call e mail or other outgoing communica tion 0061 Referring more particularly to the exemplary hard ware shown in FIG 5 a computing device 501 is provided to function as the central controller within the electronic device 500 and may generally include such components as at least one memory media element or database for storing data and software instructions as well as at least one processor In the particular example of FIG 5 one or more processor s 502 and associated memory media devices 504a 504b and 504c are configured to perform a variety of computer implemented functions software based data services The one or May 31 2012 more processor s 502 within computing device 501 may be configured for operation with any predetermined operating systems suchas but not limited to Windows XP and thus is an open system that is capable of running any application that can be run on Windows XP Other possible operating systems include Android OS WebOS BSD UNIX Darwin Mac OS X including Cheetah Leopard Snow Leopard and other variations Linux SunOS Solaris OpenSolaris and Windows NT XPNista 7 0062 At least one memory media device e g device 504a in FIG 5 is dedicated to storing software and or firm ware in the form of computer readable and executable instructions that will be implemented by the one or more processor
35. on of button 302 to activate the context aware process a third exemplary embodiment of a graphical user interface area 400 with a plurality of display elements in accordance with aspects of the presently disclosed technol ogy will be presented to the AAC device user 0056 Upon selection of DISPLAY CONTEXT VOCAB button 406 a number of words phrases symbols and or other message items may appear on SUGGESTED VOCABU LARY area 404 corresponding to suggestions based on data contained in communication context data structure 111 FIG 1A Generally these words and phrases will correspond to words and phrases not normally included in an AAC device user s MY WORDS 410 or MY PHRASES 412 selection areas or in some other static vocabulary source initialized by the AAC device Although not presently illustrated in FIG 4 the words or phrases displayed in the SUGGESTED VOCABULARY area 404 may additionally or alternatively be shown with associated symbols By providing the sug gested words and phrases an AAC device user s communi cation capabilities are significantly enhanced when commu nicating with other individuals regarding the experienced presentation 0057 Itis noted that the AAC device user does retain the option of selecting a KEYBOARD input 408 through which she may type any desired word or phrase It should be appre ciated that upon selection of any of the buttons 406 408 410 US 2012 0137254 Al 412 a corresponding area 404
36. plication Publication May 31 2012 Sheet 1 of 4 US 2012 0137254 om a a a a a a a a a a ee ee ee eee DATA GATHERING MODULES 102 106 108 114 SPEAKER GPS COMPASS USER ID DATA DATA SPECIFICATIONS FACIAL GEOLOCATION SPEECH BAR CODE RECOGNITION INFORMATION RECOGNITION DATA I i I I I I 1 I 1 i 1 1 t 1 1 I I 1 1 1 l i I I I 1 l 4 H I COMMUNICATION CONTEXT IDENTIFICATION INFORMATION i 111 1 gue COMMUNICATION CONTEXT DATA 116 118 i i t i 1 4 i i i i 1 1 f I STRUCTURE 120 SPEECH OUTPUT SOFTWARE NAVIGATION MONITOR DATA MONITOR DATA 122 DATA PROCESSING MODULE 124 LANGUAGE SUGGESTIONS E PROCESS DATA on I I I I I 1 I I I I I Patent Application Publication May 31 2012 Sheet 2 of 4 US 2012 0137254 150 ELECTRONICALLY GATHER ONE OR MORE 152 DATA ELEMENTS E G USER SPECIFICATION SPEAKER VOICE IDENTIFICATION FACIAL RECOGNITION GPS COMPASS DATA AND OR GEOLOCATION INFORMATION FOR USE IN DETERMINING A COMMUNICATION CONTEXT ELECTRONICALLY ANALYZE SELECTED OF 154 THE GATHERED DATA ELEMENTS TO MANUALLY AUTOMATICALLY DETERMINE A COMMUNICATION CONTEXT ELECTRONICALLY STORE SPEECH OUTPUT 156 AND OR SOFTWAR
37. s words phrases symbols vocabulary or other lan guage elements for inclusion in composing messages Such feature allows the user to quickly interact with identified individuals and comment on people facts or information related to the identified individuals and or to a present or previously visited location or other related location or loca tion related places events or other information 0029 The ability to provide customized word and phrase selection libraries for an electronic device provides a variety of advantages For example interfaces can be created that provide enhanced response rates for alternative and augmen tative communications AAC device users wishing for example to engage in a discussion of a location being visited for the first time which includes words and phrases that are generally new or foreign to the vocabulary normally used or currently available to the user By providing a context aware vocabulary from which the user may select words or phrases specific to her location the user will be able to more readily compose messages relating to the material Context aware libraries will also reduce the cognitive load for the user and improve the overall learning experience 0030 FIGS 1A and 1B provide schematic diagrams related to a method of providing electronic features for cre ating an automated customized context aware message item choices interface for an electronic device in accordance with present technology
38. sclosed subject matter These drawings together with the description serve to explain the principles of the disclosed technology but by no means are intended to be exhaustive of all of the possible manifestations of the present technology 0021 FIG 1 provides a schematic diagram of exemplary software modules for use in a computerized method of pro viding electronic features for creating context aware lan guage suggestions for an electronic device 0022 FIG 1B provides a flow chart of exemplary steps in a method of providing electronic features for creating con text aware language suggestions for an electronic device 0023 FIG 2 depicts a first exemplary embodiment of a graphical user interface area with a plurality of display ele ments in accordance with aspects of the presently disclosed technology 0024 FIG depicts a second exemplary embodiment ofa graphical user interface area with a plurality of display ele ments in accordance with aspects of the presently disclosed technology 0025 FIG 4 depicts a third exemplary embodiment of a graphical user interface area with a plurality of display ele ments in accordance with aspects of the presently disclosed technology and 0026 FIG 5 provides a schematic view of exemplary hardware components for use in an exemplary speech gen US 2012 0137254 Al eration device for providing context aware vocabulary sug gestion features in accordance with aspects of the present
39. sing step 158 may involve conducting a local or online search US 2012 0137254 Al relating to identified items defining a communication context e g determined location determined communicator name etc 0048 Insome embodiments local and or online databases may be configured with predetermined or adaptable links among associated vocabulary elements to readily assist with the suggestion of communicator specific message and or location specific message items When links are adaptable a user can link words for future presentation when in a given communication context is determined When speech output and or location information is recorded in conjunction with a communication context vocabulary identified from the speech and or location can be linked to the communication context For example if location information helps identify as part of the determined communication context that the user is in Asheville N C then linked location specific vocabulary elements might include Asheville North Carolina Blue Ridge Parkway Biltmore House French Broad River and the like Having these location specific message items readily at hand can facilitate a user s communication regarding his determined location In another example if a communicator is determined to a user s acquaintance Tommy and speech output while within that communication context frequently references a dog named Spike and certain items related to the game of golf then ke
40. specify a context manually In certain embodiments the elec tronic device may be provided with a camera or other visual input means and or a microphone or other audio input means to provide analysis for facial and speech recognition In other instances the electronic device may be provided with a bar code scanner to read 2D matrix or other barcodes within a user s environment to assist with determining a communica tion context In still further embodiments an electronic device may be provided with at least one speaker for provid ing audio output In such embodiments the at least one pro cessing device can be further configured to associate selected ones of the plurality of display elements with one or more given electronic actions relative to the communication of speech generated message output provided by the electronic device 0019 Additional aspects and advantages of the disclosed technology will be set forth in part in the description that follows and in part will be obvious from the description or may be learned by practice of the technology The various aspects and advantages of the present technology may be realized and attained by means of the instrumentalities and combinations particularly pointed out in the present applica tion BRIEF DESCRIPTION OF THE DRAWINGS 0020 The accompanying drawings which are incorpo rated in and constitute a part of this specification illustrate one or more embodiments of the presently di
41. speech generation devices 0006 In general a speech generation device may include an electronic interface with specialized software configured to permit the creation and manipulation of digital messages that can be translated into audio speech output or other out going communication such as a text message phone call e mail or the like Messages and other communication gen erated analyzed and or relayed via an SGD or AAC device may often include symbols and or text alone or in some combination In one example messages may be composed by auser by selection of buttons each button corresponding to a graphical user interface element composed of some combi nation of text and or graphics to identify the text or language element for selection by a user 0007 Current advancements for speech generation devices have afforded even more integrated functionality for their users For example some SGDs or other AAC devices are configured not only for providing speech based output but also for playing media files e g music video multi media etc providing access to the Internet and or even making telephone calls using the device May 31 2012 0008 As the accessibility and communications function ality of SGDs continues to increase users need to be able to communicate with enhanced vocabulary and symbol set options Conventional fixed sources or databases of such communication elements are typically lacking in dynamic development of s
42. the AAC device indicated you were facing in a west north west WNW direction a search of for example Google Earth Google Maps or MapQuest online or a mapping database local to the AAC device would reveal that you are standing in the front lawn of the Biltmore House in Asheville N C and you are looking at the house 0038 The location information gathered via one or more of the GPS data module 106 compass data module 108 and geolocation information module 110 may be ultimately pro cessed similar to the communicator identification informa tion such as by search of online or local databases to provide May 31 2012 the user with relevant location specific message item choices as an aid to message composition For example a search for the Biltmore House would reveal geolocation information 110 including for example the name of the river passing along the property French Broad River and the fact that there are a winery stables and gardens associated with the property Such a search may also reveal that the Biltmore House is America s largest private home As will be described later with respect to FIG 4 in accordance with the present technology each of these items may be displayed as location specific vocabulary suggestions to an AAC device user to assist the user in carrying on a conversation Corresponding pictures symbols phrases and or other message items may also be developed for presentation to a user 0039
43. ther display device or as control signals to still further components 0065 Referring still to FIG 5 central computing device 501 also may include a variety of internal and or peripheral components in addition to those already mentioned or described above Power to such devices may be provided from a battery 503 such as but not limited to a lithium polymer battery or other rechargeable energy source A power switch or button 505 may be provided as an interface to toggle the power connection between the battery 503 and the other hardware components In addition to the specific devices discussed herein it should be appreciated that any peripheral hardware device 507 may be provided and interfaced to the speech generation device via a USB port 509 or other com municative coupling It should be further appreciated that the components shown in FIG 5 may be provided in different configurations and may be provided with different arrange ments of direct and or indirect physical and communicative links to perform the desired functionality of such compo nents 0066 Various input devices may be part of electronic device 500 and thus coupled to the computing device 501 For example a touch screen 506 may be provided to capture user inputs directed to a display location by a user hand or stylus A microphone 508 for example a surface mount CMOS MEMS silicon based microphone or others may be provided to capture user audio inputs Other exemplary
44. uage suggestions may be provided from a local or online search relating to identified items defining a communication context e g determined location determined communicator name etc Once par ticular message items are identified for suggestion to a user such message items may be provided as selectable output to a US 2012 0137254 Al user More particularly such items may be displayed on a screen associated with the AAC device preferably in an array of scrollable and or selectable items The displayed message items ultimately can be used by a user for composing mes sages for display and or conversion to synthesized or digital file reproduced speech and or remote communication to another via text email or the like 0014 Inother more particular exemplary embodiments a communication context data structure is provided that stores not only information identifying a context but also a history of speech output made in that context and or a history of software navigation locations made in that context This addi tional information can be electronically stored for use by a user In certain embodiments GPS and compass information may be used in conjunction with geolocation software for determining physical location and place information to sug gest language to use in a particular location context 0015 It should be appreciated that still further exemplary embodiments of the subject technology concern hardware and software features o
45. uch elements that could enhance SGD com munications functionality 0009 In light of the specialized utility of speech genera tion devices and related interfaces for users having various levels of potential disabilities a need continues to exist for refinements and improvements to context sensitive commu nications While various implementations of speech genera tion devices and context recognition features have been developed no design has emerged that is known to generally encompass all of the desired characteristics hereafter pre sented in accordance with aspects of the subject technology BRIEF SUMMARY 0010 In general the present subject matter is directed to various exemplary speech generation devices SGDs or other electronic devices having improved configurations for providing selected AAC features and functions to a user More specifically the present subject matter provides improved features and steps for creating context specific message item choice selections e g for such message items as vocabulary words phrases symbols and the like for inclusion in composing messages 0011 In one exemplary embodiment a method of provid ing automatic context identification is provided According to this automatic method one or more data elements for use in determining a communication context are electronically gathered Exemplary data elements may correspond to such items as user specification speaker voice identification
46. will be presented In this man ner for example a scrollable selectable group of words and phrases as illustrated in area 404 will be presented corre sponding to the selected input button 406 408 410 412 In the case of a KEYBOARD button 408 a QWERTY type keyboard may be displayed in area 404 to assist in typing words not present in any of the other selectable areas 0058 Referring now to FIG 5 additional details regard ing possible hardware components that may be provided to implement the various graphical user interface and media player creation features disclosed herein are provided FIG 5 depicts an exemplary electronic device 500 which may cor respond to any general electronic device including such com ponents as a computing device 501 at least one input device e g one or more of touch screen 506 microphone 508 GPS device 510a compass device 5105 camera 519 or the like and one or more output devices e g display device 512 speaker 514 a communication module or the like 0059 In more specific examples electronic device 500 may correspond to a stand alone computer terminal such as a desktop computer a laptop computer a netbook computer a palmtop computer a speech generation device SGD or alternative and augmentative communication AAC device such as but not limited to a device such as offered for sale by DynaVox Mayer Johnson of Pittsburgh Pa including but not limited to the V device Vmax device
47. y words phrases and symbols 18 The computer readable medium of claim 16 wherein said data elements comprise one or more communicator specific data elements defining a user or communication part ner of the electronic device said communicator specific data May 31 2012 elements comprising one or more of user specification data speech content data voice identification data and facial rec ognition data 19 The computer readable medium of claim 16 wherein said data elements comprise one or more location specific data elements defining a current or previous location of the electronic device said location specific data elements com prising one or more of GPS data compass data and geoloca tion data 20 The computer readable medium of claim 16 wherein said computer readable and executable instructions further configure the processing device to electronically store speech output and or software navigation locations made while oper ating in a determined communication context for subsequent use in making communicator specific and or language spe cific message item suggestions for use when operating in that determined communication context
48. ywords from such speech e g dog Spike golf with optional symbols or pictures may be presented as suggested message items to a user In this fash ion while in a given communication context some or all speech output and software navigation locations can be recorded and used to determine suggested language when next in the same communication context 0049 Once particular message items e g words phrases symbols pictures and other language items are identified for suggestion to a user such message items may be provided as output to a user More particularly such items may be displayed screen associated with the AAC device preferably in an array of selectable items In one example a scrollable selectable format can be used on a display screen for suggested message items Additional aspects of how exemplary language suggestions 124 may be presented to an ACC device user will be explained more fully later with respect to FIG 4 0050 It should be appreciated at this point that while the present exemplary embodiments are described in terms of a present context the present technology may be equally well applied to past contexts that may be contained within the communication context data structure 120 and may for example become part of a searched database from which vocabulary suggestions may be offered to an AAC device user For example the AAC device user may have previously visited some other famous

Download Pdf Manuals

image

Related Search

Related Contents

operation - Boss Audio Systems  Owner`s Manual  Bluetooth® Laser Mouse VGP-BMS20  Renesas Starter Kit for SH7124 User`s Manual  JVC LT-19E610 19" Black LED TV  User manual for all Peter Lynn Kites  FRH-SD07T取扱説明書  RTXC 3.2 TRAINING COURSE jrg.ppt  User Manual, Kanban Task Manager SharePoint Add-in  Chromalox CVEP-C User's Manual  

Copyright © All rights reserved.
Failed to retrieve file