User Manual - FROG Recognizer of Gestures
Contents
1. Disconnect Calibrate Sample Rate 100 Figure 4 3 Device connected and on device filters added System Walkthrough Page 7 User Manual FROG 4 1 2 Enabling Filters Enabling filters is accomplished by clicking the desired filter in the Filters available list and clicking the gt gt arrow button to add them to the Filters in Use list Upon clicking the arrow button a prompt will appear for parameters for that particular filter Two filters come with FROG Directorial Equivalence and Idle State Filters are applied in the order they are added After filters calibration and sampling rate are entered the connection pane may be closed Additional information about filters can be found in section 4 5 1 Filters 4 1 3 Training a Gesture To train a gesture it is important to first know how FROG operates FROG requires multiple instances of a gesture for good recognition results For example to have FROG recognize a circle begin by clicking New Gesture on the right side of the panel Figure 4 4 L FROG Recognizer of Gestures Training SAX Session Device Terminal 3D Acceleration Vectors 2d Acceleration Values New Session New Gesture IF v Sun SPOT 1BE9 v Status Connected Figure 4 4 Creating a new gesture An input dialogue will pop up asking for a gesture name2. Refresh Change Picture Remove Instance Sun SPOT 1BE9 V Status Connected Figure 4 10 The Training Panel with Gesture Edit panel open 4 1 6 Loading a Session Loading a session is similar to saving it Simply go to the Session menu and choose Load Session This will open a dialog like the one depicted in Figure 4 11 Session Device Terminal 3D Acceleration Vectors 2d Acceleration Values DENIS L Open Look In My Documents v c3 metadata CI My Videos brazos cs tcu edu Cd sdk 7 Camtasia Studio 3 sdkControl properties J Downloads 5 afrog My Music D Alex frog My Pictures 53 schedule D Q n 7 E3My shapes i Gesture File Name Alex frog Files of Type Al Files v Open Cancel Sun SPOT 1BE9 r Status Connected nd Of Gesture Complete Figure 4 11 The Training Panel Load Session open and Gesture Session Alex frog selected System Walkthrough Page 12 FROG User Manual Upon loading a session no further action is required unless a device is not yet connected Loading a session prevents a user from having to train new gestures every time FROG is used Gesture sets as well as individual instances can be added or deleted from a loaded session When a session is loaded it can be re modeled with different parameters by selecting
3. side of the training panel When Train is clicked a Training Settings menu will pop up Figure 4 8 From here various parameters can be modified The default settings are suitable for the purposes of this walkthrough To start working on another gesture follow the steps starting with clicking the New Gesture button System Walkthrough Page 9 User Manual FROG l Training Settings K Means Algorithm amp Use K Means Use K Means Quantizer K Value K means Threshold 0 01 HMM Threshold 0 0010 Number of HMM States 8 Train Cancel Figure 4 6 Training Settings Although this example suggests to Train after completing all instances of one gesture set this is not required The user may Train after entering a few gesture instances add more and train again The user may also wait to train after multiple gesture sets have been entered Additionally the user can return to a particular gesture set after working with other gestures and train All should work equally well It is important to note that performed gesture instances will apply to the selected gesture highlighted Click the gesture to select it System Walkthrough Page 10 User Manual FROG 4 1 4 Saving a Session After gesture instances are created and trained they can either be saved for later use or used in other modes without saving Save a session for later retrieval by simply g
4. diagram below Error Code LED position SPOT is searching for a basestation host 1 SPOT is connected successfully 2 Basestation discovery failure 3 Accelerometer reading failure 4 Filter failure 5 onoonnonoe Data transmission failure 6 56666000006 Data receive failure 7 Watcher Thread failure 8 Troubleshooting Page 21 User Manual FROG 5 2 System Output FROG allows for output of training and recognition results through console and system messages Refer to section 4 5 2 for terminal and log output Troubleshooting Page 22
5. Page 13 User Manual L FROG Recognizer of Gestures Recognition Hat Recognize Terminal User 1 demo Circle 100 0 Device Load Session Sun SPOT 1BE9 v Status Connected User 2 demo Not recognized Device Load Session e Sun SPOT 1E98 Status Connected User 3 demo A Triangle 98 72104 Device Load Session Sun SPOT 4118 v Status Connected frog frog plugin De User 4 New Session Device Load Session No Device Connected Status Device not found Figure 4 13 Recognition Panel with connected users System Walkthrough FROG Page 14 User Manual FROG 4 3 Evaluation Mode Evaluation mode is similar to recognition mode in that it recognizes gestures but it provides more detailed feedback to a single user only The system prompts the user to perform a gesture and tracks statistics based on what the user performed Evaluation mode includes a device settings panel recognition results panel and the standard log panel Figure 4 14 S FROG Recognizer of Gestures Evaluation BAX Evaluation Device Terminal Customize Evaluation Session Gesture Session Loaded demo Make a Sample Size 10 C Randomize Triangle Stop Gesture Correct Incorrect Not Recognized Percent Correct Average Certainty a v Circle 6 4 0 60 0 100 0 v Square 9 0 1 90 0 100 0 Triangle
6. 1 4 0 20 0 100 0 Circle Square Triangle X Z Session 20 15 10 Number 5 Correct Incorrect Not Recognized Event frog Bayesian Sun SPOT 1BE9 ogni re Circle with probability 1 0 P Status Connected frog plugin D ture Complete 4 T na Figure 4 14 Evaluation panel Using a gesture library the user will select which gestures from that library will be evaluated The system provides the ability to present a user specified number of test gestures in sequential or random order When sample size and gestures to be evaluated are chosen evaluation can be started with the Start button Clear Stats will remove recognition results from the recognition System Walkthrough Page 15 User Manual FROG results panel but only after the all samples have been generated The results panel displays the number of correct incorrect or not recognized gestures as well as average certainty of correct gestures Additionally graphical display of these values is also available Saving a session after evaluation causes the evaluation results to be added to the frog file Reloading a session with saved evaluation results will display previous recognition results 4 4 Demo Mode In this mode up to four users can connect to play a competitive game using pre trained gesture libraries Each library should be user specific for the best recognition accuracy Initially this mod
7. Delete frog Bayesian Ej cognized Gesture Circle with probability 1 0 Triangle frog plugin Devi a d Of Gesture Complete E Sun SPOT 1BE9 bd Status Connected Figure 4 21 The Training Window with Terminal toolbar open System Walkthrough Page 19 User Manual frog lal frog plugin D ureHMM Trained HMM Number of ste Training thr pi 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 D 0 0 0 D D 0 0 0 0 0 D 0 0 0 0 m 0 o 0 0 4 0 0 0 0 0 0 0 4410 0 Figure 4 22 Detached Terminal window System Walkthrough Page 20 User Manual FROG 5 Troubleshooting 5 1 Errors Malfunctions and Emergencies The most conspicuous error message received will be a gesture not recognized message when in evaluation mode This message simply indicates either the training or evaluating gesture were different than what was intended Let the system try to recognize the gesture a few more times If results don t improve re train the gesture from the main training mode In normal use there should not be any errors during runtime of the code itself but if that is not the case errors will be outputted to the console and written to the log file if that option is enabled The onboard LEDs serve to display error and status messages regarding normal operation The following table displays the possible LED codes encountered with the position marked on the
8. FROG Recognizer of Gestures User Manual Team Better Recognize Version 2 0 April 28 2010 F R OG Copyright 2009 2010 Computer Science Department Texas Christian University User Manual FROG Revision Sign off By signing the following the team member asserts that he she has read the entire document and has to the best of his or her knowledge found the information contained herein to be accurate relevant and free of typographical error Name Signature Date Josh Alvord Alex Grosso Jose Marquez Sneha Popley Phillip Stromberg Ford Wesner Revision Sign off Page i Revision History The following is a history of revisions of this document User Manual FROG Document Version Date Edited Changes Version 1 0 03 09 10 Initial draft Version 1 1 04 15 10 Section content complete Version 2 0 04 28 10 Finalized screenshots updated log description Revision History Page ii User Manual FROG Table of Contents Revision es LOTION isrener o Debs Mu Ina pa Men MN DUM MUN ND SU M UMEN EE Mi i Revisi n NLS NEY errereen r aE E EO EEE EE EET eee ii EB ere rc 1 11 PUR PONG c o 1 12 i o M 1 L3 Definition of Dents aec eee cred E EE eerie eee 1 2 Describing the System iccgcnteelenicieolictute cen ateideeceliiee aera eda 2 24 d su oce 2 209 e E CDL RR RSS E T
9. Figure 4 5 and then a pop up prompt for an image gif format to associate with the gesture will appear Figure 4 6 This will bring up a file dialogue to select an image for association While an image is not strictly necessary it helps identify gestures at a glance System Walkthrough Page 8 User Manual FROG Mew Gesture Creation New Gesture Creation What would you like to call the new gesture circi 2 Do you wish to associate a picture with this gesture circle Cancel Yes No Cancel Figure 4 5 New Gesture Naming Figure 4 6 New Gesture Picture Association zi FROG Recognizer of Gestures Training oog Session Device Terminal New Session 3D Acceleration Vectors 2d Acceleration Values X ie Y New Gesture Sun SPOT 1BE9 Status Connected Sample rate set to 100Hz on Sun SPOT 0014 4F01 0000 1BE9 Figure 4 7 New gesture circle created and picture loaded At this point begin performing the gesture Hold down the device s gesture start button consult the device s plug in manual for more information and do not release the button until the gesture has been completed It is recommended the gesture length not exceed three seconds Do this repeatedly ideally at least ten times When all instances of this gesture are created you may click Train on the particular gesture in the Gestures
10. S 2 PASBUMEMM oso PONSE S 2 NES CULO EC 2 PAIS EMEN Taco RC 2 2 1 5 Non Mode Specific Utilities us uscscei eoe eoa sep SOME QUA detta Rim CAEN E frd e tuu PAR eU pq in EUR 3 po UBUSIBOBEIEDE o cenis e e e ea fad e aa A cM Mb RAE 3 3 Installing Starting and Stopping the System eese 4 3 Tnst lling Ses NN OUT 4 2 dtr TS Systeri oup cae desea aes mE MM S MIENNE ee es 3 33 Stopping the i NER 5 4 System WV ea RINNE iiio ch iba ea uetus aiana Sate esi ape ed deta Cava maak uke cides 6 BN bcne TP 6 ALI Connecting a SU PP ESTE 6 21 2 Enabline Filters ueber oa n e oie eae ee Pp SM EM ER MEE 8 4 1 3 Training a CHO SIU ue ee toI to epe onda a ESE EESE 8 4 1 4 Saving a SEsSsSiON sictir E meo ERE TE NEM ERE 11 415 Editing a GESTE ct 11 Z L6 L ding a 89810 cuesta ipio babe lbupii um buta bs tds Pn tmu ua E Er MIND dU RNE 12 42 coe anii edidisse c 13 S MES Cinco Mod P 15 44 Demo Mod iesene E A RE EA RE S 16 45 Non Mod Specific LOGS rse eria a E E acess R 17 AMEN E EEE E E tevesacaaveasagibons iuatngda vasoeassiasabaeva eens tanieeeanannente 17 4 5 2 Termin l and LOS S ttihgS iisiseisssuerigissisinuesisiniie iieiaeie 18 Se Troubleshootin Teesra ger DIS NOH E E A E E E RT 21 5 1 Errors Malfunctions and Emergencies ccs sccisscsc
11. Train once more for the particular gesture to modify 4 2 Recognition Mode Recognition mode allows up to four users to be simultaneously connected and perform gestures Each user must have a session loaded to have a gesture reference library Clicking Recognition from the FROG main menu will bring the user to the screen depicted in Figure 4 12 L amp FROG Recognizer of Gestures Recognition BAX Recognize Terminal User 1 User 2 New Session New Session panan Device Device Load Session Sun SPOT 1BE9 wv Status Connected Load Session x No Device Connected Status Device not found User 3 New Session User 4 New Session Device Device Load Session Load Session x No Device Connected Status Device not found x No Device Connected Status Device not found Figure 4 12 Recognition Panel Each user must connect their device Device button and load a library Load Session button to begin recognition Note If a user has used training or evaluation mode previous to this session that user s device will be automatically loaded into the first available slot and the previously created library will be used Once the libraries are loaded users can perform gestures As they are recognized the name of the gesture or Not recognized if FROG could not determine the gesture with enough certainty will be displayed under the user s name Figure 4 13 System Walkthrough
12. allows a window to be moved around the screen for easy viewing System Walkthrough Page 18 User Manual FROG The log has the following options for output All Acceleration Data Writes acceleration vectors as they are received by FROG Gesture Model Writes HMM data when the HMM is created after clicking Train Classifier Writes gesture recognized and probability when the model is classified Quantizer Writes quantizer creation parameters and the actual mean values Filters If a vector is modified by a filter the filter used and the vector data are written Devices Writes device connects and disconnects when a gesture is completed and sampling rate for the device when first connected FROG Writes top level messages such as the path FROG is running from Session Writes session commands such as Training all gestures when the session is saved loaded or re created Choose Log File Allows output to be saved to a specified file rather than an automatic one with a timestamp as the file name S FROG Recognizer of Gestures Training Joe Session Device Terminal 3D Acceleration demo W All Acceleration Data ici Gesture Model M Classifier Circle v Quantizer W Filters M Devices Tram l Edit Delete m FROG be I V Session V Log to Standard Out Ssd New Terminal Window cii shi T 7 NN Square Choose Log File Ctrl Shift N Train Edit
13. e looks and acts similar to recognition mode with a device connection and library buttons specific to each user Figure 4 15 When players are connected and gesture libraries loaded the game can be started using the Start Game button The game does require a specific set of gestures to play A set of five gestures are used and should be labeled as such Circle Square Triangle X and Z Each of these are required but are not case sensitive S FROG Recognizer of Gestures Demo BAX Demo Player 1 Player 2 alexW Load New Library alexW Load New Library 2 Sun SPOT 1BE9 Sun SPOT 4118 Status Connected Status Connected Device Device Player 3 Player 4 alex Load New Library alexW Load New Library p Sun SPOT 2AAE p Sun SPOT 1E37 Status Connected Status Connected Device Device Start Game Exit Demo Figure 4 15 Demo setup panel System Walkthrough Page 16 User Manual FROG The objective of the game is to create gestures displayed on UFOs on screen in order to prevent them from abducting cows wandering on the ground Each correctly recognized gesture with the correct UFO on screen will cause the UFO to explode and that user to gain points for the kill incorrect gestures do not penalize the user Users have until a UFO has abducted a cow and moved off screen to save a cow Scores a
14. e simply select it from the list and hit the connect button 2 2 Environment FROG requires the Java Runtime Environment version 6 0 FROG was tested extensively on a 3 2GHz machine with IGB RAM running Windows XP It was also observed to run equally well on a 1 6GHz Intel Atom based netbook running Windows XP and Windows 7 Describing the System Page 3 User Manual FROG 3 Installing Starting and Stopping the System 3 1 Installing the System Prerequisites for FROG include the following e Java Runtime Environment JRE o To download and install the latest version of JRE please visit http www java com en download manual jsp e Project self extracting executable includes SPOT plug in o Double click to extract all files to target destination e Device specific Plug ins o FROG alone cannot interface with any devices Plug ins for specific devices other than Sun SPOTs must be added o Refer to the device specific plug in manuals e FROG enabled mobile device see appropriate plug in manual o Appropriate software on mobile device Installing Starting and Stopping the System Page 4 User Manual FROG 3 2 Starting the System To start up FROG simply double click the bat file obtained from the project self extracting executable Once loaded the main FROG screen should look like Figure 3 1 e FROG Recognizer of Gestures Bax Welcome to FROG FROG Recognizer of Gestures F R O G Train Reco
15. ed A training session is a collection of training sets created by a user A session is saved in a file format and reloaded to perform recognition A training session can be thought of as a project file containing representations of multiple gestures Training Session A training set is a sequence of training instances created by the user in Training Set order to train a gesture A training set can be thought of as the complete set of data used to create an HMM representation of a gesture Introduction Page 1 User Manual FROG 2 Describing the System 2 1 Key Features FROG is an acceleration based gesture recognition system With the appropriate plug in installed the user can connect to any FROG enabled mobile device containing an accessible three axis accelerometer and perform gesture training and or recognition FROG is divided into four modes each briefly described below 2 1 1 Training This mode allows one user to connect a 3D acceleration sensing mobile device and train the system with multiple gestures The training data is captured analyzed and saved for later use in the recognition evaluation and demo modes 2 1 2 Recognition This mode allows up to four users to simultaneously connect to FROG for gesture recognition Each user must load a library of previously trained gestures in order to perform recognition One library can be used for multiple users but this is not recommended for best results When a gest
16. gnize Evaluate Demo o jj Figure 3 1 The Main Panel of FROG From this screen a mode may be selected team information can be displayed by selecting the bottom left information icon or the program can be exited through the bottom right power button icon 3 3 Stopping the System The system may be stopped at any time by closing the main window Stopping the system with unsaved gesture data will result in data loss Installing Starting and Stopping the System Page 5 User Manual FROG 4 System Walkthrough 4 1 Training Mode Before any recognition or evaluation can be done a session of gestures must be trained This is accomplished in the Training mode Training may be entered by clicking the corresponding button on the main screen of FROG Figure 3 1 Figure 4 1 shown below demonstrates a training session in progress 4 FROG Recognizer of Gestures Training cle Session Device Terminal 3D Acceleration Vectors 2d Acceleration Values New Session E x 7 EI New Gesture Y eniti cotes Sun SPOT 1BE9 j Status Connected Figure 4 1 The Training Panel 4 1 1 Connecting a Device In order to obtain acceleration data for training FROG must first connect to a device via the Device Connection dialog This is performed from the Device menu on the toolbar by selecting Configure Device The Device Connection and Setup window sho
17. ng vectors is as follows 4x y z lt Threshold If the above inequality is true the vector is discarded l Device Filters e Sun SPOT 1BE9 Fiara Euan Piters in Use Status Connected Filter Parameter Input Enter comma separated parameters for this filter 0 2 Disconnect Caliby LZ Sample Rate 100 Close Figure 4 20 Adding a Directorial Equivalence filter with a threshold of 0 2 If a device supports filtering onboard the filters it supports will appear in the filter list when it has been connected These can be added and removed just like FROG s built in filters The difference is that all processing for these filters is done by the device itself before being transmitted to FROG 4 5 2 Terminal and Log Settings FROG features an output terminal in Training Recognition and Evaluation modes Located on the toolbar the terminal settings allow console output and logging of various data including acceleration vectors filtering training and recognition results and system messages Figure 4 21 If a Log File is chosen through the appropriate dialogue the system will save all log data to a user specified text file otherwise it is saved to a system created file named with the date and time The terminal can also be displayed in a separate window using the New Terminal Window option Figure 4 22 This leaves the default terminal window running as well but
18. oing to the Session menu Figure 4 9 and selecting Save Session This will cause a save dialog to open in which a location and name for a frog file can be chosen that will contain all gesture related data f Li S FROG Recognizer of Gestures Training Jon Session Device Terminal New Session New Session fors V 2d Acceleration Values Load Session cii Save Session Cti S Exit Training Ctri ee i Y New Gesture Sun SPOT 1BE9 gt bd Status Connected Figure 4 9 The Training Panel with Session Dialogue open 4 1 5 Editing a Gesture From the individual gesture under the Gestures side of the training panel the Edit and Delete options can be selected Delete will delete the entire gesture set and remove all training instances From the Edit screen Figure 4 10 the picture associated with the gesture can be changed or individual gesture instances can be removed The list can be refreshed if it does not reflect the actual number of gesture instances trained System Walkthrough Page 11 User Manual FROG l ni tures Trainin Jae Session Device Terminal 3D Acceleration Vectors 2d Acceleration Values New Session Editing Gesture Circle Instance 1 Instance 2 Instance 3 Instance 4 Instance 5 2 Instance 6 New Gesture Instance 7 Instance 8 Instance 9
19. re tracked at the left of the screen for each user Once all cows have been successfully abducted the game will end To exit press the escape key to return to the demo setup panel UN Um gp N Figure 4 16 Demo game panel 4 5 Non Mode Specific Utilities 4 5 1 Filters FROG includes two built in filters Directorial Equivalence and Idle State Some devices support onboard filtering where the device itself filters vectors The Device Connection and Setup window handles both these types of filters Directorial Equivalence is a filter that discards acceleration vectors based on their similarity to previous vectors Directorial Equivalence takes a single parameter for its argument a threshold level The default threshold if nothing is entered is 0 2 The formula for discarding vectors is as follows System Walkthrough Page 17 User Manual FROG y G4 x2 4 y2 4 z2 lt Threshold If the above inequality is true the second vector is discarded The reference vector x1 Y1 Zi above is the last vector not eliminated by this filter The dle State filter removes vectors whose magnitude is less than a specified threshold This is to remove accelerations that come in while the device is mostly immobile and therefore do not contribute much to the overall gesture Idle State takes a single parameter for its argument a threshold level The default threshold if nothing is entered is 1 2 The formula for discardi
20. sevecsansestensaseacdennceassdaasearcuansonnsteraecateanets 21 UMEES S C ugue 22 Table of Contents Page iii User Manual FROG 1 Introduction 1 1 Purpose This document is intended to provide assistance to any user of the FROG system It provides a general introduction to FROG the installation procedure and a walkthrough of the four modes and non mode specific utilities Additionally this manual provides a troubleshooting guide and describes options for advanced users 1 2 Scope This document will be useful as a reference to first time users of FROG as well as users already familiar with motion based gesture recognition With the appropriate plug in installed the FROG framework supports input from multiple heterogeneous mobile devices Explanations of plug in specific features are described in their respective manuals 1 3 Definition of Terms A continuous combination of motions made by an individual usually with Gesture the hands that are related and meaningful as a whole Gestures are the entities which shall be modeled as well as recognized by the FROG project Double stochastic mathematical model that forms the foundation for the training and recognition algorithms used in FROG For a more in depth explanation see the FROG Software Requirements Specification Hidden Markov Model HMM A training instance is a single motion of the mobile device by a user Training Instance representing a gesture to be train
21. uld appear System Walkthrough Page 6 User Manual FROG If plug ins have been loaded correctly and the device to be used has been correctly configured please see a device s specific plug in manual for more information pressing the Discover button should result in the Devices available list being populated with device names as they are found Figure 4 2 2 Device Connection and Set up Device Filters Devices available Filters available Filters in Use Sun SPOT 1BE9 DirectorialEquivalence Discover Sun SPOT 1E98 L l IdleState Sun SPOT 4118 Connect Figure 4 2 The Device Connection and Setup Dialog Selecting a device followed by pressing Connect will cause FROG to connect to that device and begin listening for communication Figure 4 3 Any device specific filters will also appear in the Filters available list Sampling rate is given in samples per second and may be subject to limitations imposed by the particular device Calibrate will perform device specific calibration Refer to the device s plug in manual for more information tl Device Connection and Set up Device Filters gt Sun SPOT 1BE9 Filters available bd Status Connected DirectorialEquivalence IdleState Directorial Equivalence on d Idle State on device Filters in Use Directorial Equivalence on d Idle State on device
22. ure is received in this mode the system will either display the gesture that best matched or notify that it was unrecognizable 2 1 3 Evaluation This mode allows one user to connect and evaluate the recognition accuracy In this mode the system prompts the user to perform a specific gesture and tracks whether it matched the requested gesture another gesture or was not recognized The user can select what gestures to evaluate how many times each gesture should be prompted and if the gestures should be presented in a random order Evaluation results are presented on screen 2 1 4 Demonstration In this mode up to four users can connect to play a competitive game using pre trained gesture libraries Each library should be user specific for the best recognition accuracy The object of the game is to demonstrate the practical applications of the gesture recognition technology employed by FROG Describing the System Page 2 User Manual FROG 2 1 5 Non Mode Specific Utilities FROG Console This console can be set to display and log any input output of the system Log and display settings can be configured from the Terminal Controls section of the interface window Device Panel This pop up panel is used to locate and connect available devices to the system as well as to enable filters The discover button will fill a list of available devices displaying their device type i e SPOT Android Windows Mobile HTC Fuze To connect a devic
Download Pdf Manuals
Related Search