Eye / Gaze tracking system Client Software User Manual
Contents
1. Size 13 7 MB yw xlabs client_1 0 0_win64 exe Date modified 2 07 2014 xS Application 07 014 6 7 DRA Open using Resource Hacker 7 Annlicatinn 14 037 KB Run as administrator N Troubleshoot compatibility Edit with Notepad Git Init Here Git Bash Pin to Taskbar Pin to Start Menu Restore previous versions Send to b Cut Copy Paste Create shortcut Delete Rename Figure 15 Right click the downloaded file and select Run as administrator to start the install on Windows SE yLabs Client installer ET This program will install the xLabs Client package for eye gaze tracking Please specify the install location C Program Files x86 xlabs client Cora Figure 16 Selecting the install location in Microsoft Windows 20 P Windows Security Alert Windows Firewall has blocked some features of Java TM Platform SE binary on all public and private networks Name ES Publisher Path Allow Java TM Platform SE binary to communicate on these networks Private networks such as my home or work network Public networks such as those in airports and coffee shops not recommended because these networks often have litte or no security What are the risks of allowing a program through a firewall Java TM Platform SE binary C program files x86 pdabs cient jre bin ava exe Windows Firewall has blocked some features of this program e Canc2. Browser Extension Click the Browser Extension link at the top of the registration page Then when the page opens click the Download link The browser extension will be installed and the last warning message will disappear The system is now ready to use 2 6 Calibrate amp Use system To calibrate the system first enable the xLabs graphical interface by clicking the x icon in the browser extensions toolbar A popup menu will show Click Enable graphics in this menu xLabs Eye Gaze Tracking xlabs com au Chromium j xLabs Eye Gaze Trackin x Extensions x t gt H Q xlabs 9522 extension htr x4 7 i labs lt labs Eye Gaze tracking system IAN About License Registration Browser extension Privaq User unknown Browser Extension You need to download an enhancement to your web browser to enable the eye gaze tra Download link Enable tracking Plugin Menu After installing the browser extension click the x icon to open the plugin menu Settings XX Fh W labs User bob m Figure 6 Click the x icon of the xLabs Browser Extension to show the popup menu Click Enable Graphics to activate the in browser interface When graphics are enabled the toolbar will be displayed This is the main interface for using the system xLabs Eye Gaze Tracking test page xlabs com au Chromium xLabs Eye Gaze Trackin x Extensions s 9 Q xlabs 9522 test htm
3. psychology experiments and online training e Detect patterns of gaze that indicate enhanced interest confusion or frustration and make the web page respond appropriately e Detect when items are not noticed and promote them alternatively hide things that are repeatedly ignored e Create stunning dynamic interactive interfaces that respond visually to both looks movement and mouse actions e Capture sequences of events and interactions between site content what s read what the user does next and build better more intuitive interfaces Infer emotional states and reactions Improve safety by checking alertness Infer distraction and exhaustion Know when your content should pause or shut up or simplify itself to ease the burden on the user Make the site fit into the user s world when something takes them away from the screen stop what s happening and wait for them to return 1 3 Our mission Everything beyond the screen Our mission is to capture everything that s happening in the user s world enabling content providers to build more human interfaces 90 of human communication is nonverbal Researchers believe that the vast majority of natural human communication is accomplished by gestures gaze and actions rather than by words One of the frustrating and alien things about computers is that they re totally insensitive to most of the expressions produced by people Fixing this will make computer interface
4. the EULA Developers are encouraged to use the EULA to evaluate the system and create new products and services that use on the system A full set of licences terms and conditions can be viewed at http www xlabsgaze com licence Please contact us to discuss specific licencing arrangements The system is closed source with the exception of the SDK Javascript API US patent application No 62 006 428 covers the algorithms and methods used in the system All copyrights are reserved by xLabs Pty Ltd Australia 18 4 2 Hardware Requirements For best results your computer must match or exceed the following requirements Processor Intel Core 2 Duo 2GHz or better Intel 15 or 17 recommended Approximately 15MB required for our software 200MB required for installers including bundled Java Runtime Environment JRE Camera Resolution 640 x 480 or better Compatible with OpenCV camera drivers 4 3 Operating System The following operating systems are currently supported Operating System Installation Notes Windows 7 and 8 64 bit A self extracting exe is provided Apple Mac OSX 10 5 and better A pkg installer is provided Linux 64 bit A Debian install package Is provided On all systems Administrator privileges are required to install the software because we adda service or daemon that runs on startup 4 4 Software dependencies We require a Java Runtime Environment JRE version 1 5 or better to run our software
5. the old user s model 7 8 Fast movements Very rapid movements cause blurred images in the camera When this happens the system will briefly lose track of the user s face and then find it again a fraction of a second later For best results try not to move too quickly especially during calibration clicks Blurring is also more common in low light because cameras will choose longer exposures 7 9 Wrong user Calibration is dependent on learning the shape and appearance of each user s face If the user changes the calibration won t work Make sure the user in the plugin menu shows the correct user name In future we will automate this with face recognition 7 10 Camera has moved During calibration the system learns the relationship between the positions of the camera and the screen So if this relationship changes the calibration will be wrong If the camera is built into the screen this is not a problem you will move the screen and camera together Laptops can be moved around no problem However if you have a webcam placed on top of your screen you will need to clear existing calibration data and re calibrate if you move the camera to a different position relative to the screen 7 11 Too many faces The system currently only works with a single user at any one time The system tracks the largest face in view and assumes this is the current user It might get confused if more than one person is in view of the camera D
6. to enable calibration Check that none of the red warning icons are showing at times when the system fails to work If no warnings are visible try deleting the calibration data and starting again You can delete the calibration data by pressing d in Calibration graphics mode If you continue to have problems try adjusting your seating or screen position It is possible that we are not getting a good view of your face The Feedback graphics mode can help you check the positioning of camera screen and face 7 7 Calibrated Pose Space The system learns to associate particular poses and positions of the head and face with gaze coordinates on screen It generalizes beyond the observed training data to try to fill in the gaps 35 to be able to predict gaze for poses it has not seen before With more training the system gets better at generalization Imagine the set of poses the system has seen as a bubble or cloud around your head The more poses you have calibrated the larger this cloud becomes Eventually all poses you find comfortable will be within the calibrated cloud Changing the angle of a laptop screen can move your head into an uncalibrated position This is easily fixed by clicking a few marks in the new position which adds it to the calibrated cloud Calibration does not transfer between users You need to recalibrate for every user When changing users you need to enable calibration to clear
7. to remedy these errors More details can be found in the SDK manual 28 6 5 Graphics Modes The following images show some of the graphics effects we have implemented to showcase the capabilities of the eye gaze tracking system Most of the effects present live realtime gaze data in different ways The final one Heatmap accumulates gaze data and constructs a heatmap of gaze intensity across the screen over time The live realtime modes can be accessed by using the lt and gt buttons to change the graphics mode to spotlight and then cycling through the different effects using the p key Use gt to change from Spotlight to Sequence graphics mode and then again from Sequence to Heatmap While in Heatmap graphics mode the r key will reset the heatmap Gaze data is then accumulated until r is pressed again File Mars Viking 12f203 png Wikimedia Commons Chromium ia org wiki File Mars Viking_12f203 png Spotlight Ome ORAWY m Figure 22 Spotlight graphics mode default effect Spotlight TOM OFEN m Figure 23 Spotlight graphics mode hotspot effect 29 File Mars Viking 12f203 png Wikimedia Commons Chromium J File mars Viking 12f203 x t gt A commons wikimedia org wiki File Mars_V Spotlight m EWN E i a v File Mars Viking 12f203 png Wikimedia Commons Chromium J amp File Mars Viking 12f2
8. 03 x WIKIMEDIA COMMONS Main page Welcome Community portal Village pump Help portal Language select English __ a Select Participate Upload file Recent changes Latest files Random file Contact us Tools What links here Related changes Special pages Printable version Permanent link Page information Nominate for deletion 29 commons wikimedia org wiki File Mars_Viking_12f203 Spotlight o BMOxe AD File Discussion View Edit History Q File Mars Viking 12f203 png From Wikimedia Commons the free media repository File File history File usage on Commons File usage on other wikis Download all sizes Use this file on the web Use this file on a wiki Email a link to this file Information about reusing E 2 W 4 6 gt No higher resolution available Mars_Viking_12f203 png 583 x 512 pixels file size 493 KB MIME type image png Figure 25 Spotlight graphics mode laser effect File Mars Viking 12f203 png Wikimedia Commons Chromium J amp File Mars Viking 12f203 x 7 9 3 commons wikimedia org wiki File Mars_Viking_12f203 png WIKIMEDIA COMMONS Main page Welcome Community portal Village pump Help portal Language select English xj Select Participate Upload file Recent changes Latest files Random file Contact us Tools What links here Related changes Special pages Printable version Permanent link Pag
9. 1 Warning icons and their meanings 6 4 1 No calibration This icon means that the system is currently not calibrated for the current user If you have already calibrated the current user then enable calibration to fix this If you have not calibrated this user head over to the calibration graphics mode and do it This icon can also mean that there is a problem with the calibration data for this user 6 4 2 Uncalibrated pose or position Recall that our system aims to provide a calibration solution for any pose position of the user To do this it has to generalize and fill in the gaps between the poses it has already observed 27 This icon indicates that the user s current pose is not within the range of positions the system has already encountered and that the system thinks it is unlikely the current calibration is going to work Fix this error by adding an extra 4 or 8 calibration clicks in the current pose 6 4 3 Image too dark This icon goes red when the image is too dark Move somewhere brighter This can also indicate a camera fault or setup problem 6 4 4 Image too bright This icon turns on when the image is saturated or over bright When this happens part or all of the face is impossible to see clearly Move somewhere with more even lighting and consider some of our other environment setup tips 6 4 5 No face found This means that the system has not detected a face in the current video feed from the camera This is perfec
10. Also it may take longer to get clicks from all over the screen and the rate of clicks is less controlled The system will tell you when it thinks its ready 17 4 How to get it The eye gaze tracking system can be downloaded from http www xlabsgaze com download To download the software specify the target operating system and enter a valid email address A temporary download link will be emailed to the address specified Open the link in your browser to download the software You will receive a self installing package that is fully capable of eye gaze tracking and the web based interfaces to control it We call this the client software 4 1 Licencing Licencing is designed to encourage widespread adoption In particular we provide an End User Licence Agreement EULA at no cost and charge for Commercial Licences Without going into legalese our intentions are The system is free for end users to install and use under EULA Anyone can write develop software that uses the system under EULA If you develop software that uses or interacts with our software you cannot distribute or make available that software unless you obtain a separate commercial or developer licence Commercial use including providing a service that profits from or exploits gaze data requires the service or software provider to obtain a separate licence End users do not need to purchase licences from xLabs but instead can use our system under
11. Document Version 1 4 1 19 June 2014 xLabs Pty Ltd Australia http www xlabsgaze com http www xlabs com au talk xlabsgaze com Eye Gaze tracking system Client Software User Manual labs 1 Introduction The xLabs Eye Gaze tracking system is able to continuously calculate where you are looking on a computer screen while allowing natural freedom of movement It does not require any special hardware just any ordinary webcam This gaze data is calculated locally and in realtime i e it is available as soon as it happens To the best of our Knowledge there is no other software only system with these capabilities The system is free to download and use for non commercial purposes The interface is integrated into web browsers Such as Chrome Safari and Firefox allowing eye gaze to be tracked as part of any website or web application We encourage users and developers to build useful tools and programs on top of the system using our Javascript interface Figure 1 Spotlight on screen tracking user s gaze 1 1 Features The eye gaze tracking system has many features in addition to working out what you re looking at Many are inherent to capturing head and face pose in 4D All these features are now available to Javascript developers and can be added to website content and functionality e Streaming video We can stream the user s webcam to the browser making the feed available in Javascript You can put the video fee
12. LLER_ FILE Where INSTALLER_FILE is the file that you downloaded You will be prompted to enter your password to escalate privileges The system must then be rebooted to refresh environment variables After reboot open your browser and enter this address http xlabs 9522 You will then be prompted to download and install the browser plugin to complete the installation Note that other non debian Linux variants will probably work though some effort may be required to complete the install manually Details are provided in the SDK manual 22 5 Privacy amp Security User privacy and security really are important to us As such we have published a separate and comprehensive user guide on this topic The guide is available from our website http www xlabsgaze com privacy Our primary user privacy principle is informed consent This means that the end user should always be aware of when and what data is captured where it is stored and consent to use of the data by third party software In Summary we ensure a balance between our business and your privacy and consent by the following mechanisms 5 1 Camera light We are not able to activate the camera without also activating the activity light that is featured on most cameras If the camera light is off the camera is off 5 2 On Screen icon We also put a notification icon in the toolbar of the web based user interface to note when the camera is on W becomes w
13. Toolbar Figure 7 Browser extension toolbar after Enable Graphics Click the lt and gt arrows to change display mode Click the gt symbol to change the graphics mode to Feedback Click the W for Webcam icon to turn the camera on You should get a view like this j X xLabs Eye Gaze Trackin x t gt H D file home dave workspace xlabs www persistent test html XK x Feedback WT Vv O 4 120 OFCN m Figure 8 Webcam enabled in feedback mode Face pose is shown on screen Check that there are no red warning icons and that the white face ellipse follows you around the screen This means the system is successfully detecting and tracking your face and providing this data to the web browser Click gt again to change to calibration mode You will see a grid of marks Move the mouse to one of the marks then PRESS and HOLD until the countdown reaches 0 and the yellow circle goes green Then release the mouse You MUST look at the circle throughout the countdown otherwise it won t calibrate properly 10 xLabs Eye Gaze Tracking test page xlabs com au Chromium BO 40 Omen oa e ET OR NE Figure 9 Calibrating on a mark You must press and hold down the left mouse button while looking at the circle until the countdown finishes and the circle becomes completely green We recommend you press and hold on at least 4 marks initially to give a ro
14. We bundle a suitable JRE with out Microsoft Windows installer On Apple OSX and Linux we require Java to be pre installed On Apple OSX it is necessary to install the Java Development Kit JDK to enable the Java command line executable You can test that a suitable Java is installed on your machine by running java version on the command line Our software will work as long as java is a recognized executable on the system path 4 5 Web Browsers Currently only Google Chrome versions 30 and higher are supported Compatibility with Safari and Firefox will be added in the near future 19 4 6 Installation Installation instructions are specific to each operating system 4 6 1 Microsoft Windows Right click the self extracting exe and select Run as Administrator Follow the instructions on screen to install the software The final step should open your web browser and show our local web pages You will then be prompted to download and install the browser plugin to complete the installation GQ gt Computer Local Disk C workspace xlabs_installer v 4 Search xlat Date modified Type Size Organize v e Open New folder wr Favorites Name E Desktop X xlabs client_1 0 0_win64 exe mi Downloads Recent Places a Libraries Documents a Music Pictures F Videos JM Computer amp Local Disk C lt 3 CD Drive D GRMSI amp fullfiles wboxsry S temp vboxsry F hi Network
15. alibration data by pressing d in Calibration graphics mode If you continue to have problems try adjusting your seating or screen position It is possible that we are not getting a good view of your face The Feedback graphics mode can help you check the positioning of camera screen and face 7 4 Gaze updates slowly Ensure that you are using a reasonably modern and fast computer Check that red warning icons are not showing Check that there aren t too many browser windows or tabs open It may also help to reset the camera by turning off the W Webcam and restarting it Some cameras adapt to low light levels by taking a long time to grab frames This can cause slow updates to the gaze system 7 5 Gaze tracking doesn t work Ensure that you have registered the system by entering your email address on the Registration page http xlabs 9522 registration html Check that no red warning icons are showing Check that the calibration warning Is not showing The W Webcam and T Tracking buttons must be enabled for gaze tracking to work If you are unable to achieve successful results please contact us for more assistance 7 6 Calibration doesn t work For calibration to work the Webcam and Tracking features must be on You need to click some marks in calibration mode at least 4 or 8 for a rough calibration Finally you need to press c in calibration mode or click the C button
16. ble gaze tracking you will be asked once whether you agree to allow the page to enable the system If you disagree the page will be denied all functionality If you agree the system will send a message to us that only contains the URL of the page that requested gaze tracking Your IP address is not recorded This enables us to track which websites use our system If you manually enable gaze tracking the system does not send a message to us 24 6 Web User Interface The main user interface for the software is provided within your web browser via a browser extension In addition we provide a set of local web pages that can be accessed through the browser This section will these interfaces in detail 6 1 Local Web Pages The local web pages can be accessed via a special URL whether you are online or not http xlabs 9522 Some of these pages are critical for completing installation of the system You can return to them at any time The licence page shows details of the End User License Agreement EULA The Browser Extension page helps you to download the rest of the user interface if it is not already installed You need to visit the registration page and enter your email to activate the system This allows us to track usage of the system The email address is only used to report usage as described in the Privacy section of this document 6 2 Browser Extension Plugin or Add On The browser extension provides a quick and conven
17. d anywhere in the page and draw on it Video is provided as streaming MJPEG and will easily render in most browsers Face detection We signal whether a face is present at the user s computer Eye Gaze tracking Of course we provide the focus of user s gaze on screen Face pose We provide the pose of the face so you can respond to movements Face recognition We can recognise the user of the computer by appearance allowing you to individualise their experience of your web content coming Sept 2014 e Head gestures You can respond to user head movements as deliberate gestures providing another control input e Eye pose In addition to where the user is looking on the screen we can provide details on raw eye movements which can reveal mental states e g tiredness e Blink detection Eyes closed either deliberately or due to blinking can be a useful user input or relevant to psychology experiments coming Sept 2014 e Facial expressions We continuously model the shape and pose of the face allowing us to detect changes in expression mouth movements etc coming Sept 2014 1 2 Benefits Vision is the superhighway into the human brain The way in which we respond to appearance reveals much about conscious and subconscious mental state This information is of tremendous value to both users and website or content providers Many applications can be built on this data e Track focus of attention for User eXperience UX studies
18. ds to be able to see your face and eyes clearly Bright lights close to your face can actually make things worse if they cause dark shadows in some areas Diffuse well lit soaces are best If working during the day it is better for the user to face the window so daylight falls on the face Working outdoors is not a problem as long as the background isn t much brighter than the face The troubleshooting section at the end of this document has some tips on setting up a good environment 3 3 User Appearance Currently the system does not work with glasses although if your glasses have thin light coloured frames it might be OK Future versions will work with glasses once some extra R amp D is complete Heavy makeup particularly thick black eyeliner can also disrupt the system Given how creative people are with makeup we can t promise to ever be able to deal with all makeup But this is something we will be improving over time If your hair falls over your eyes this can also cause difficulties Please temporarily tie it back so we can see your face clearly 3 4 Camera setup The computer must be connected to a typical webcam with resolution 640x480 pixels or better The system uses the OpenCV libraries to talk to the camera most ordinary cameras are compatible with these libraries 14 The camera must be positioned so that a good view of the user s face is provided If the camera is moved relative to the screen the system must b
19. e information Nominate for deletion Sequence ORC E File Discussion View Edit History Q File Mars Viking 12f203 png From Wikimedia Commons the free media repository File File history File usage on Commons File usage on other wikis Download all sizes wy Use this file on the web W Use this file on a wiki Email a link to this file f Information about reusing E No higher resolution available Mars_Viking_12f203 png 583 x 512 pixels file size 493 KB MIME type image png Figure 26 Sequence graphics mode A chain of dots shows the most recent gaze foci 30 BO AG Figure 27 Heatmap graphics mode Gaze foci accumulate over time to build a picture of the most viewed areas on the screen 6 6 Keyboard Commands The following keyboard commands can be used in specific graphics modes Note that when keyboard capture is not enabled the commands will not work raphieswode Wey eet ae J eserin useful if there is an error of if you have captured some bad data Changes graphics mode to Spotlight camain a orerar ons meae ro seguens Sron P crue way ne svoan s presoren 31 7 Troubleshooting Make no mistake the eye gaze tracking system is very complex It tries to work with everyone regardless of appearance using any camera on any computer anywhere in the world It s not simply running a program again and again It s learning and calibrating i
20. e recalibrated If the camera and screen are moved together re calibration is not required Good positions for the camera include around the edges of the screen Many laptops include a built in camera These are ideal The camera doesn t need to be exactly in the middle of the screen Fl Life amp style Travel Environment Tech TV Video Datings edi Education Society Get Women Law Data Foot Figure 13 Typical camera placement Left An external camera Right Camera built into laptop screen 3 5 Browser integration The system is configured and controlled via our web user interfaces As mentioned above these have two parts a browser extension which provides an interface on top of the current web page and a set of local web pages You can access the local web pages by entering this address into your browser http xlabs 9522 You can also control the system through our browser plugin Click the xLabs x icon in the toolbar to open the quick access menu Click Enable graphics to show our main interface on top of the current web page The Settings button takes you to the local web user interface The plugin menu is shown in figure 12 An explanation of the browser extension interface is provided in section 6 3 6 Calibration Calibration is the process of teaching the system to match the appearance of faces in camera images to coordinates on screen It is important to understand that best results take quite a lot
21. el Figure 17 On Microsoft Windows you may be prompted to allow our web services access to 4 6 2 Apple Mac OSX networking We use Java to run our web services Double click the pkg file to open the installer Follow the instructions on screen until the install has completed The final step should open your web browser and show our local web pages You will then be prompted to download and install the browser plugin to complete the installation AN Applications a Desktop i Documents E Downloads fet xlabs DEVICES E osx dev SHARED xlabs amp Lal David s M DJ s Pro n amp MAP s Mac L4 Razorblade TAGS Red O Orange Figure 18 Open the downloaded pkg file in Finder on OSX Applications B Desktop 9 Documents Downloads Movies i Music Pictures Public C temp Name Kind Size Created Modified Last opened xlabs client 1 0 0 pkg Installer package 79 5 MB 26 Jun 2014 6 56 am 26 Jun 2014 6 57 am 26 Jun 2014 6 57 am 21 AOO Install xlabs client 1 0 0 Welcome to the xlabs client 1 0 0 Installer Introduction Destination Seled You will be guided through the steps necessary to install this software Installationaype Installatigh Hy l wji l uy l Lay 5 GoBack Continue Figure 19 Installer dialog interface on Apple OSX 4 6 3 Linux Debian To install the deb package open a terminal window and type sudo dpkg i INSTA
22. era After clicking wait a second or two for the camera light to turn on Once the camera is active extra features such as image quality checks and calibration are available 6 3 6 Toggle enable tracking T This button turns on eye gaze tracking If the camera is not already on it will be turned on Any existing calibration will be loaded if available If the Webcam is enabled without tracking then you can get video and validation of the images but you won t be able to do gaze tracking Turning tracking off when not in use will Save power 26 6 3 7 Calibrate C Clicking this button will request calibration Note that this will only work when calibration data has been captured and the Webcam and Tracking are enabled The Calibrate button will disable itself once calibration has finished typically 1 second 6 3 8 Toggle enable Validation V When enabled this feature analyses the camera image to look for a good trackable face It will report errors by lighting red warning icons described below 6 4 Warning Icons Figure 21 shows the warning icons used in the system In general grey icons mean that everything is OK Red icons mean that a problem has been detected Occasional warnings are nothing to worry about but consistently or frequently red icons may affect system performance Not Image No face Excess Uneven calibrated too dark found yaw lighting Uncalibrated Validation Camera position enabled error Figure 2
23. ery dark in comparison Figure 30 Environmental problems Avoid spotlights or desklamps that only illuminate part of the scene and leave dark shadows elsewhere We need to see the whole face clearly not just one side It s better to have distant diffuse or back lighting 33 Figure 31 Environmental tips During the day sit facing towards bright lights such as windows This ensures that the face rather than the background is well lit d Figure 32 Environmental tips At night or indoors find a well lit room with bright overhead lighting There should be few or no distinct shadows on the face Bouncing light off walls is better than direct illumination 7 2 Poor accuracy in one part of the screen If gaze tracking is not accurate in one area of the screen try adding one or two new calibration clicks in that area Also check that no red warning icons are visible If you see warnings fix them before continuing If you continue to get warnings when looking at a particular area of the screen try adjusting your seating or screen position It is possible that we are not getting a good view of your face The Feedback graphics mode can help you check the positioning of camera screen and face 34 7 3 System works intermittently Check that none of the red warning icons are showing at times when the system fails to work If no warnings are visible try deleting the calibration data and starting again You can delete the c
24. he subjects to come to the hardware rather than moving hardware to the people Are peoples attitudes to advertising affected by their environment Of course they are Bigger data solves this problem By getting larger samples at the same or lower cost and by allowing collection on users own computers at home the data becomes more representative and robust to individual variations from day to day 1 5 Creating the Gaze Aware Web This is the first and only software eye gaze tracking system that allows natural movement and continuous long term use Once thoroughly calibrated the system will work automatically as soon as you sit down at the computer Maybe one day this technology will allow your computer to infer your state of mind This is not simply a computer program It is a new interaction tool that works in conjunction with the mouse keyboard and other input devices to enhance your experience and make the computer more intelligent more human We believe that interaction with the computer should be as natural as talking to a friend But for this to be possible the computer needs to understand all the nonverbal parts of human communication as well as voice keyboard and mouse input When enough software is available to make use of the eye gaze tracking system the computer will be able to understand what you re looking at and react to it seamlessly The computer will understand your attention and your mental context It w
25. hite 5 3 OnlOff The end user can turn the system on and off at any time 5 4 Consent to turn On If a web page requests that we enable gaze tracking the user must consent before it is enabled The web page is only allowed to ask you once per site If you change your mind you must refresh the page to be asked again 5 5 Documented file content We document the format and location of all data files to enable the end user to inspect and review data captured about them This information is contained within the SDK manual 5 6 No remote video stream The built in video streaming only sends video to the user s own computer Video cannot be streamed to other computers This is a feature hard coded into the program and cannot be altered So no one else can see your camera images 5 7 No remote video recording For developers we provide a method of saving video from the camera to disk There is no Capability to send these video files anywhere else If you want to send them you have to do it manually 23 5 8 Email during registration In order to make the system free to use and prevent unauthorised commercial use we require a working email address to download the software 5 9 System Usage information When activated the system sends a single message to us to say on what date it was used This message only contains the email address used to register the system and a timestamp When you visit a web page and that page requests you to ena
26. ient way to access the features for configuring and controlling the eye gaze tracking system The extension overlays a toolbar and some graphics on whatever web page you re currently visiting x Hide toolbar W Enable Webcam GC Enable Calibration Current graphics mode K Toggle capture Keyboard Toolbar M E Current T E Toggle T Enable ae username grap Tool Tip capture Tracking aie mode validation Mouse Figure 20 xLabs Browser Extension toolbar Tool tips appear when you hover the mouse over any clickable control 6 3 Control features See figure 20 above to see the icons associated with these controls In general controls that are filled white are enabled controls with are disabled All errors and warnings are red If a control is requested to be enabled but does not operate it will have a OO f a control is requested to be disabled but is enabled it will have a yellow 6 3 1 Username 25 The gaze data and calibration setup is associated with a specific username Note that this is not related to the email address used to register the system We allow multiple individuals to use the same system with their own calibration settings and gaze output To change the username type a new name into the box in the popup menu and press lt ENTER gt 6 3 2 Toggle hide toolbar X When visible the toolbar can be hidden by clic
27. ill know what you were doing It will Know when you re distracted and it should shut up or pause a video or some other context dependent smartness We know that to realise this vision we need to support developers to build software on top of eye gaze tracking We also need users to install and play with the system It has to provide immediate benefits for everyone which is why we provide it freely to end users and encourage developers to play and create useful applications with it 1 6 Table of Contents secon rows de 2 Quick Start This section will try to jump you through all the hoops so you can have a play with the system and come back later to read about how it actually works If you get stuck you might have to read the rest of the manual Here s what you need to do 1 Register to receive a download link 2 Click the link and download the Client software 3 Install amp configure the Client software 4 Download the Browser Extension 5 Calibrate amp Use 2 1 Download Page Open your browser and go to http www xlabsgaze com download Scroll down to the registration form and enter a working email address Select your computer s operating system then click Register Download form Operating system Windows 7 or 8 64 bit Windows 7 or 8 64 bit Apple OSX Linux 64 bit debian Figure 2 Form to register and receive a link to download the client software 2 2 Download Client software You
28. king the x xLabs logo next to the username To bring it back use the popup menu Enable Graphics toggle The configuration of the toolbar and system is remembered while it is hidden 6 3 3 Change graphics mode The lt and gt buttons allow the graphics mode to be changed Graphics modes showcase different presentations of gaze data and allow configuration of the system calibration and feedback on the camera image The available graphics modes are in order Toolbar only shows the toolbar nothing else Feedback shows the video from the camera and gives feedback on quality Calibration allows you to calibrate the system Spotlight presents live gaze data as a spotlight effect Sequence presents live gaze data as a sequence of red dots Heatmap builds a heatmap over time showing cumulative gaze data 6 3 4 Toggle capture Keyboard K This button determines whether keyboard events are captured by the xLabs system or are allowed to pass through to the underlying web page This allows you to navigate the web normally while using the system 6 3 4 Toggle capture Mouse M This button determines whether mouse events are captured by the xLabs system or are allowed to pass through to the underlying web page This allows you to navigate the web normally while using the system 6 3 5 Toggle enable Webcam W This button activates the program that connects to the camera and tries to get video from the cam
29. n how to use the interface see section 6 In this mode we show a grid of marks We ask that you move the mouse to the mark and then while looking at the press and hold the left mouse button down A yellow dot and a countdown timer will appear The dot will turn yellow and then green When it becomes green you can let go of the mouse button and look away You should repeat this about 8 times for a quick calibration It doesn t matter if you blink You don t have to remain completely still but very fast movements might blur the image Just be relaxed and natural and it will work We encourage whatever head movement is comfortable to look around the screen The calibration needs to be trained on a range of screen positions To make it easy to pick a random mark to click we randomly place a red dot on one mark Feel free to click or ignore the red dot 16 xLabs Eye Gaze Tracking test page xlabs com au Chromium BO 40 Omen oa E AE f IR NEJE Figure 14 The calibration interface Press the mouse button on a mark and keep looking at the countdown timer until it finishes Then release the mouse 3 8 Passive calibration coming Sept 2014 If you prefer the system can be calibrated by watching you surf the web Basically we turn on the camera and watch you When you click on links we assume you re looking at the mouse This is usually true but not always so it may take longer to calibrate this way
30. nge proportions or colours the camera settings need to be changed In future versions we will provide an interface to allow users to modify camera settings However until that time camera settings must be modified manually via settings files Details are provided in the SDK manual recommended for expert users only 7 15 Browser Extension button is missing If you are unable to see the browser extension button in the browser toolbar it may need to be removed and reinstalled Currently only the Google Chrome browser is supported other browsers late 2014 37 7 16 Can t see the web user interface The local web pages interface should be accessible using the URL http xlabs 9522 If the pages are not accessible the system may need to be reinstalled However xlabs is simply an alternative way of addressing your local computer Try to access the web pages using http localhost 9522 http 127 0 0 1 9522 If these alternative URLs work then the HOSTS file does not correctly redirect xlabs to localhost This can be fixed by adding the appropriate entry If the local web pages are not accessible using any of the URLs above the local web service may not be running The web service is scheduled to run on startup in all operating systems In Microsoft Windows the web service is run as a system service named xLabsWebServices In Apple OSX and Linux the web service runs as a system daemon Reinstalling may fix this iss
31. of calibration we need to see how you look in several different positions with different combinations of head and eye movements Calibration is achieved via our browser extension web interface see section 6 or figure 9 15 If you keep quite still the system can be calibrated in 4 clicks But it will only be calibrated for that one head position As you do more clicking in different poses the system starts to learn to generalize and expand the calibrated space Eventually all the positions you regularly use will be calibrated and you won t need to do it again unless you change your seating or desk height Normally once well calibrated you can leave your seat and return without needing to do more calibration Calibration makes the assumption that you are looking where you are clicking If you don t cooperate during calibration you might need to clear and reset the calibration data You can come back to calibration at any time via our web interface You might need to do quick calibrations often at first but eventually you will not need to If you change your seating position an additional 4 calibration clicks are usually enough to add the new position to the range of poses that are calibrated 3 7 How to calibrate Figure 9 shows the calibration interface You can get to this interface while on any web page by clicking the plugin to show the toolbar and then changing the display mode to Calibration For more details o
32. on t do that In future we will add face recognition tools to make this less restrictive 7 12 Camera doesn t turn on If the camera doesn t turn on or turns on briefly and then off again there is a problem with the configuration of the camera program 36 Figure 33 shows the possible states of the Enable Webcam W button in the toolbar You should normally only see the Black White and blue text states The other states indicate a problem accessing the camera Try rebooting the computer and check the camera is plugged in correctly Failing that you may need to reinstall our client software Figure 33 The webcam has 4 possible states which can be distinguished by the colours of the button From left to right Mae 2 Webcam enabled but not working 3 Webcam enabled and working ZAMAJE mee Tete Mees KONA Please contact us for individual support if you observe that the webcam is unable to turn on when requested and restarting the computer doesn t help 7 13 Camera doesn t turn off See notes above regarding problems activating the camera and consult figure 33 If the camera refuses to turn off try restarting the computer The camera will not turn on by default on boot If the problem persists it is likely an incompatibility with the OpenCV camera drivers 7 14 Camera picture is bad You can check the camera picture using the Feedback graphics mode of the browser extension If the camera picture has stra
33. s far more intuitive responsive and natural more human Technically we can hope to capture the nonverbal parts by watching the user through the low quality webcams that are now so widely available And with a little help from some clever artificial intelligence This gives us the architecture of the xLabs Eye Gaze tracking system Eventually we will add speech recognition to the platform probably as a separate module In future you will interact with your computer as if with a person who really gets you Unlike many companies starting to offer web based gaze tracking we see the user as the primary beneficiary rather than a data source to be mined 1 4 Why scale matters Gaze tracking hardware is well established in niche applications using a standard hardware kit Infra Red IR cameras and Illuminators typically LEDs But although these systems are very precise reliance on this kit limits the accuracy and quality of results because it restricts sample size People simply don t behave the same way every time It s crazy to put too much emphasis on individual results even when they are very carefully collected Focusing on the accuracy of measurement collection ignores the elephant in the room that the measurements themselves are inherently quite random The use of gaze tracking hardware also threatens to invalidate results simply by moving people into alien artificial laboratories Due to cost gaze tracking studies ask t
34. see figure 11 The first part is a software program that must be installed The second part is a browser extension that provides the user interface Due to current security systems within web browsers we need to install a program onto the end user s computer to communicate with the camera on demand We provide the software as a single installation package We call this the xLabs Client software You need to install both the client software and the browser extension to use the system xLabs Client software Web Browser Web pages pE EE EE EE E ME ee ee ee ee ee M Web Service Daemon Browser Extension http xlabs 9523 http xlabs 952 Figure 11 The eye gaze tracking system is installed as two parts A client software package and a browser extension that provides the user interface within the web browser Once installed the interface to the system is inside your web browser To access it type into the address bar http xlabs 9522 When you have installed our browser extension you can also reach the interface by clicking the xLabs icon in your browser s toolbar 13 labs User bob Enable graphics Enable tracking Calibrate Settings Figure 12 xLabs browser extension Clicking the x icon opens the menu Click Enable Graphics to show the interface on top of the current web page 3 2 User Enivronment The system works best in a well lit environment It nee
35. t cross platform builds We used Jetty to provide a lightweight local web service Thanks to everyone who contributes to open source software Jetty lightweight Java webserver hitowwweclipseorgietyh g jetty BOOST C libraries http www boost org l STASM library face fitting using Active Shape http www milbo users sonic net stasm Models libSVM library Support Vector Machines http www csie ntu edu tw cjlin libsvm lib CURL library HTTP requests in C C http curl haxx se libcurl 7 Zip was used to create self extracting http www 7 Zip org installers on Microsoft Windows Many other open source projects contributed to this effort in less dramatic ways for example being used on our personal and home computers 39
36. ted 2 Get Browser 3 Calibrate amp Extension Use 1 Register The next step is to register the system and then to download our web browser extension aka add on or plug in Once the extension is available you will be able to control the system from within your web browser This address xLabs 9522 can always be accessed from your computer to configure the Eye Gaze tracking system Figure 4 Landing page displayed in browser after running the installer After clicking Registration you will see the registration page xLabs Eye Gaze Tracking xlabs com au Chromium o xLabs Eye Gaze Trackin x t gt 9 D xlabs 9522 registration html labs Eye Gaze tracking system About License Registration Browser extension Privacy Help topics Registration We ask that you register this software so that we can track the number of downloads and users This helps us to let this software freely downloaded and used for non commercial purposes We won t sign you up for cat facts or other spam or do anything nasty with your email address You can change the registered email address at any time Once registered you are free to use the system under the terms of our end user licence agreement EULA Update Email Registered email address some body company com Figure 5 Registration page displayed in browser Enter your email and click Register One of the warning messages will disappear 2 5 Download the
37. tly OK if you re not there it demonstrates the face detection feature 6 4 6 Face too small This feature is really testing the resolution of the face within the camera image We need a fairly detailed picture of the face to detect small movements of the eyes If your camera is low resolution you typically need to sit 20 80 cm from the camera This is a comfortable distance for most setups 6 4 7 Excess Yaw We need to track the pose of the head in order to calculate gaze on screen If you turn your head too much it can sometimes hide part of the eyes and we lose the ability to track gaze If you see this warning try learning further back or moving further away from the camera Sudden movements can also cause this icon to illuminate briefly while tracking is restored 6 4 8 Uneven lighting Due to the dynamic gain capabilities of most webcams excessive illumination of one part of the face will cause a loss of detail elsewhere This means that we can t see the whole face clearly This icon illuminates to warn that the mix of light and dark areas in the image is excessive and may be affecting system performance 6 4 9 Camera error This icon indicates that the system was unable to get images from the camera Our system uses the OpenCV interface to communicate with the camera Therefore problems and solutions to OpenCV camera drivers will also apply to our system We allow users to pass camera configuration arguments to the OpenCV drivers
38. tself in every new environment it sees Not easy So occasionally things will go wrong 7 1 Environment amp lighting The environment and particularly lighting amp illumination are critical for good results Cameras react to varying levels of light by automatically adjusting the brightness of the image In low light long exposures will produce blurry or noisy images that we can t use for precision eye gaze tracking The camera doesn t know which bit of the picture we re interested in the bit with the face in it Instead it balances brightness over the whole image So if the background is very bright it will make the face very dark Eye gaze tracking needs good and even illumination over the whole face and especially around the eyes Shadows can be a problem Good bright diffuse lighting overhead or daylight is best Desk lamps can be a problem because the light can be quite focused and cause shadows elsewhere If you are using daylight it is best for the user to face the window This means that the face will be well lit rather than having a bright sky behind the user s head and shadowed face Figure 28 Environmental problems The system will not work in a dark room because we won t be able to see the face clearly 32 Figure 29 Environmental problems Avoid having a bright light behind the head because the camera won t be able to see the face clearly While the background will be clear the face will be v
39. ue If not please contact us for individual support 7 17 Toolbar doesn t respond e g can t change graphics mode This problem is usually due to the local web service not running In this case only very limited functionality is available to allow normal web browsing Check that the local web pages are accessible and follow the instructions in the section above titled Can t see the web user interface 7 18 Feedback amp more help We re always trying to improve the system Please let us know if you have any suggestions tips questions or need some help Email us with talk xlabsgaze com 38 8 Credits 8 1 xLabs team The system was conceived and created by the xLabs team They are in alphabetical order Joe Hanna Dave Rawlinson Steve Roberts Alan Zhang 8 2 Open Source software This system wouldn t have been possible without many Open Source and free software contributions We have used many libraries under BSD like licences that generously allow commercial closed source use Unfortunately since we have a secret algorithm to protect we cannot make our software open source at this time Some of our software was developed using computers running Linux operating systems particularly Linux Mint In particular we used OpenCV for camera connectivity and for basic image processing tasks We use the STASM library for face detection and face fitting We use BOOST for C to provide all the bells and whistles for robus
40. ugh calibration for one user pose Figure 10 shows a recommended pattern of 4 marks to click Once you have pressed and held on at least 4 marks press c to trigger a calibration If successful you will now see that a cloud that follows your eyes around the screen This rough calibration will only work for one head position because the system hasn t learnt to generalize yet You can try a few more calibration clicks and see how the training improves or if you are observing red warning icons head to the troubleshooting section to see what s up Hopefully this quick start procedure has given you a taste of what the system can do 11 cLabs fyerGare Tracking test page dabe cem au Chromium oy alaks Eyaiicne Packin H ay Estorsions T xakas Ea Calibration K MEW T o gt 7 2 v OELSE J Figure 10 To get a quick and rough calibration press and hold on 4 marks that are well separated in X and Y e g the positions marked as 1 4 above Then press c to calibrate 12 3 How it works The system works by looking at you with a webcam It measures the appearance of your face and eyes in camera images and then tries to match these measurements to screen coordinates To do this matching successfully the system must be calibrated either via our grid interface see figure 9 or by letting the system watch while you surf the web 3 1 xLabs Client Software amp Browser Extension Our system has two parts
41. will very shortly receive by email a link to download the Client software Click the link to download the software xLabs eye gaze tracking download link Inbox x gt m S registration xlabs com au Jun 24 9 days ago to me 4 Thanks for registering to use our eye gaze tracking system You can download an installer by opening this link in your web browser This link will expire after a few hours You can generate another link by registering again Please let us know if you have any problems using the software Regards The xLabs development team http www xlabs com au Figure 3 Successfully received an email with a download link 2 3 Install Client software You will receive an installer program for the operating system you selected Run this program to install it Note in Windows it is preferable to Right Click and select Run as Administrator In OSX keep clicking Next until the install finishes 2 4 Configure Client software Once the installer has finished it will open your default web browser and show the Client software web pages Click Registration xLabs Eye Gaze Tracking xlabs com au Chromium 7 xLabs Eye Gaze Trackin x gt f xlabs 9522 index htmI Xlabs Eye Gaze tracking system About License Registration Browser extension Privacy Help topics Welcome If you can read this message the system has been successfully installed Thankyou Getting Star
Download Pdf Manuals
Related Search