WinCandide 1.3 User's Manual
Contents
1. The Toolbars Background toggles if the background image is visible or not If no background is loaded this menu item is disabled Model toggles if the face model is visible or not If no face model is loaded this menu item is disabled 6 J rgen Ahlberg Texture toggles if the face model is textured or not If not a wireframe model is shown provided a model is loaded This menu item is enabled if the face model has a texture to show the texture can be a generated from texture modes be grabbed from a background image or input video or loaded from an image file 4 3 The Model Menu Take texture copies the current background or the currently captured video frame to the texture of the model The texture coordinates are copied from the current vertex coordi nates e Known bug 2 This menu item is available only when a background is loaded so to take texture from a captured video frame an image file needs to be imported as background image first Fix texture coords copies the current vertex coordinates to the texture coordinates Global motion opens a dialogue window where the x y and z rotation scale and trans lation can be edited Approximate texture maps the background image or the currently captured video frame under the face model to a texture and then generates a new texture using the texture modes of the face model The new generated texture is computed as to minimize the summed squar2. Stop Capture stops the video capture and disconnects from the video input device Pause Capture stops the video capture but does not disconnect from the video input device Capture is restarted by picking Start Capture Video Format Display Source Settings are dependent on the video capture device driver These menu items are available if capture is started and if the used device driver sup ports the corresponding settings dialogue windows 4 5 The Tracking Menu This menu is only available in the InterFace version of WinCandide Find face is currently disabled Track face in video from files opens a dialogue box see Figure 7 where settings for face and facial feature tracking input and output can be edited If the tracking is done on images on files the path to files and the number of the first and last frame should be given The images should be JPEG files named 000 jpg 001 jpg starting with the given first frame number J rgen Ahlberg WinCandide supports a number of ways to output the results of the tracking Images saves each frame with the wireframe face model drawn on it Models saves an amd file for each frame FAPs to ASCII file saves the result as MPEG 4 Facial Animation Parameters in ASCII form The FAPs are saved to the file asc fap Note that the number of frames in the first row of the file is always set to 100 FAPs to binary file saves the result as MPEG 4 Facial Animation Param
3. is a reference documenting the functionality of each menu item in WinCandide 4 1 The File Menu New clears the current face model the background image remains Open opens a dialogue box for picking and loading a face model from a wfm smd or amd file e Save Save As saves the current face model to aa wfm smd or amd file e Import has several sub menu items Texture loads a texture image The texture will not be visible unless the face model has texture coordinates Background loads an image to be displayed as a background The image will be reshaped to the size 512x512 pixels so preferably the image is a square one Texture Modes load texture modes from a tmd or an aan file Active Modes loads texture modes and Active Model update data from an aam file This function is only available in the InterFace version of WinCandide Texture Coords load a face model from a wfm smd or amd file and copies its tex ture coordinates to the current face model Shape Animation Parameters load a face model from a wfm smd or amd file and copies its shape animation parameter values to the current face model e Export has one sub menuitem Image saves the image that is currently visible in the workspace Print is currently not functional e Exit exits WinCandide 4 2 The View menu The different toolbars can be turned on off using the first items in this menu The toolbars are described in Section 5
4. tracking Choose Tracking Track face from files The tracking dialogue will appear Write seq2 as input path 11 as first frame number and 20 as last frame number only frames 11 through 20 are included in the package e Drag the tracking dialogue away from the main window and press Start The model should now track the face through the 10 frames and finally look like Figure 5 right 3 Files and File Types Five proprietary file types for face modelling are defined by WinCandide Also WinCandide handles some more or less standard file types for images and textures 3 1 Face Model Files The wireframe file wfm The basic file type for face models is the wireframe file or the wfm file The exakt format for wfm files is defined in 1 A wfm file is an ASCII file containing e a list of vertex coordinates e a list of surfaces e definitions of animation and shape units e animation and shape parameter values e texture coordinates and a texture image file name e global animation parameters rotation scale translation 4 J rgen Ahlberg The static model file smd The static model file contains the definition of the head shape for one person The definition is stored as a list of shape parameter values that is how much each shape unit should deform the basic wireframe model The definitions of the shape units are not stored in the smd file The smd file contains a reference to the wfm file containing the rest of the data nee
5. 0 CAP NUM SCF Figure 9 The Status bar The text to the left tells that the Active Model just performed 25 iterations in 0 2642 s but that it has still not converged The fields to the right tells that the model is translated 0 242 along the x axis 0 097 along the y axis and not at all along the z axis Also Caps Lock Num Lock and Scroll Lock have been pressed 5 3 The Animation Shape and Texture Toolbars These three toolbars have the same appearance and functionality but they operate on differ ent parameters The Animation and Shape toolbars appear automatically when a face model containing definitions of animation units and shape units respectively is loaded and the Texture toolbar appear when a dynamic texture or an Active Model is loaded The leftmost button resets all the parameters controlled by the toolbar that is pressing Reset all parameters e on the Animation toolbar will make the face model show a neutral face 10 J rgen Ahlberg e on the Shape toolbar will make the face model have the standard shape e on the Texture toolbar will make the face model have the texture of the average face pos sibly very dark since the DC level is also set to zero In the Combo Box to the right of the reset button the parameter to modify can be chosen In Figure 10 the parameter controlling the Shape unit Head height is chosen on the Shape toolbar The parameter values all ranging from 1 to 1 can then be co
6. 1 2 3 4 5 6 7 WinCandide 1 3 User s Manual 11 Tabell 1 Necessary libraries for WinCandide 1 3 3 Library name File Version WinCandide libraries Face Model Library facemodel dll Image Processing Library imgproc dll GLtTracker Library gltracker dll Maths amp Stats Library mathstat dll MPEG 4 FBA Coder Library fbacoder dll Intel Performance Suite libraries Intel Signal Processing Library NSP nsp dll nspX dll Intel Image Processing Library IPL ipl dll nspX dll Intel JPEG Library UL ijl15 dll 7 References J Ahlberg CANDIDE 3 an updated parameterized face Report No LiTH ISY R 2326 Image Coding Group Dept of EE Link ping University Sweden January 2001 http www icg isy liu se candide The InterFace project Multimodal Analysis Synthesis System for Human Interaction to Virtual and Augmented Environments European Union 5 th framework Information Society Technologies http www ist interface org P Ekman and W V Friesen Facial Action Coding System Consulting Psychologist Press 1977 MPEG Working Group on Visual International Standard on Coding of Audio Visual Objects Part 2 Visual ISO 14496 2 1999 T F Cootes and C J Taylor Statistical Models of Appearance for Computer Vision Draft Report Wolfson Image Analysis Unit University of Manchester December 2000 http www wiau man ac uk Intel Perform
7. 2 5 Active Models Active Models are supported only in the InterFace version of WinCandide An Active Model is a face model that can reshape adapt itself to fit an image In WinCan dide Active Model data can be imported to a previously loaded face model Importing Active Model data also imports a dynamic texture An Active Model uses a modified version of the Active Appearance Algorithm 5 to itera tively refine an estimated model adaptation By picking the menu item Model Active Model Step or pressing Ctrl 1 the algorithm starts ending when it has converged or when the maximum 2 J rgen Ahlberg Figure 4 Dynamic texture Left image A dynamic texture has been imported to the model currently showing the average face only Middle image Texture Mode 2 has been emphasized by increasing the corresponding texture parameter from 0 0 to 0 15 in the Texture toolbar Right image The model is adapted to an image and the texture modes used to approximate Ctrl A the image under the model number of iterations is reached Convergence is declared when the algorithm can find no better model adaptation or when the largest suggested parameter change is smaller than a certain threshold the parameter min max threshold The maximum number of iterations and the parameter min max threshold can be set by pick ing the menu item Model Active Model settings Usage example This example shows step by step how to use some of the functions in WinC
8. WinCandide 1 3 User s Manual J rgen Ahlberg ahlberg isy liu se Report no LiTH ISY R 2344 Image Coding Group Dept of Electrical Engineering Link ping University SE 581 83 Link ping SWEDEN 4 April 2001 Abstract WinCandide is a piece of software for handling visualizing and synhtesiz ing wireframe face models It is specifically designed for the CANDIDE model included in the package but could be used with other face models as well WinCandide is also able to capture video and it can perform face and facial feature extraction tracking A somewhat limited version of the WinCandide software package is freely available from the Image Coding Group at Link ping University This manual will be continuously updated as the development of WinCan dide and the WinCandide libraries proceed 1 Introduction WinCandide is a program for handling face models like CANDIDE 1 that is a wireframe model of a human face possibly with texture on the wireframe surfaces WinCandide also supports dynamic texturing by eigenfaces or texture units modes and can automatically approximate an underlying background image using those There are two versions of WinCandide one public version and one internal version for the InterFace project 2 The internal version from now on called the InterFace version supports automatic face and facial feature tracking 2 J rgen Ahlberg 1 1 Distribution The public version of WinCandide can be fr
9. ance Library Suite http developer intel com software products perflib J Str m et al Very Low Bit Rate Facial Texture Coding Proc IWSNHC3DI Rhodes Greece September 1997 pp 237 240 12 J rgen Ahlberg
10. andide All neces sary files are included in the WinCandide package e Load a face model with the shape of the person to be tracked Choose File Open or press Ctrl O and pick the file jag jag smd e Load Active Model data Choose File Import Active Modes or press Ctrl M Pick the file 40x42rgb aam for RGB computations or 40x42gray aam for grayscale computations e Show the wireframe again Choose View Texture or press AlttV X e Load an image to adapt the model to Choose File Import Background or press Ctrl B Pick the file segq2 jpg e Drag the face model with the mouse so that it is positioned on the face e Resize the model by holding Space while dragging the model Make the model have approximately the same size as the face for example as in Figure 5 left e Adapt the model to the face Choose Model Active Model step or press Ctrl 1 e If everything works correctly the model should now be nicely adapted to the face as in Fig ure 5 middle On the Status bar you can read how many iterations was used how long time it took and if it has converged or not If the model adaptation has not converged you can run the Active Model step once more WinCandide 1 3 User s Manual 3 Figure 5 The face model approximately adapted to the face image left The face model adapted by the Active Model step after pressing Ctrl 1 twice middle After tracking the face through frames 11 20 in the test sequence right e Face
11. animation units under some circumstances called animation modes telling how to change the dynamic shape of the face due to for example lip or eye movement Examples of animation units are the Action Units from FACS 3 or the feature point move ments parameterized by the MPEG 4 FAPs Facial Animation Parameters see 4 Corre spondingly there can be a set of shape units under some circumstances called shape modes 1 Often called just faces but that can cause confusion in this context 0 J rgen Ahlberg controlling the static shape of the face That is the shape units model such deformations that change between different individuals like eye separation distance head width height ratio etc When a face model is loaded animation and shape units might be loaded as well if they are defined in the face model file In that case the Animation toolbar and the Texture toolbar will appear see Figure 1 or Section 5 The Toolbars Using these toolbars the animation parameters and the shape parameters can be modified reshaping the face model 2 2 Background Images and Video Capture A background image can be loaded into WinCandide as shown in Figure 1 Since the work space OpenGL viewport is quadratic 512x512 pixels the loaded image is reshaped to that size An image to be used as background in WinCandide should therefore preferably be quadratic Pick the menu item File Import Background or press Ctrl B to loa
12. d a background image WinCandide can also be used for video capture of course provided that you have a video device connected to your computer If you pick the menu item Capture Start Capture WinCandide will ask which video capture device to use and then start the video capture The captured frames will be shown centred in the workspace as shown in Figure 2 WinCandide supports only video frames that fit inside the workspace that is they must be smaller than 512x512 pix els Also the captured frames must be in 24 bit RGB format If these requirements are not fulfilled no video will be visi ble in WinCandide In that case the video format settings Home Os Video capture Uranie can be changed by picking the menu item Capture size 352x288 pixels in Video Format Settings provided that your video capture WinCandide device supports a video format dialogue box 2 3 Textured Face Models To be textured a wireframe model needs texture coordinates and a texture If a face model is loaded and a background image or a captured video frame is present the texture can be acquired from the background by pressing Ctrl T or picking the menu item Model Take texture This command will a copy the background to the model s texture b copy the model s current coordinates to the model s texture coordinates The texture coordinates can be changed by picking the menu item Model Fix texture coords or pressing Ctrl F this will perf
13. ded for a face model The animated model file amd The animated model file contains the local and global animation parameter values and a refer ence to a wfm or an smd file Typically for an image sequence showing one person the shape of the head would be extracted once and stored as an smd file Then for each frame the local and global motion of the face would be extracted and stored in an amd file 3 2 Texture Modes and Active Model Data The texture mode file tma A texture mode file contains information how to load a dynamic texture see Section 2 4 Dynamic Textures above A dynamic texture consists of a mean average texture a set of texture modes and a list of texture coordinates The texture coordinates are stored as wire frame coordinates in a face model file The Active Model data file aam The Active Model data file contains information about how to load a dynamic texture as the tmd file and how to load update images enabling the face model to actively adapt itself to a face and its features in an image see Section 2 5 Active Models above aam files are handled by the InterFace version only of WinCandide 3 3 Images WinCandide handles four types of image files for textures backgrounds dynamic textures and Active Model update images JPEG jpg portable graymaps pgm portable pixmaps ppm and portable floatmaps pfm WinCandide 1 3 User s Manual 5 4 The Menus This section
14. ed difference between the approximated and generated texture Active Model step modifies the shape of the face model by changing the global and local animation parameters so that it should be better adapted to a face in the background image or the currently captured video frame This function is only available in the Inter Face version of WinCandide Create normalized images is used for creating Cieate nonmalized images x texture modes from training data The dialogue box shown in Figure 6 will be shown and with the Data path examplepath parameters given as in the figure the following will happen when Start is pressed Normalized image format Width 128 Height 192 1 A face model defining the normalized shape is loaded from the file MV Grayscale examplepath 128x192 128x192 amd cen Figure 6 The dialogue box for creating normalized images 2 The first row of the file examplepath con tents txt is read This ASCII file should on each row specify a directory and the number of training images in this directory Assume that the first row is jag 25 3 The background files 000 ppm 024 ppm and the face model files 000 amd 024 amdin the directory examplepath jag are loaded 4 Each of the loaded models will acquire the texture from its corresponding background image and they will then be reshaped to the normalized shape WinCandide 1 3 User s Manual 7 5 The resulting image
15. eely downloaded at the CANDIDE home page http www icg isy liu se candide WinCandide is postcardware that is send a postcard with a picture of your town university home to the author Please state name and email address on the postcard The InterFace version is available to all partners in the InterFace consortium 2 Both versions include the Candide face model and a few example files background images dynamic textures 1 2 System requirements WinCandide runs in the Microsoft Windows environment and it is so far tested on Win dows 98 only Since WinCandide uses the Intel Performance Library Suite see Section 6 Libraries it might not work on computers using non Intel processors WinCandide uses OpenGL for visualization and Active Model computations and will thus be really slow on a computer without a 3D graphics card with OpenGL support 2 Overview of WinCandide The basic functionality provided by WinCandide is to load manipulate and visualize wire frame face models using a graphical user interface GUI The main components of WinCa nide s GUI is shown in Figure 1 A loaded face model can the be manipulated using the different toolbars by dragging it with the mouse or by editing the global motion parameters The most basic manipulations are performed directly with the mouse e To translate the model in plane just drag the model with the mouse The current x y and z translation of the face m
16. eters in binary com pressed form using the MPEG 4 reference coder e Known bug 3 The binary FAP file is saved in the current directory and not in the given output path e Known bug 4 The MPEG 4 reference coder always outputs a file out fap in the current directory Tracker settings x File input Path First frame no a gt Last frame no 100 Output Path fout FAPs to ASCII file Images l FAPs to binary file Models FAPs via socket on port 1000 To WinCandide on port 1000 Cancel Figure 7 The Tracker Settings dialogue box FAPs to socket waits for another computer or another program on the same computer to connect to the given port It then starts the tracking and compresses the FAPs using the MPEG 4 reference coder and send them to the receiving program Note that the same port can usually not be used twice without exiting WinCandide in between To WinCandide on port waits for another WinCandide instance to connect to the given port see below and then starts the tracking It the communicates with the other Win Candide using an internal WinCandide bitstream format Note that the same port can usually not be used twice without exiting WinCandide in between Receive from socket connects to an instance on WinCandide performing face tracking The currently loaded face model is then animated according to the animation parameters coming from the other WinCandide instance 4 6 The Help Men
17. ntrolled by the moving the slider or editing the value directly These toolbars can be shown hidden via the View menu or by pressing Alt V A H E Reset all parameters Sliders controlling the parameter values AUYO Upper lip raiser AU10 ead height exture Mode 2 Name of the choosen texture shape animation unit The parameter values Figure 10 The Animation toolbar top the Shape toolbar middle and the Texture toolbar bottom 6 Libraries WinCandide requires a bunch of dynamic link libraries DLLs to run some included in the WinCandide package some provided from other places The libraries and the required ver sions are listed in Table 1 and briefly described below The WinCandide Libraries are continuously developed and the WinCandide application is mainly a GUI for the functionality they provide They are developed by the same author as WinCandide with the exception of the MPEG 4 FBA Coder Library which was created by Igor Pandzic from the MPEG 4 Reference Software The Intel Performance Libraries are a bit special since they are processor dependent The basic libraries ipl dl11 and nsp dll are always needed but in addition two more DLLs are needed For example if your computer has a Pentium III processor ipla6 dll and nspa6 dll are needed but if the processor is a Pentium II ip1m5 d11 and nspm5 d11 are the required ones If your computer does not have an Intel processor the behaviour is unde fined
18. odel will be shown on the Status bar e Known bug 1 When dragging translating a face model all animation units seem to bet set to zero in fact they are not really set to zero but they are visualized that way e To scale the model press the Space key and drag the model upwards for increasing the size of the model and downwards for decreasing the size The current x y and z scaling will be shown on the Status bar e To perform in plane rotation that is around the z axis press the Shift key while dragging the model to the left right The current x y and z rotation angles will be shown on the Status bar WinCandide 1 3 User s Manual 1 Menu bar Animation toolbar Shape toolbar VAW nC arbia paj med Texture toolbar Main toolbar riw E H pi ial a T Wireframe Workspace face model Background image Naas iy oo co The Status bar Figure 1 The main components of WinCandide s graphical user interface e To perform out of plane rotation that is around the x and y axis press the Ctrl key while dragging the model to the left right for rotation around the y axis or up down for rotation around the x axis The current x y and z rotation angles will be shown on the Status bar 2 1 Face Models A face model is basically a set of vertices and a set of surfaces Each vertex is a set of coor dinates in a 3D space and a surface connects three vertices Additionally a face model can have a set of
19. orm b but not a above Taking a texture from a background image is illustrated in Figure 3 WinCandide 1 3 User s Manual 1 Figure 3 Taking a texture from a background image Left image The model has been adapted to the background image Middle image The texture has been taken as texture Ctrl T and the background made invisible Alt V B Right image The model is moved away from the previous position and modified slightly using the animation units Even if a model has a texture it is not necessary to show it pick the menu item Model View Texture to toggle the model between a textured model and a plain wireframe model 2 4 Dynamic Textures A face model can also be textured by a dynamic texture A dynamic texture consists of a mean average face and a set of texture modes or geometrically normalized eigenfaces 7 When a dynamic texture is loaded by picking the menu item File Import Texture modes tex ture coordinates are loaded at the same time The face model will now look like the average face with all the texture parameters set to zero well not really all typically the first texture mode is for convenience just the DC level of the average face The texture modes can then be used for approximating the underlying image as illustrated in Figure 4 Use the Texture toolbar see Figure 1 or Section 5 The Toolbars to change the texture parameters or press Ctrl A to automatically approximate the background image
20. s size 128x192 pixels are saved in example path 128x192 jag If Grayscale is not checked they will be saved as 000 ppm 024 ppm Otherwise they will be converted to grayscale and saved as 000 pgm 024 pgm 6 The next row is read from contents txt and steps 3 5 are repeated until end is found in contents txt Thus normalized representations of all the faces in the training set as defined in the file contents txt will be computed and stored From the resulting files texture modes can be created performing a PCA this is however not done by WinCandide Active Model training is used for creating Active Model update images from training data This function is only available in the InterFace version of WinCandide Active Model settings shows a dialogue window where the settings for the Active Model steps can be edited Currently only two settings are changeable The min max parameter threshold and the maximum number of iterations see Section 2 5 Active Models This function is only available in the InterFace version of WinCandide 4 4 The Capture Menu Start Capture checks what video input devices that are connected to the computer and asks which device to use If the user confirms one of them video capture is started Note that WinCandide currently supports only a few input formats the captured frames should be neither higher nor wider than 512 pixels and the pixel format must be 24 bit RGB
21. u About WinCandide gives some very basic information about WinCandide Libraries gives information about what WinCandide and Intel DLLs that are loaded see Section 6 Libraries WinCandide 1 3 User s Manual 9 5 The Toolbars 5 1 The Main Toolbar The Main toolbar appears on the left side of the WinCandide win dow and it is more or less the standard toolbar for MFC Applica New D tions The first four buttons have exactly the same functionality as G Open the menu items New Open Save and Print on the File menu The Save HH Help button is the same as the menu item Help About WinCandide Print Finally the button Reset global motion sets the global motion parameters rotation translation to zero and the scale to one Help I Reset g global To show hide the Main toolbar pick the menu item View Toolbar or motion press Alt V T Figure 8 The Main toolbar 5 2 The Status Bar The Status bar is located below the workspace The left part shows messages from WinCan dide to the user for example information about Active Model adaptation To the right there is anumber of fields The first one has the letters TRA ROT or SCA and tells if the follow ing three files show the translation rotation or scale of the face model along around the x y and z axis respectively The three rightmost fields are the standard ones from Windows see Figure 9 25 iterations in 264 2 ms not converged TRA 0 242 0 097 0 00
Download Pdf Manuals
Related Search