visage|toolkit user manual
Contents
1. 1 PP p b m Put bed mill 2 FF f v Far voice 3 TH T D Think that 4 DD t d Tip doll 38 MPEG 4 FBA overview BAP grouping and masking 5 kk k g Call gas 6 CH tS dZ S Chair join she 7 SS S Z Sir zeal 8 nn n I Lot not 9 RR r Red 39 MPEG 4 FBA overview BAP grouping and masking 10 aa Car 11 E Bed 12 ih Tip 13 oh Top 14 ou book 40 MPEG 4 FBA overview BAP grouping and masking 8 Annex 3 MPEG 4 FBA primary expressions 41 MPEG 4 FBA overview BAP grouping and masking 1 Joy The eyebrows are relaxed The mouth is open and the mouth corners pulled back toward the ears 2 Sadness The inner eyebrows are bent upward The eyes are slightly closed The mouth is relaxed 3 Anger The inner eyebrows are pulled downward and together The eyes are wide open The lips are pressed against each other or opened to expose the teeth Fear The eyebrows are raised and pulled together The inner eyebrows are bent upward The eyes are tense and alert Disgust The eyebrows and eyelids are relaxed The upper lip is raised and curled often asymmetrically surprise The eyebrows are raised The upper eyelids are wide open the lower relaxed The jaw is opened 42 MPEG 4 FBA overview BAP grouping and masking 9 Annex 4 MPEG 4 Body Animation Parameters 43 MPEG 4 FBA overview BAP grouping and mas2. 240 120 67 vc6_torsion Twisting of vertebra C6 256 240 120 Forward backward motion of 120 vertebra C6 in the sagittal plane 240 120 ve7_roll Sideward motion of vertebra C7 256 240 120 ee eee nls 70 vc7_torsion Twisting of vertebra C7 256 240 120 71 Forward backward motion of 120 vertebra C7 in the sagittal plane 240 120 72 vtl_roll Sideward motion of vertebra T1 128 660 360 660 360 73 vtl_torsion Twisting of vertebra T1 360 660 360 74 Forward backward motion of 360 vertebra T1 in the sagittal plane 660 360 75 vt2_roll Sideward motion of vertebra T2 256 240 120 76 vt2_torsion Twisting of vertebra T2 256 240 120 77 Forward backward motion of 120 vertebra T2 in the sagittal plane 240 120 78 vt3_roll Sideward motion of vertebra T3 256 240 120 ee ee ole 79 vt3_torsion Twisting of vertebra T3 256 240 120 47 MPEG 4 FBA overview BAP grouping and masking vertebra T3 in the sagittal plane 240 120 81 vt4_roll Sideward motion of vertebra T4 256 240 120 Ol ee lan 82 vt4_torsion Twisting of vertebra T4 256 240 120 83 Forward backward motion of 120 vertebra T4 in the sagittal plane 240 120 84 vt5 roll Sideward motion of vertebra T5 256 240 120 85 vt5_torsion Twisting of vertebra TS 256 240 120 Forward backward motion of 120 vertebra T5 in the sagittal plane 240 120 87 vt6_roll Sideward motion of v
3. 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 BIN 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 18 59 1359 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 65 MPEG 4 FBA overview 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859
4. 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 FIN 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 FPX 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 FPN 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 BRR 25 0 0 BME 3 s kas Dr 3 73 38 3 803 730 3 37 3433 340 0 OF 0 0 BQU 1 BIX 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859
5. corner Depth displacement of jaw Side to side displacement of jaw Depth displacement of bottom middle lip Depth displacement of top middle lip Upward and compressing movement of the chin like in sadness Vertical displacement of top left eyelid BAP grouping and masking MPEG 4 FBA overview BAP grouping and masking displacement of top right eyelid 21 close_b_l_eyelid Vertical 600 240 displacement of bottom left a 22 close_b_r_eyelid Vertical 240 displacement of bottom right eyelid yaw_l_eyeball Horizontal 420 orientation of left eyeball yaw_r_eyeball Horizontal 420 orientation of right eyeball T D 5 T N N So fa No S 5 ea No S 5 6 pitch_r_eyeball Vertical 300 orientation of right eyeball thrust_l_eyeball Depth 180 displacement of left eyeball thrust_r_eyeball Depth 180 displacement of right eyeball T D 5 ea D 5 25 pitch_l eyeball Vertical 300 orientation of left eyeball 29 Idilate I pupil Dilation of left pupil 30 dilate_r_pupil Dilation of right pupil 1 raise_l_i_eyebrow Vertical displacement of left inner eyebrow 420 420 2 raise_r_i_eyebrow Vertical displacement of 360 right inner eyebrow 33 raise_l_m_eyebrow Vertical 360 displacement of left middle eyebrow 34 raise_r_m_eyebro Vertical 360 w displacement of right middle eyebrow 33 O
6. frame is encoded If these values are not low enough 1 e if a FAP goes out of range while encoding results are unpredictable The first 7 values correspond to visemel viseme2 viseme blend expression expression intensity expression2 expression2 intensity The rest of the parameters correspond to the low level FAPs p_max This is an array of 73 integer values setting the maximum value that each FAP can take when an P frame is encoded If these values are not big enough i e if a FAP goes out of range while encoding results are unpredictable The first 7 values correspond to visemel viseme2 viseme blend expression expression intensity expression2 expression2 intensity The rest of the parameters correspond to the low level FAPs p_min This is an array of 73 integer values setting the minimum value that each FAP can take when an P frame is encoded If these values are not low enough i e if a FAP goes out of 26 MPEG 4 FBA overview BAP grouping and masking range while encoding results are unpredictable The first 7 values correspond to visemel viseme2 viseme blend expression expression intensity expression2 expression2 intensity The rest of the parameters correspond to the low level FAPs bap i max This is an array of 296 integer values setting the maximum value that each BAP can take when an I frame is encoded If these values are not big enough i e if a FAP goes out of range while encoding result
7. if used gt lt group 2 fap mask type gt lt group 10 fap mask type gt lt FAP group 10 mask if used gt FQU lt fap_ quant 0 31 gt BQU lt bap quant 0 31 gt FIX lt FAP3 I max gt lt FAP68 I max gt FIN lt FAP3 I min gt lt FAP68 I min gt FPX lt FAP3 P max gt lt FAP68 P max gt FPN lt FAP3 P min gt lt FAP68 P min gt BIX lt BAP1 I max gt lt BAP296 I max gt BIN lt BAP1 I min gt lt BAP296 I min gt BPX lt BAP1 P max gt lt BAP296 P max gt BPN lt BAP1 P min gt lt BAP296 P min gt GDR lt gender bit l male 0 female gt CDM lt coding type l segment based 0 frame based gt The minimum entry for one frame is FBM 1 FRR lt frame number as in the FAP file gt ISI 0 1 for P or I frame respectively Any combination of the remaining lines can be present ONLY in an I frame after ISI 1 64 MPEG 4 FBA overview An example EPF file 13 1 An example EPF file The full file emotions epf is available in the folder Samples Encoder Parameter Files epf Here is the extract of the file EPF file for integration of Marco30 amp welcome sequences EPFL Tolga Capin January 5 1999 FBM 3 FFR 0 BFR 0 TSL oh FRR 30 0 0 FMT 0 2 Vd Ae e eT T ee de a O00 0 2 Ay Ae ae le 0 00 3 lt 0 0 8 235 30 0 FQU 2 FIX 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 600 840 840 600 600 600 600 600 600 600 600 600 600 600 600 600
8. moves the Feature Point 2 1 bottom of the chin downwards and is expressed in MNS mouth nose separation units The MNS unit is defined as the distance between the nose and the mouth see Figure 1 divided by 1024 Therefore in this example a value of 512 for the FAP no 3 means that the bottom of the chin moves down by half of the mouth nose separation The division by 1024 is introduced in order to have the units sufficiently small that FAPs can be represented in integer numbers All low level FAPs are listed and defined in Annex 1 MPEG 4 Facial Animation Parameters Description FAPU Value IRISDO 3 1 y 3 3 y Iris diameter by definition it is equal to the IRISD IRISDO 1024 3 2 y 3 4 y distance between upper ad lower eyelid in neutral face MW0 8 3 x 8 4 x Mouth width MW MW0 1024 Table 1 Facial Animation Parameter Units FAPU The specification includes two high level FAPs expression and viseme Expression can contain two out of a predefined list of six basic expressions Intensity values allow to blend the two expressions The expressions are are listed and defined in Annex 3 MPEG 4 FBA primary expressions Similarly the Viseme parameter can contain two out of a predefined list of 14 visemes and a blending factor to blend between them The visemes are are listed and defined in Annex 2 MPEG 4 Visemes MPEG 4 FBA overview FAP grouping and masking Figure 1 Face Animation Parame
9. to the shoulder joint 1E henee t shoulder twist Figure 7 Front and side views of the mobilities of the arm in rest position At the clavicula sternoclavicular abduct up and down motion in the coronal plane sternoclavicular _rotate rotation in the transverse plane 17 MPEG 4 FBA overview Detailed body topology and degrees of freedom At the scapula acromioclavicular_abduct up and down motion in the coronal plane acromioclavicular _rotate rotation in the transverse plane At the shoulder shoulder_flexion forward backward motion in the sagital plane shoulder_abduct sideward motion in the coronal plane shoulder_twisting along the forearm axis At the elbow elbow_flexion flexion extension of the arm in the sagital plane elbow_twisting along the arm axis At the wrist wrist_twisting along the arm axis This DOF is redundant with the elbow twisting except that only the hand rotate and not the forearm wrist_flexion rotation of the hand in the coronal plane wrist_pivot rotation of the hand in the sagital planes Head rotations Note that there are three BAPs defined for head rotation skullbase_roll skullbase_torsion skullbase_tilt There are also 3 FAPs for head rotation If both FAPs and BAPs are used for head rotation then these FAPs shall denote the head rotation with respect to the skullbase coordinate system Hands The hand mobilities have a standard struct
10. 102 103 104 111 112 113 vclroll vcltorsion vcltilt vt3roll vt3torsion vt3tilt vt5roll vt5torsion vtStilt vt9roll vt9torsion vt9tilt 51 52 53 78 79 80 84 85 86 96 97 98 1_pinky1_flexion 1_pinky2_flexion _pinky3_flexion 1_ring1_flexion _ring2_flexion 1_ring3_flexion I middlel flexion I middle2 flexion l middle3 flexion I indexl1 flexion 1 index2 flexion Il index3 flexion I thumbl1 flexion I thumbl1 pivot I thumb2 flexion I thumb3 flexion 125 131 133 137 143 145 149 155 157 161 167 169 171 173 177 179 22 MPEG 4 FBA overview BAP grouping and masking 16 Right hand1 r_pinky1_flexion r_pinky2_flexion r_pinky3_ flexion r_ring1_flexion r_ring2_flexion r_ring3_flexion r middlel flexion r middle2 flexion r middle3 flexion r_index1_ flexion r index2 flexion r_index3_ flexion r_thumb1_flexion r_thumb1_pivot r_thumb2_flexion r_thumb3_flexion 126 132 134 138 144 146 150 156 158 162 168 170 172 174 178 180 17 Left hand2 1_pinky0_flexion I pinkyl pivot _pinky1_twisting I ringO flexion 1 ringl pivot l_ring1_twisting 1_middleO_flexion l middlel pivot I middlel twisting I index0 flexion I indexl pivot 1 indexl twisting I thumbl1 twisting 123 127 129 135 139 141 147 151 153 159 163 165 175 Right hand2 r_pinkyO_flexion r pinkyl pivot r_pinky1_twisting r_ring0_flexion r ringl pivot r_ring1_twisting r middle0 flexion r_middlel_ pivot r_middle1_twisting r_index0_fl
11. 120 vertebra T12 in sagittal plane 240 120 108 vll_roll Sideward motion of vertebra L1 128 660 360 660 360 109 vll torsion Twisting of vertebra L1 128 660 360 660 360 110 Forward backward motion of 360 vertebra L1 in sagittal plane 660 360 111 vl2 roll Sideward motion of vertebra L2 256 240 120 cic esd eal er 112 vl2_torsion Twisting of vertebra L2 256 240 120 113 Forward backward motion of 120 vertebra L2 in sagittal plane 240 120 114 vl3_roll Sideward motion of vertebra L3 128 660 360 660 360 115 vl3_torsion Twisting of vertebra L3 360 660 360 116 Forward backward motion of 360 vertebra L3 in sagittal plane 660 360 117 vl4 roll Sideward motion of vertebra L4 256 240 120 118 vl4_torsion Twisting of vertebra L4 256 240 120 119 Forward backward motion of 120 vertebra L4 in sagittal plane 240 120 120 vI5_roll Sideward motion of vertebra L5 128 660 360 121 vl5_torsion Twisting of vertebra L5 128 660 360 660 360 49 MPEG 4 FBA overview BAP grouping and masking Forward backward motion of 128 vertebra LS in sagittal plane 660 360 the pinky finger 240 4 120 the pinky finger 240 4 120 125 1_pinky1_flexion First knukle of the pinky finger 128 1140 240 240 240 126 r_pinky1_flexion First knukle of the pinky finger 128 1140 240 240 240 127 II pinkyl pivot Lateral mobility
12. 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 An example EPF file 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1 859 1859 1859 1359 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1659 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 BPX 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1
13. 2 0 8929 0 0690 thumb1 0 1912 0 8734 0 0657 thumb2 0 1912 0 8156 0 0079 thumb3 0 1912 0 8007 0 0070 55 MPEG 4 FBA overview BAP grouping and masking _index0 0 1912 0 8259 0 0460 _index1 0 1912 0 8050 0 0460 index2 0 1912 0 7595 0 0460 index3 0 1912 0 7370 0 0460 I middleo 0 1912 0 8282 0 0710 _middle1 0 1912 0 8071 0 0710 _middle2 0 1912 0 7525 0 0710 _middle3 0 1912 0 7263 0 0710 _ringO 0 1912 0 8302 0 0972 I ring1 0 1912 0 8098 0 0972 I ring2 0 1912 0 7570 0 0972 I ring3 0 1912 0 7328 0 0972 _pinkyO 0 1912 0 8373 0 1211 _pinky1 0 1912 0 8173 0 1211 _pinky2 0 1912 0 7759 0 1211 _pinky3 0 1912 0 7584 0 1211 r_sternoclavicular 0 0757 1 4844 0 0251 r_acromioclavicular 0 0899 1 4525 0 0322 r_shoulder 0 1968 1 4642 0 0265 r_elbow 0 1982 1 1622 0 0557 r_wrist 0 1972 0 8929 0 0690 r_thumb1 0 1912 0 8734 0 0657 r_thumb2 0 1912 0 8156 0 0079 r_thumb3 0 1912 0 8007 0 0070 r_index0 0 1912 0 8259 0 0460 r_index1 0 1912 0 8050 0 0460 r_index2 0 1912 0 7595 0 0460 r_index3 0 1912 0 7370 0 0460 r_middle0 0 1912 0 8282 0 0710 r_middle1 0 1912 0 8071 0 0710 r_middle2 0 1912 0 7525 0 0710 r_middle3 0 1912 0 7263 0 0710 r_ringO 0 1912 0 8302 0 0972 r_ring1 0 1912 0 8098 0 0972 r_ring2 0 1912 0 7570 0 0972 r_ring3 0 1912 0 7328 0 09
14. 4 FBA overview An example BAP file 13 Annex C The EPF file format 63 MPEG 4 FBA overview An example BAP file The Encoder Parameter File is designed to give the encoder all the informations related to the corresponding FAP and BAP files like I and P frames masks frame rate quantization scaling factor and so on The EPF file is necessary for encoding FAP and BAP files into binary MPEG 4 FBA bitstreams The apparent complexity of the EPF file comes from the many options However in practical use most of the time the default EPF file given below as example will work well and will only require adjusting the number of frames this does not mean that it will achieve the best coding efficiency though Note the mask defined in the EPF file overrides the mask defined in the corresponding FAP and BAP files The syntax is the following comment to be skipped FBM lt fba_paramset_mask 0 3 gt each fba frame starts with this line FFR lt face frame number as listed in FAP file gt only used if FBM amp l BFR lt body frame number as listed in BAP file gt only used if FBM amp 2 ISI lt is_intra gt FRR lt framerate 0 255 gt lt seconds 0 15 gt lt ntsc offset gt BRR lt framerate 0 255 gt lt seconds 0 15 gt lt ntsc offset gt TC lt timecode hours gt lt minutes gt lt markerbit gt lt seconds gt SKN lt number of frames to skip gt FMT lt group 1 fap mask _type gt lt FAP group 1 mask
15. 72 r_pinkyO 0 1912 0 8373 0 1211 r_pinky1 0 1912 0 8173 0 1211 r_pinky2 0 1912 0 7759 0 1211 r_pinky3 0 1912 0 7584 0 1211 56 MPEG 4 FBA overview BAP grouping and masking 11 Annex A The FAP file format 57 MPEG 4 FBA overview BAP grouping and masking The MPEG 4 ASCII Facial Animation Parameters FAP file is used for storing and exchanging MPEG 4 facial animation data in an ASCII form Syntax at the beginning of a line indicates a comment Header FAP Mask Frame Number FAP Values the above two lines are repeated for each frame of animation where Header lt float file format version gt lt string sequence name gt lt int frame rate gt lt int number of frames gt FAP Mask lt int 68 FAP mask gt The FAP mask consists of 68 space separated 1 or 0 digits I indicates that a FAP is used in this particular frame O indicates that it is not used In the next line after the frame number values for all used FAPs are listed space separated Frame Number lt int frame number gt FAP Values lt int viseme selectl 0 14 gt lt int viseme select2 0 14 gt lt int viseme blend 0 63 gt lt int viseme def 0 1 gt lt int expression selectl 0 6 gt lt int expression intensityl 0 63 gt lt int expression select2 0 6 gt lt int expression intensity2 0 63 gt lt int init face 0 1 gt lt int expression def 0 1 gt lt int FAP values for used FAPs gt 5
16. 8 11 1 MPEG 4 FBA overview An example FAP file An example FAP file The full file emotions fap is available in the folder Samples MPEG 4 ASCII Animation fap Here is the extract of the file an empty line space indicates a new line in order to distinguish from long lines that wrap Header description 2 1 emotions 25 1050 07 208 215 Deed dD ds Ld 00000000000 0 Orr Ore One Ore Ou Ae ONS La je oO je je je oO O jo je oO 00000000000 0 100 100 80 800000000000 00 je O O O je O je O je O je O Or oOo O S ome Lue JR a ja Or O je O je O je O je O je je 10000000000 0 100 100 80 800000000000 00 Qo Ore DO m E Or je je oO je oO je oO je as O oO 200000000000 100 100 80 8000000000000 0 OO WAST sD aT et ids Te EA SE TO Oe 0 FOR On TLA ST sD ad et TTS Teun OO 0000000000000000000000000000 0 0 1049000000000 0 0 100 100 80 800000000000 0 0 59 MPEG 4 FBA overview An example FAP file 12 Annex B The BAP file format 60 MPEG 4 FBA overview An example FAP file The MPEG 4 ASCII Body Animation Parameters BAP file is used for storing and exchanging MPEG 4 body animation data in an ASCII form Syntax at the beginning of a line indicates a comment Header BAP Mask Frame Number FAP Values the above two lines are repeated for each frame of animation where Header lt float file form
17. 859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 BPN 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1899 1859 1659 1859 1859 1859 1859 1859 FBM 3 FFR 1 BFR 1 ISI 0 FBM 3 FFR 2 BFR 2 ISI 0 FBM 3 FFR 3 BFR 3 ISI 0 FBM 3 FFR 4 BFR 4 ISI 0 FBM 3 FFR 5 BFR 5 ISI 0 66 1859 1859 1859 1859 1859 18659 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859 1859
18. 860 600 sagittal plane 1860 600 I shoulder abduct Sideward motion in the coronal 64 240 600 5 Ir shoulder abduct Sideward motion in the coronal 64 1860 600 I shoulder twisting Rotation along the scapular axis 256 960 360 960 360 37 r_shoulder_twisting Rotation along the scapular axis 256 960 360 45 w ON id el ud h MPEG 4 FBA overview BAP grouping and masking 38 l elbow_flexion Flexion extension of the arm in 64 60 600 the sagittal plane 1560 600 39 r_elbow_flexion Flexion extension of the arm in 64 60 600 the sagittal plane 1560 600 40 1_elbow_twisting Rotation of the forearm along the 256 960 360 upper arm axis 960 360 41 r_elbow_twisting Rotation of the forearm along the 256 960 360 upper arm axis 960 360 42 1_wrist_flexion Rotation of the hand in the 128 960 600 coronal plane 960 600 43 r_wrist_flexion Rotation of the hand in the 128 960 600 coronal plane 960 4 600 44 1_wrist_pivot Rotation of the hand in the 128 660 360 sagittal planes 660 360 45 Ir Wwrist pivot Rotation of the hand in the 128 660 360 sagittal planes 660 360 46 1_wrist_twisting Rotation of the hand along the 256 660 360 forearm axis 660 360 47 r_wrist_twisting Rotation of the hand along the 256 660 360 forearm axis 660 360 48 skullbase_roll Sideward motion of the skull 128 1860 600 a
19. Ps HumanoidRoot_tr_vertical HumanoidRoot_tr_lateral HumanoidRoot_tr_frontal is defined in millimeters The neutral body position when all BAPs are 0 is defined by standing posture illustrated in Figure 3 This posture is defined as follows the feet should point to the front direction the two arms should be placed on the side of the body with the palm of the hands facing inward Figure 3 Default body posture 12 MPEG 4 FBA overview Detailed body topology and degrees of freedom Detailed body topology and degrees of freedom The degrees of freedom DOF sufficient to locate and animate a human body are decomposed into six DOF for the global location and 180 DOF for the internal mobilities The topology of the suggested joints is given below The hands are optional with an additional set of joints skeleton_root sacroiliac_tilt sacroiliac_torsion vi5roll I vl5tilt I hip flexion r_hip abduct r_hip twisting I hip_abduct vitroll I hip twisting r knee flexion I knee flexion vit tilt r knee twisting I knee twisting r ankle twisting _ankle_twisting r_ankle_flexion I ankle flexion r_subtalar_fl _subtalar fl r_midtarsalf vt12roll vt12torsion vti 2tilt I Ng I midtarsalf _metatarsalf r_metatarsalf til vi6tilt _clav_abduct r_clav_abduct _clav_rotate r_clav_rotate scap abduct r scap abduct l scap rotate r_scap_rotate shoulder _ flexion _sh
20. S S u NO S 5 Ke MPEG 4 FBA overview raise I o eyebrow raise_r_o_eyebrow squeeze_l_eyebrow squeeze_r_eyebro W shift_tongue_tip raise_tongue_tip thrust_tongue_tip raise_tongue head_roll Vertical displacement of left outer eyebrow Vertical displacement of right outer eyebrow Horizontal displacement of left eyebrow Horizontal displacement of right eyebrow Horizontal displacement of left cheeck Horizontal displacement of right cheeck N N S S S S Vertical displacement of left cheek Vertical displacement of right cheek 7 E E E E E E E Horizontal MW displacement of tongue tip Vertical displacement of tongue tip Depth displacement of tongue tip N W forwar d i 48 head_pitch Head pitch angle AU down 7 from top of spine 49 head_yaw Head yaw angle AU left 7 from top of spine S S S Vertical displacement of tongue Rolling of the tongue into U shape concav e upward Head roll angle from top of spine 34 BAP grouping and masking dni ine nn K nen nin EE 170 1860 170 1860 na na a 170 1860 600 n 6 MPEG 4 FBA overview lower_t_midlip _o raise_b_midlip_o stretch 1 cornerlip stretch r cornerlip o lower_t_lip_Im _o lower t lip rm _o 7 im raise_b_lip_rm_o raise_l_cornerlip_o raise_r_cornerlip o ENS Vertical top middle outer lip displacement Vertical bott
21. _sternoclavicular_abduct _sternoclavicular_rotate 1_acromioclavicular_abduct _acromioclavicular_rotate 1_elbow_twisting I wrist pivot 1 wrist twisting 24 26 28 30 40 44 46 Right arm2 r_sternoclavicular_abduct r_sternoclavicular_rotate r_acromioclavicular_abduct r_acromioclavicular_rotate r_elbow_twisting r_wrist_pivot r_wrist_twisting 25 27 29 31 41 45 47 Spinel skullbase_roll skullbase_torsion skullbase_tilt vc4roll vc4torsion vc4tilt vt6roll vt6torsion vt6tilt vl3roll vl3torsion vl3tilt 48 49 50 60 61 62 87 88 89 114 115 116 21 7 Right arml r shoulder flexion r shoulder abduct r shoulder twisting r_elbow_flexion r_wrist_flexion 33 35 37 39 43 MPEG 4 FBA overview Left hand1 BAP grouping and masking ve2roll ve2torsion ve2tilt vtlroll vtltorsion vt1tilt vtlOroll vt10torsion vtlOtilt vilroll vlltorsion vl1tilt vl5roll vl5torsion vl5tilt 54 55 56 72 73 74 99 100 101 108 109 110 120 121 122 ve3roll ve3torsion ve3tilt vc6roll vc6torsion vc6tilt vt4roll vt4torsion vt4tilt vt8roll vt8torsion vt8tilt vt12roll vt12torsion vt12tilt vl4roll vl4torsion vl4tilt 57 58 59 66 67 68 8 1 82 83 93 94 95 105 106 107 117 118 119 ve5roll vcStorsion vcStilt ve7roll ve7torsion vc7tilt vt2roll vt2torsion vt2tilt vt7roll vt7torsion vt7tilt vtllroll vtl1torsion vtl 1 tilt vl2roll vl2torsion v12tilt 63 64 65 69 70 71 75 76 77 90 91 92
22. at version gt lt string sequence name gt lt int frame rate gt lt int number of frames gt BAP Mask lt int 296 BAP mask gt The BAP mask consists of 296 space separated 1 or 0 digits 1 indicates that a BAP is used in this particular frame O indicates that it is not used In the next line after the frame number values for all used BAPs are listed space separated Frame Number lt int frame number gt BAP Values lt int BAP values for used BAPs gt 61 An example BAP file MPEG 4 FBA overview 12 1 An example BAP file Here is the extract of the file an empty line space indicates a new line in order to distinguish from long lines that wrap Header description 3 2 lit a bap 10 44 00000000000 0000000000000000000001010i10 1010000012 0000000000000000000000000000 0 0000000000000000000000000000000000000 0 0oO000000000000000000000000000000000000 0 0O000000000000000000000000000000000000 0 0O000000000000000000000000000000000000 0 0oO000000000000000000000000000000000000 0 0000000000000000000000000000 0 0 0 61086 68067 64577 216420 31415 26179 0oO000000000000000000000000000000000000 0 0oO000000000000000000000000000000000000 0 0O000000000000000000000000000000000000 0 0O000000000000000000000000000000000000 0 0O0000000000000000007T00000000000000000 0 0oO000000000000000000000000000000000000 0 0oO000000000000000000000000000000000000 0 0000000000000000000000000000 0 0 I 31415 62 MPEG
23. e sequence Range 0 255 seconds This is a 4 bit unsigned integer indicating the fractional reference frame rate The frame rate is computed as follows frame rate frame_rate seconds 16 Range 0 15 frequency_offset This is a 1 bit flag which when set to 1 indicates that the frame rate uses the NTSC frequency offset of 1000 1001 This bit would typically be set when frame_rate 24 30 or 60 in which case the resulting frame rate would be 23 97 29 94 or 59 97 respectively When set to 0 no frequency offset is present I e if frequency offset 1 frame rate 1000 1001 frame rate seconds 16 Range 0 1 fap_quant This is a 5 bit unsigned integer which is the quantization scale factor used to compute the FAPi table step size Range 0 31 bap_quant This is a 5 bit unsigned integer which is the quantization scale factor used to compute the BAPi table step size Range 0 31 i_max This is an array of 73 integer values setting the maximum value that each FAP can take when an I frame is encoded If these values are not big enough i e if a FAP goes out of range while encoding results are unpredictable The first 7 values correspond to visemel viseme2 viseme blend expression expression intensity expression2 expression2 intensity The rest of the parameters correspond to the low level FAPs i_min This is an array of 73 integer values setting the minimum value that each FAP can take when an I
24. e of performing complicated motions and are included in the body hierarchy There are totally 29 degrees of freedom on each hand assuming that the hand has a standard structure with five fingers MPEG 4 FBA overview FAP grouping and masking 2 Face Animation Parameters FAPs MPEG 4 FBA overview FAP grouping and masking For the face the MPEG 4 specification defines 66 low level Face Animation Parameters FAPs and two high level FAPs The low level FAPs are based on the study of minimal facial actions and are closely related to muscle actions They represent a complete set of basic facial actions and therefore allow the representation of most natural facial expressions Exaggerated values permit the definition of actions that are normally not possible for humans but could be desirable for cartoon like characters All low level FAPs are expressed in terms of the Face Animation Parameter Units FAPUs listed in Table 1 and illustrated in Figure 1 These units are defined in order to allow interpretation of the FAPs on any face model in a consistent way producing reasonable results in terms of expression and speech pronunciation They correspond to distances between key facial features and are defined in terms of distances between the MPEG 4 facial Feature Points FPs see Figure 2 For each FAP it is defined on which FP it acts in which direction it moves and which FAPU is used as the unit for its movement For example FAP no 3 open jaw
25. e_twisting Rotation along the knee axis 256 960 360 960 360 44 P Nn 7 BA P ID MPEG 4 FBA overview BAP grouping and masking 1_subtalar_flexion Sideward orientation of the foot 600 780 600 r_subtalar_flexion Sideward orientation of the foot 256 780 600 ee ee eae called navicular joint in anatomy 180 4 4120 e 5 18 19 0 2 N N r_midtarsal_ twisting Internal twisting of the foot also 256 180 120 called navicular joint in anatomy 180 4120 1_metatarsal_flexion Up and down rotation of the toe 256 780 600 in the sagittal plane 780 600 r_metatarsal_flexion Up and down rotation of the toe 256 780 600 in the sagittal plane 780 4 4600 1_sternoclavicular_abduct Up and down motion in the 128 60 120 coronal plane 240 120 Oo Nn coronal plane 60 120 1_sternoclavicular_rotate Rotation in the transverse plane 128 120 60 120 60 7 r_sternoclavicular_rotate Rotation in the transverse plane 128 120 60 120 60 1_acromioclavicular_abduc Up and down motion in the 128 60 120 coronal plane 360 120 r_acromioclavicular_abdu Up and down motion in the 128 360 120 ct coronal plane 60 120 30 1_acromioclavicular_rotate Rotation in the transverse plane 128 360 120 360 120 Ne I Ir acromioclavicular rotate Rotation in the transverse plane 128 360 120 sagittal plane 1
26. ertebra T6 128 660 360 660 360 88 vt6_torsion Twisting of vertebra T6 128 660 360 660 360 Forward backward motion of 360 vertebra T6 in the sagittal plane 660 360 vt7_roll Sideward motion of vertebra T7 256 240 120 Aa en Dn Fe 91 vt7_torsion Twisting of vertebra T7 256 240 120 92 Forward backward motion of 120 vertebra T7 in the sagittal plane 240 120 93 vt8 roll Sideward motion of vertebra T8 256 240 120 94 vt8 torsion Twisting of vertebra T8 256 240 120 95 Forward backward motion of 120 vertebra T8 in the sagittal plane 240 120 vt9_roll Sideward motion of vertebra T9 256 240 120 97 vt9_torsion Twisting of vertebra T9 256 240 120 Forward backward motion of 120 vertebra T9 in the sagittal plane 240 120 vt_10roll Sideward motion of vertebra T10 128 660 360 100 vt10_torsion Twisting of vertebra T10 128 660 360 660 360 48 MPEG 4 FBA overview BAP grouping and masking 101 Forward backward motion of 128 vertebra T10 in sagittal plane 660 360 102 vt11_roll Sideward motion of vertebra T11 256 240 120 pees ee male 103 vt11_torsion Twisting of vertebra T11 256 240 120 104 Forward backward motion of 120 vertebra T11 in sagittal plane 240 120 105 vt12_roll Sideward motion of vertebra T12 256 240 120 106 vtl2_ torsion Twisting of vertebra T12 256 240 120 107 Forward backward motion of
27. exion r_index1_ pivot r_index1_twisting r_thumb1_twisting 124 128 130 136 140 142 148 152 154 160 164 166 176 Global HumanoidRoot_tr_vertical positioning HumanoidRoot_tr_lateral HumanoidRoot_tr_frontal HumanoidRoot_rt_body_turn HumanoidRoot rt body roll HumanoidRoot rt body tilt 181 182 183 184 185 186 Additionally groups 20 to 24 contain extension BAPs Extension BAPs are user defined BAPs Thus the extension BAP groups are as follows GROUP ID GROUP NAME BAPs 20 Extension1 187 208 21 Extension2 209 230 22 Extension3 231 252 23 Extension4 253 274 24 Extension5 275 296 23 MPEG 4 FBA overview BAP grouping and masking When BAPs are encoded into an FBA file bitstream a masking mechanism is used to select which BAPs are encoded and which are not The purpose of this mechanism is to achieve more efficient coding i e the BAPs that are not used do not need to be encoded BAPs are selected by using a two level mask hierarchy The first level is the mask type There is one mask type parameter for each group The mask type can be 0 1 or 3 The meaning of the values is as follows 0 no BAPs are encoded in the group 1 a group mask is set indicating which BAPs in the group are encoded 3 all BAPs in the group are encoded If mask type for a particular group is 1 a group mask must be defined for this group The group mask has one value for each BAP in the group So for e
28. g the shank axis This joint is redundant with the knee_twisting except that only the foot rotate and not the shank segment ankle_flexion flexion extension of the foot in the sagital plane At the foot complex The foot complex region is completely described with three degrees of freedom with independent position and orientation the subtalar joint the mid_foot joint between the subtalar joint and the toe joint to capture complex internal relative movement of the foot bones also called the navicular joint in the literature 14 MPEG 4 FBA overview Detailed body topology and degrees of freedom subtalar_flexion sideward orientation of the foot midtarsal_flexion internal twisting of the foot also called navicular joint in anatomy metatorsal_flexion up and down rotation of the toe in the sagital plane pelvic_torsion elvic_torsion pelvic_roll hip_abduct r hip flexion l hip flexion hip twisting Zoom on foot knee twisting Subtalar metatarsal r_metatarsal_ flexion flexion toe end r_ankle twisting l _ ankle_twisting toe base Heel midtarsal Figure 5 Front and side views of the mobilities of the leg Upper Body The suggested upper body degrees of freedom and their corresponding axes of rotation are shown in the following diagrams 15 MPEG 4 FBA overview Detailed body topology and degrees of freedom head_top ve2torsion ve2torsio cervical vc4tors
29. ion vc4torsio vtitorsion vtltorsio vt6torsion lombar_region skeleton_root skeleton_root Figure 6 Front and side views of the mobilities of the simple spine At the spine The vertebrae are dispatched into 5 lumbar 12 thoracic and cervical groups A total of 72 DOFs are defined for the spine therefore complex applications with whole spine mobilities can be developed However typically the application will use simpler spines that balance the computational speed with the realism of animation Therefore 5 groups of spine are defined from the simplest to most complex It is suggested that the spine groups Spinel Spine2 are used for simple spine 16 MPEG 4 FBA overview Detailed body topology and degrees of freedom At the arms The arm structure is attached to the spine The mobilities are similar to the ones defined for the leg when the arm is twisted such that the hand palm is facing backward The name of the DOF respects this structure similarity Strictly speaking the two clavicula DOF and the two scapula DOF are not part of the arm structure they only initiate its articulated chain The scapula joints improve the mechanical model of the shoulder complex It should be noted that such a chain representation is only a step toward a mechanically correct representation of this region mainly because the shoulder complex is in fact a closed loop mechanism The scapula holds the same mobility as the clavicula but very close
30. is 16 Each value corresponds to one FAP and can be 0 or 1 1 means that the FAP is encoded and used If the value is 0 the FAP is not encoded in MPEG 4 FBA overview FAP grouping and masking the bitstream but depending on the value of the mask type the decoder can either interpolate this FAP from other FAPs e g using left right symmetry or retain a previously set value for that FAP If FAPs are encoded from an ASCII FAP file the FAP masks are set in the Encoder Parameter File EPF The format of the EPF file is described in Annex C The EPF file format When visage SDK is used to encode FAPs the mask types and group masks are set in the FAPs class 10 MPEG 4 FBA overview FAP grouping and masking 3 Body Animation Parameters BAPs MPEG 4 FBA overview FAP grouping and masking For the body there are 186 Body Animation Parameters BAPs plus 110 extension user defined parameters which makes the total of 296 BAPs BAP parameters are the angles of rotation of body joints connecting different body parts These joints include toe ankle knee hip spine C1 C7 T1 T12 L1 L5 shoulder clavicle elbow wrist and the hand fingers The detailed parameters are given in Annex 4 MPEG 4 Body Animation Parameters The rotation angles are assumed to be positive in the counterclockwise rotation direction with respect to the rotation normal The unit of rotations BAPU is defined as 10 radians The unit of translation BAPs BA
31. king Min Max Min Max Quan I Frame P Frame BAP NAME DESCRIPTION quantized quantized values values sacroiliac_tilt Forward backward motion of the 64 960 600 pelvis in the sagittal plane 960 600 sacroiliac_torsion Rotation of the pelvis along the 64 960 600 body vertical axis defined by 960 600 skeleton root sacroiliac_roll Side to side swinging of the 64 960 600 pelvis in the coronal plane 960 600 _hip_flexion Forward backward rotation in the 128 1260 600 sagittal plane 1260 600 r_hip_flexion Forward backward rotation in the 128 1260 600 sagittal plane 1260 600 I hip abduct Sideward opening in the coronal 128 960 600 plane 480 600 r hip abduct Sideward opening in the coronal 128 960 600 plane 480 600 I hip twisting Rotation along the thigh axis 256 960 360 960 360 r_hip_twisting Rotation along the thigh axis 256 360 960 360 1_knee_flexion 128 600 the sagittal plane 180 4 600 r_knee_flexion 128 600 the sagittal plane 180 4 600 I knee twisting Rotation along the shank axis 256 960 360 960 360 NN N r_knee_twisting Rotation along the shank axis 256 960 360 960 360 1_ankle_ flexion 128 600 the sagittal plane 780 600 r_ankle_ flexion 128 600 the sagittal plane 780 600 I ankle twisting Rotation along the knee axis 256 960 360 960 360 r_ankl
32. long the frontal axis 1860 600 49 skullbase_torsion Twisting of the skull along the 128 1860 600 vertical axis 1860 600 50 skullbase_tilt Forward backward motion in the 128 1860 600 sagittal plane along a lateral axis 1860 4 600 51 vcl_roll Sideward motion of vertebra C1 256 240 120 240 120 52 vcl_torsion Twisting of vertebra C1 256 240 120 240 120 53 vcl_tilt Forward backward motion of 256 240 120 vertebra C1 in the sagittal plane 240 120 54 vc2_roll Sideward motion of vertebra C2 128 660 360 660 360 55 vc2_torsion Twisting of vertebra C2 128 660 360 660 360 56 vc2_tilt Forward backward motion of 128 660 360 vertebra C2 in the sagittal plane 660 360 57 ve3 roll Sideward motion of vertebra C3 256 240 120 240 120 58 vc3_torsion Twisting of vertebra C3 256 240 120 240 120 46 MPEG 4 FBA overview BAP grouping and masking vertebra C3 in the sagittal plane 240 120 vc4_roll Sideward motion of vertebra C4 128 660 360 660 360 61 vc4_torsion Twisting of vertebra C4 128 660 360 660 360 62 Forward backward motion of 360 vertebra C4 in the sagittal plane 660 360 63 vc5_roll Sideward motion of vertebra C5 256 240 120 64 vc5_torsion Twisting of vertebra C5 256 240 120 65 Forward backward motion of 120 vertebra C5 in the sagittal plane 240 120 vc6 roll Sideward motion of vertebra C6 256
33. of the pinky 128 420 120 finger 120 120 128 Ir pinkyl pivot Lateral mobility of the pinky 128 120 120 finger 420 120 129 1_pinky1_twisting Along the pinky finger axis 256 180 120 360 120 130 r_pinky1_twisting Along the pinky finger axis 256 360 120 180 120 131 1_pinky2_flexion Second knuckle of the pinky 128 1380 240 number 60 240 132 r_pinky2_flexion Second knuckle of the pinky 128 1380 240 133 1_pinky3_flexion Third knuckle of the pinky finger 128 960 240 60 240 134 r_pinky3_flexion Third knuckle of the pinky finger 128 960 240 60 240 ring finger 180 120 the ring finger 180 120 137 l ringl flexion First knukle of the ring finger 128 1140 240 Lene ee D lilo 138 r_ring1_flexion First knukle of the ring finger 128 1140 240 360 240 139 II ringl pivot Lateral mobility of the ring finger 128 240 120 120 120 140 Ir ringl pivot Lateral mobility of the ring finger 128 120 120 240 120 141 l ringl_twisting Along the ring finger axis 256 240 120 peas erener oo ao 142 r_ringl_twisting Along the ring finger axis 256 240 120 240 120 50 MPEG 4 FBA overview BAP grouping and masking 143 lring2_flexion Second knuckle of the ring 128 1380 240 number 60 240 144 Ir ring2 flexion Second knuckle of the ring 128 1380 240 fone frame EBs 7 145 1_ring3_flexi
34. om middle outer lip displacement Horizontal displacement of left outer lip corner Horizontal displacement of right outer lip corner Vertical displacement of midpoint between left corner and middle of top outer lip Vertical displacement of midpoint between right corner and middle of top outer lip Vertical displacement of midpoint between left corner and middle of bottom outer lip Vertical displacement of midpoint between right corner and middle of bottom outer lip Vertical displacement of left outer lip corner Vertical displacement of right outer lip corner BAP grouping and masking E i i 180 MPEG 4 FBA overview BAP grouping and masking displacement of left side of nose stretch_r_nose Horizontal right displacement of 2 1 right side of nose 63 raise_nose Vertical u 3 1 displacement of nose tip 64 bend_nose Horizontal right 3 1 900 180 displacement of nose tip i l raise r ear Vertical displacement of right ear 67 pull_l_ear Horizontal displacement of left ear pull r ear Horizontal displacement of right ear Pp E B ne we oe 65 raise_l_ear Vertical ENS displacement of left ear ne we T 36 MPEG 4 FBA overview BAP grouping and masking 7 Annex 2 MPEG 4 Visemes 37 MPEG 4 FBA overview BAP grouping and masking Viseme Viseme Some Example Visual example no name corresponding phonemes 0 sil none N A
35. on Third knuckle of the ring finger 128 960 240 60 240 146 r_ring3_flexion Third knuckle of the ring finger 128 960 240 60 240 middle finger 4120 4120 the middle finger 120 120 149 middlel_flexion First knukle of the middle finger 128 1140 240 360 240 150 r_middlel_flexion First knukle of the middle finger 128 1140 240 360 240 finger 180 120 finger 180 120 180 4120 180 4120 155 1_middle2_ flexion Second knuckle of the middle 128 1380 240 number 60 240 156 r_middle2_ flexion Second knuckle of the middle 128 1380 240 number 60 240 157 1_middle3_flexion Third knuckle of the middle 128 960 240 finger 60 240 158 r_middle3_ flexion Third knuckle of the middle 128 960 240 index finger 460 460 the index finger 60 60 360 240 360 240 163 1_index1_pivot Lateral mobility of the index 128 120 60 finger 240 60 51 MPEG 4 FBA overview BAP grouping and masking 164 Ir indexl pivot Lateral mobility of the index 128 240 60 finger 120 60 180 120 240 120 167 l index2_flexion Second knuckle of the index 128 1380 240 number 60 240 168 r_index2_ flexion Second knuckle of the index 128 1380 240 number 60 240 169 1_index3_flexion Third knuckle of the index finger 128 960 240 60 240 170 r_index3_ flexion Third knuckle of the index finger 128 960 240 60 240 171 1 thumb1_ flexion Fir
36. oulder_ abduct r_shoulder_abduc r_shoulder_twisting _elbow_flexion r_elbow_flexion _elbow_twisting r_elbow_twisting ve7torsion _wrist_twisting r_wrist_twisting _wrist_flexion r_wrist_flexion _wrist_pivot hand root r wrist pivot n i O Z Q 0 2 0 x OO pan I lt lt lt o gt fo c a 2 z Q 3 skullbaseroll iskullbasetorsion skullbasetilt Figure 4 Body topology MPEG 4 FBA overview Detailed body topology and degrees of freedom Lower Body From the skeleton_root node three DOF allow flexible pelvic orientation followed by nine DOF per leg from the hip joint to the toe joint First the pelvic mobilities are very important to convey a gender personification to the motion The naming convention of these mobilities is also used for the spine DOF The degrees of freedom are sacroiliac_tilt forward backward motion in the sagital plane sacroiliac _torsion rotation along the body vertical axis defined by skeleton root sacroiliac roll side to side swinging in the coronal plane The leg mobilities follow in this order At the hip hip_flexion forward backward rotation in the sagital plane hip_abduct sideward opening in the coronal plane hip_twisting rotation along the thigh axis At the knee knee_flexion flexion extension of the leg in the sagital plane knee_twisting rotation along the shank axis At the ankle ankle_twisting rotation alon
37. s are bap i min This is an array of 296 integer values setting the minimum value that each BAP can take when an I frame is encoded If these values are not low enough i e if a BAP goes out of range while encoding results are unpredictable bap p max This is an array of 296 integer values setting the maximum value that each BAP can take when an P frame is encoded If these values are not big enough i e if a BAP goes out of range while encoding results are unpredictable bap_p_min This is an array of 296 integer values setting the minimum value that each BAP can take when an P frame is encoded If these values are not low enough i e if a BAP goes out of range while encoding results are unpredictable 27 MPEG 4 FBA overview BAP grouping and masking 5 References 28 MPEG 4 FBA overview BAP grouping and masking ISO14496 ISO IEC 14496 MPEG 4 International Standard Moving Picture Experts Group www cselt it mpeg Pandzic02 MPEG 4 Facial Animation The standard implementations applications Igor S Pandzic R Forchheimer Editors John Wiley amp Sons Ltd 29 MPEG 4 FBA overview BAP grouping and masking 6 Annex 1 MPEG 4 Facial Animation Parameters 30 MPEG 4 FBA overview BAP grouping and masking MPEG 4 FAP definitions group assignments and step sizes FAP names may contain letters with the following meaning left r right t top b bottom i inner 0 outer m middle The quan
38. semes and expressions 2 2 jaw chin inner lowerlip cornerlips midlip 16 3 eyeballs pupils eyelids 12 4 eyebrow 8 5 cheeks A 6 tongue 5 7 head rotation 3 8 outer lip positions 10 9 nose 4 10 ears 4 The table of FAPs in Annex 1 MPEG 4 Facial Animation Parameters indicates for each FAP to which group it belongs When FAPs are encoded into an FBA file bitstream a masking mechanism is used to select which FAPs are encoded and which are not and how the decoder shall behave when decoding the bitstream The purpose of this mechanism is to achieve more efficient coding i e the FAPs that are not used do not need to be encoded FAPs are selected by using a two level mask hierarchy The first level is the mask type There is one mask type parameter for each group The mask type can be 0 1 2 or 3 The meaning of the values is as follows 0 no FAPs are encoded in the group 1 a group mask is set indicating which FAPs in the group are encoded FAPs not selected by the group mask retain their previous value 2 a group mask is set indicating which FAPs in the group are encoded FAPs not selected by the group mask retain must be interpolated by the decoder 3 all FAPs in the group are encoded If mask type for a particular group is 1 or 2 a group mask must be defined for this group The group mask has one value for each FAP in the group So for example the length of the group mask for group is 2 for group 2 it
39. st knukle of the thumb finger 128 480 240 960 240 172 r_thumb1_flexion First knukle of the thumb finger 128 960 240 480 240 173 1_thumb1_pivot Lateral mobility of the thumb 128 120 240 finger 1080 240 174 r_thumb1_pivot Lateral mobility of the thumb 128 1080 240 a a DF Pe 175 1_thumb1_twisting Along the thumb finger axis 256 720 240 120 240 176 r_thumb1_twisting Along the thumb finger axis 256 720 240 120 240 177 l thumb2_flexion Second knuckle of the thumb 128 780 240 number 60 240 178 r_thumb2_flexion Second knuckle of the thumb 128 780 240 number 60 240 179 thumb3_ flexion Third knuckle of the thumb 128 960 240 180 r_thumb3_flexion Third knuckle of the thumb 128 960 240 finger 60 240 direction 1860 600 direction 1860 600 direction 1860 600 52 MPEG 4 FBA overview 184 HumanoidRoot_rt_body_t HumanoidRoot_rt_body_r oll HumanoidRoot_rt_body_ti It Rotation of the skeleton root along the body coordinate system s vertical axis Rotation of the skeleton root along the body coordinate system s frontal axis Rotation of the skeleton root along the body coordinate system s lateral axis 53 BAP grouping and masking MPEG 4 FBA overview BAP grouping and masking 10 Annex 5 Suggested body joint center positions 54 MPEG 4 FBA overview BAP grouping and masking These joint centre positions may be
40. ter Units FAPU The neutral position of the face when all FAPs are 0 is defined as follows e the coordinate system is right handed head axes are parallel to the world axes e gaze is in direction of Z axis e all face muscles are relaxed e eyelids are tangent to the iris e the pupil is one third of IRISDO e lips are in contact the line of the lips is horizontal and at the same height of lip corners e the mouth is closed and the upper teeth touch the lower ones e the tongue is flat horizontal with the tip of tongue touching the boundary between upper and lower teeth feature point 6 1 touching 9 11 see Figure 2 All FAPs are expressed as displacements from the positions defined in the neutral face Beside the standard itself SO14496 there are other excellent references Pandzic02 covering the subject of MPEG 4 Face Animation and we recommend more serious users to obtain these MPEG 4 FBA overview FAP grouping and masking YA ki y Nose ee pli 8 8 lt Pe ae Tongue Mouth 82 Feature points affected by FAPs Other feature points Figure 2 Facial Feature Points FP 2 1 MPEG 4 FBA overview FAP grouping and masking FAP grouping and masking This section is interesting only for users who wish to understand more about the coding of FAPs in the FBA files bitstreams FAPs are grouped into 10 groups as shown in this table Group Number of FAPs in the group 1 vi
41. tizer step size is a scaling factor for coding FAP name FAP description units FDP Qua Min Ma Min M r i p rp num quantiz ed values viseme Set of values na 1 i _ viseme_ determining the blend mixture of two 63 visemes for this frame e g pbm fv th expression A set of values expressi expressi determining the on_inten on inte mixture of two sityl nsityl facial expression expressi expressi on_inten on inte sity2 nsity2 63 3 open_jaw Vertical jaw displacement does not affect mouth opening lower t midlip Vertical top middle inner lip displacement raise b midlip Vertical bottom middle inner lip displacement stretch 1 cornerlip Horizontal displacement of left inner lip corner stretch r cornerlip Horizontal displacement of right inner lip corner lower t lip Im Vertical displacement of midpoint between left corner and middle of top inner lip 31 MPEG 4 FBA overview lower_t_lip_rm raise b lip Im raise b lip rm raise_l_cornerlip raise_r_cornerlip push_b_lip push_t_lip close_t_l_eyelid close tr eyelid Vertical displacement of midpoint between right corner and middle of top inner lip Vertical displacement of midpoint between left corner and middle of bottom inner lip Vertical displacement of midpoint between right corner and middle of bottom inner lip Vertical displacement of left inner lip corner Vertical displacement of right inner lip
42. ure for the five fingers This structure is organized as a flexing DOF for closing the first knuckle 18 MPEG 4 FBA overview Detailed body topology and degrees of freedom a pivoting rotation for the lateral mobility of the finger a twisting rotation for small adjustments of the finger grasping orientation two other flexing DOF for closing the second and third knuckles Figure 8 Mobilities of the right hand 19 MPEG 4 FBA overview Detailed body topology and degrees of freedom C2 C4 T1 T6 T10 Li L3 LS Figure 9 Example spine mobilities 20 MPEG 4 FBA overview BAP grouping and masking 3 2 BAP grouping and masking This section is interesting only for users who wish to understand more about the coding of BAPs in the FBA files bitstreams BAPs are grouped into 24 groups as shown in this table GROUP ID GROUP NAME BAPS 1 I hip flexion I hip abduct I knee flexion I ankle flexion 4 6 10 14 Right leg r_hip_flexion r_hip_abduct r_knee_flexion r_ankle_flexion 5 7 11 15 I hip twisting I knee twisting _ankle_twisting 1_subtalar_flexion 1_midtarsal_ flexion 1_metatarsal_ flexion 8 12 16 18 20 22 Right leg2 r_hip_twisting r_knee_twisting r_ankle_twisting r_subtalar_flexion r_midtarsal_flexion r_metatarsal_flexion 9 13 17 19 21 23 Left arm1 1_shoulder_flexion I shoulder abduct I shoulder twisting I elbow flexion I wrist flexion 32 34 36 38 42 Left arm2 1
43. used to construct a default body HumanoidRoot 0 0000 0 9723 0 0728 sacroiliac 0 0000 0 9710 0 0728 I hip 0 0956 0 9364 0 0000 I knee 0 0956 0 5095 0 0036 I ankle 0 0946 0 0762 0 0261 _subtalar 0 0956 0 0398 0 0069 _midtarsal 0 1079 0 0317 0 0670 I metatarsal 0 0942 0 0092 0 1239 r_hip 0 0956 0 9364 0 0000 r_knee 0 0956 0 5095 0 0036 r_ankle 0 0946 0 0762 0 0261 r_subtalar 0 0956 0 0398 0 0069 r_midtarsal 0 1079 0 0317 0 0670 r_metatarsal 0 0942 0 0092 0 1239 vI5 0 0000 1 0817 0 0728 vl4 0 0000 1 1174 0 0727 vI3 0 0000 1 1525 0 0727 vl2 0 0000 1 1795 0 0727 vli 0 0000 1 2161 0 0727 vt12 0 0000 1 2527 0 0727 vt11 0 0000 1 2918 0 0727 vt10 0 0000 1 3098 0 0737 vt9 0 0000 1 3375 0 0752 vt8 0 0000 1 3631 0 0758 vt7 0 0000 1 3875 0 0745 vt6 0 0000 1 4116 0 0712 vt5 0 0000 1 4351 0 0657 vt4 0 0000 1 4569 0 0587 vt3 0 0000 1 4832 0 0482 vt2 0 0000 1 5011 0 0397 vt1 0 0000 1 5201 0 0300 vc7 0 0000 1 5382 0 0213 vc6 0 0000 1 5607 0 0073 vc5 0 0000 1 5770 0 0012 vc4 0 0000 1 5912 0 0014 vc3 0 0000 1 6050 0 0033 vc2 0 0000 1 6178 0 0033 vci 0 0000 1 6394 0 0036 skullbase 0 0000 1 6440 0 0036 _sternoclavicular 0 0757 1 4844 0 0251 _acromioclavicular 0 0899 1 4525 0 0322 _shoulder 0 1968 1 4642 0 0265 I elbow 0 1982 1 1622 0 0557 _wrist 0 197
44. visage technologies MPEG 4 Face and Body Animation MPEG 4 FBA An overview Visage Technologies AB Tegskiftesgatan 291 SE 58334 Link ping Sweden www visagetechnologies com MPEG 4 FBA overview TABLE OF CONTENTS 1 Introduction 2 Face Animation Parameters FAPs 2 1 FAP grouping and masking 3 Body Animation Parameters BAPs 3 1 Detailed body topology and degrees of freedom 3 2 BAP grouping and masking 4 Coding parameters 5 References 6 Annex 1 MPEG 4 Facial Animation Parameters 7 Annex 2 MPEG 4 Visemes 8 Annex 3 MPEG 4 FBA primary expressions 9 Annex 4 MPEG 4 Body Animation Parameters 10 Annex 5 Suggested body joint center positions 11 Annex A The FAP file format 11 1 An example FAP file 12 Annex B The BAP file format 12 1 An example BAP file 13 Annex C The EPF file format 13 1 An example EPF file FAP grouping and masking 11 13 21 25 28 30 37 41 43 54 57 59 60 62 63 65 1 Introduction MPEG 4 FBA overview FAP grouping and masking The purpose of this document is to provide background knowledge on MPEG 4 Face and Body Animation MPEG 4 FBA ISO14496 International Standard for the users of Visage Technologies software It provides the details on the Face Animation Parameters FAPs and Body Animation Parameters BAPs Casual users or developers using only the high level interfaces of the visage SDK typically do not need to be familiar with details of MPEG 4 FBA so they ma
45. xample the length of the group mask for group 1 is 3 for group 2 it is 4 etc Each value corresponds to one BAP and can be 0 or 1 1 means that the BAP is encoded If the value is 0 the BAP is not encoded in the bitstream If BAPs are encoded from an ASCII BAP file the BAP masks are set in the Encoder Parameter File EPF The format of the EPF file is described in Annex C The EPF file format When visage SDK is used to encode BAPs the mask types and group masks are set in the BAPs class 24 MPEG 4 FBA overview BAP grouping and masking 4 Coding parameters 25 MPEG 4 FBA overview BAP grouping and masking When FAPs and BAPs are encoded into the bitstream file several additional parameters are used to control the encoding and later decoding process visage SDK and visagelinteractive use default values which are appropriate for most purposes Most of the time users will not need to change or even understand these parameters If necessary they can be set through appropriate methods in visage SDK or in case of encoding from an ASCII FAP file the parameters are set in the EPF file The main purpose of tweaking these parameters is to achieve better coding efficiency i e smaller files or lower bitrates Considerable improvements over default parameters can be achieved particularly in special cases The coding parameters are the following frame_rate This is an 8 bit unsigned integer indicating the reference frame rate of th
46. y skip this document The users wishing to explore the advanced functionalities will find detailed information in this document to support them For the face the MPEG 4 specification defines 66 low level Face Animation Parameters FAPs and two high level FAPs The low level FAPs are based on the study of minimal facial actions and are closely related to muscle actions They represent a complete set of basic facial actions and therefore allow the representation of most natural facial expressions Exaggerated values permit the definition of actions that are normally not possible for humans but could be desirable for cartoon like characters The detailed parameters are given in Annex 1 MPEG 4 Facial Animation Parameters Annex 2 MPEG 4 Visemes and Annex 3 MPEG 4 FBA primary expressions MPEG 4 also defines a standard set of facial feature points Figure 2 on page 8 that can be used for any application that needs well defined facial landmarks For the body there are 196 Body Animation Parameters BAPs BAP parameters are the angles of rotation of body joints connecting different body parts These joints include toe ankle knee hip spine C1 C7 T1 T12 L1 L5 shoulder clavicle elbow wrist and the hand fingers The detailed parameters are given in Annex 4 MPEG 4 Body Animation Parameters The rotation angles are assumed to be positive in the counterclockwise rotation direction with respect to the rotation normal The hands are capabl
Download Pdf Manuals
Related Search