Now you see it, now you don’t - frequency distribution of articulatory information reflected in speech face motion

Abstract

Although the increase in intelligibility of spoken language when watching the speaker’s face is well documented, the characteristics and the distribution of phonetic information over the frequency range of visual speech remain largely unknown. For this study face motion and tongue movements were measured simultaneously for three speakers of American English. The motion signals were subjected to a multiresolution analysis using spline wavelets and partial least squares regression was applied to estimate tongue movements from face motion. The amount of recovered variance was found to be small (below 30%) compared to previous studies and more pronounced in the lower frequencies.