The aim of this research is to analyse and employ visual movements of facial features for the purpose of lip-reading and non-verbal communication (NVC) analysis. This starts by producing a method for efficiently and robustly tracking facial features. These include obvious parts of the face such as the eye shape, eyebrows and lip shapes (both inner and outer lips). Some of these features can be very difficult to track, for example points on the inner lip contour, due to the large amounts of visual variations that occur when a mouth opens and closes. Additionally, other features, such as the eye shape can exhibit extremely fast motions, such as blinks. To tackle the above issues, a novel technique based on selected linear predictors was proposed. Specifically a learnt person-specific but importantly, a data-driven approach to achieve accurate and real-time tracking of facial features using only intensity information.
This facial feature tracking has then been used to create automatic recognition of non-verbal communication (NVC). NVC is used in every day conversations to compliment the spoken words we use. An automatic NVC system may be useful for user product evaluations, computer game characters and learning tools. We also use unconstrained spontaneous conversations to make the data more applicable outside the lab. Annotation of videos was conducted using public Internet worker pools (including Amazon Mechanical Turk, Samasource) to collect NVC perception data from multiple cultures. This data is publicly available as the TwoTalk corpus. This enables us to investigate cultural differences in NVC perception and to create culturally specialized NVC recognition systems.
Tracking Facial Features