Gamasutra

If you missed SIGGRAPH 2017 watch a selection of recorded Live Streaming Sessions.

You are here

Technical Papers

Speech and Facial Animation

Renderosity

3D Artist

Wednesday, 2 August, 9:00 am - 10:30 am, Los Angeles Convention Center, Los Angeles Convention Center - Room 153
Session Chair: Nikunj Raghuvanshi, Microsoft Research

A Deep Learning Approach for Generalized Speech Animation

This simple and effective deep-learning approach for automatically generating natural-looking speech animation that synchronizes to input speech runs in real time, is easily edited to create stylized and emotional speech, and generalizes well to novel input speech sequences including singing and foreign languages.

Sarah Taylor
University of East Anglia

Taehwan Kim
California Institute of Technology

Moshe Mahler
Disney Research

James Krahe
Disney Research

Anastasio Garcia Rodriguez
Disney Research

Jessica Hodgins
Carnegie Mellon University

Yisong Yue
California Institute of Technology

Iain Matthews
Disney Research

Audio-Driven Facial Animation by Joint End-to-End Learning of Pose and Emotion

A machine-learning technique for driving 3D facial animation by audio input in real time with low latency. This study's primary goal is to model the speaking style of a single actor, but the technique yields reasonable results even for speakers of different gender with different accents or languages.

Tero Karras
NVIDIA Corporation

Timo Aila
NVIDIA Corporation

Samuli Laine
NVIDIA Corporation

Antti Herva
Remedy Entertainment Ltd.

Jaakko Lehtinen
Aalto University, NVIDIA Corporation

Synthesizing Obama: Learning Lip Sync From Audio

Given audio of President Barack Obama, this work synthesizes photorealistic video of him speaking with accurate lip sync. Trained on many hours of video footage, the recurrent neural-net approach synthesizes mouth shape and texture from audio, which are composited into a reference video.

Supasorn Suwajanakorn
University of Washington

Steven Seitz
University of Washington

Ira Kemelmacher
University of Washington

VoCo: Text-Based Insertion and Replacement in Audio Narration

Editing audio narrations using conventional software typically involves many painstaking low-level manipulations. This paper presents a system for synthesizing a new word or short phrase such that it blends seamlessly in the context of an existing narration. The system supports fully automatic synthesis as well as simple editing interfaces.

Zeyu Jin
Princeton University

Gautham Mysore
Adobe Research

Stephen DiVerdi
Adobe Research

Jingwan Lu
Adobe Research

Adam Finkelstein
Princeton University