Style-Controllable Speech-Driven Gesture Synthesis Using Normalising Flows

dc.contributor.authorAlexanderson, Simonen_US
dc.contributor.authorHenter, Gustav Ejeen_US
dc.contributor.authorKucherenko, Tarasen_US
dc.contributor.authorBeskow, Jonasen_US
dc.contributor.editorPanozzo, Daniele and Assarsson, Ulfen_US
dc.date.accessioned2020-05-24T12:53:57Z
dc.date.available2020-05-24T12:53:57Z
dc.date.issued2020
dc.description.abstractAutomatic synthesis of realistic gestures promises to transform the fields of animation, avatars and communicative agents. In off-line applications, novel tools can alter the role of an animator to that of a director, who provides only high-level input for the desired animation; a learned network then translates these instructions into an appropriate sequence of body poses. In interactive scenarios, systems for generating natural animations on the fly are key to achieving believable and relatable characters. In this paper we address some of the core issues towards these ends. By adapting a deep learning-based motion synthesis method called MoGlow, we propose a new generative model for generating state-of-the-art realistic speech-driven gesticulation. Owing to the probabilistic nature of the approach, our model can produce a battery of different, yet plausible, gestures given the same input speech signal. Just like humans, this gives a rich natural variation of motion. We additionally demonstrate the ability to exert directorial control over the output style, such as gesture level, speed, symmetry and spacial extent. Such control can be leveraged to convey a desired character personality or mood. We achieve all this without any manual annotation of the data. User studies evaluating upper-body gesticulation confirm that the generated motions are natural and well match the input speech. Our method scores above all prior systems and baselines on these measures, and comes close to the ratings of the original recorded motions. We furthermore find that we can accurately control gesticulation styles without unnecessarily compromising perceived naturalness. Finally, we also demonstrate an application of the same method to full-body gesticulation, including the synthesis of stepping motion and stance.en_US
dc.description.number2
dc.description.sectionheadersSynthesizing Gestures, Motion, and Interactions
dc.description.seriesinformationComputer Graphics Forum
dc.description.volume39
dc.identifier.doi10.1111/cgf.13946
dc.identifier.issn1467-8659
dc.identifier.pages487-496
dc.identifier.urihttps://doi.org/10.1111/cgf.13946
dc.identifier.urihttps://diglib.eg.org:443/handle/10.1111/cgf13946
dc.publisherThe Eurographics Association and John Wiley & Sons Ltd.en_US
dc.rightsAttribution 4.0 International License
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.subjectComputing methodologies
dc.subjectMotion capture
dc.subjectAnimation
dc.subjectNeural networks
dc.titleStyle-Controllable Speech-Driven Gesture Synthesis Using Normalising Flowsen_US
Files
Original bundle
Now showing 1 - 5 of 8
Loading...
Thumbnail Image
Name:
v39i2pp487-496.pdf
Size:
9.05 MB
Format:
Adobe Portable Document Format
No Thumbnail Available
Name:
erratum.zip
Size:
12.59 MB
Format:
Zip file
No Thumbnail Available
Name:
evaluation1.zip
Size:
59.8 MB
Format:
Zip file
No Thumbnail Available
Name:
evaluation1_style.zip
Size:
42.1 MB
Format:
Zip file
No Thumbnail Available
Name:
evaluation2_fullbody.zip
Size:
30.28 MB
Format:
Zip file
Collections