Diffusion-based Human Motion Style Transfer with Semantic Guidance

Simple item page
dc.contributor.authorHu, Leien_US
dc.contributor.authorZhang, Zihaoen_US
dc.contributor.authorYe, Yongjingen_US
dc.contributor.authorXu, Yiwenen_US
dc.contributor.authorXia, Shihongen_US
dc.contributor.editorSkouras, Melinaen_US
dc.contributor.editorWang, Heen_US
dc.date.accessioned2024-08-20T08:42:10Z
dc.date.available2024-08-20T08:42:10Z
dc.date.issued2024
dc.description.abstract3D Human motion style transfer is a fundamental problem in computer graphic and animation processing. Existing AdaINbased methods necessitate datasets with balanced style distribution and content/style labels to train the clustered latent space. However, we may encounter a single unseen style example in practical scenarios, but not in sufficient quantity to constitute a style cluster for AdaIN-based methods. Therefore, in this paper, we propose a novel two-stage framework for few-shot style transfer learning based on the diffusion model. Specifically, in the first stage, we pre-train a diffusion-based text-to-motion model as a generative prior so that it can cope with various content motion inputs. In the second stage, based on the single style example, we fine-tune the pre-trained diffusion model in a few-shot manner to make it capable of style transfer. The key idea is regarding the reverse process of diffusion as a motion-style translation process since the motion styles can be viewed as special motion variations. During the fine-tuning for style transfer, a simple yet effective semantic-guided style transfer loss coordinated with style example reconstruction loss is introduced to supervise the style transfer in CLIP semantic space. The qualitative and quantitative evaluations demonstrate that our method can achieve state-of-the-art performance and has practical applications. The source code is available at https://github.com/hlcdyy/diffusion-based-motion-style-transfer.en_US
dc.description.number8
dc.description.sectionheadersCharacter Animation I: Synthesis and Capture
dc.description.seriesinformationComputer Graphics Forum
dc.description.volume43
dc.identifier.doi10.1111/cgf.15169
dc.identifier.issn1467-8659
dc.identifier.pages12 pages
dc.identifier.urihttps://doi.org/10.1111/cgf.15169
dc.identifier.urihttps://diglib.eg.org/handle/10.1111/cgf15169
dc.publisherThe Eurographics Association and John Wiley & Sons Ltd.en_US
dc.subjectCCS Concepts: Computing methodologies → Motion processing; Artificial intelligence
dc.subjectComputing methodologies → Motion processing
dc.subjectArtificial intelligence
dc.titleDiffusion-based Human Motion Style Transfer with Semantic Guidanceen_US
Files
Original bundle
Now showing 1 - 3 of 3
No Thumbnail Available
Name:
v43i8_cgf15169.pdf
Size:
15.93 MB
Format:
Adobe Portable Document Format
Download
No Thumbnail Available
Name:
paper1009-supp.pdf
Size:
5.85 MB
Format:
Adobe Portable Document Format
Download
No Thumbnail Available
Name:
paper1009.mp4
Size:
55.59 MB
Format:
Video MP4
Download
Collections
43-Issue 8