.png)
November 21, 2024
The animation of images, particularly portraits, using audio tracks has made significant progress in recent years. However, the realistic representation of facial expressions and head movements, synchronized with the audio input, remains a challenge. While current diffusion-based models deliver impressive results in terms of video quality and lip synchronization, they often struggle with limitations regarding video length, inter-frame continuity, and computational cost. JoyVASA, a novel diffusion-based method, promises to remedy this.
JoyVASA generates facial dynamics and head movements for audio-driven facial animation in a two-stage process. In the first stage, a decoupled facial representation framework is used, which separates dynamic facial expressions from static 3D facial representations. This decoupling allows for the creation of longer videos by combining any static 3D facial representation with dynamic motion sequences. The second stage involves a diffusion transformer trained to generate motion sequences directly from audio signals, regardless of the character's identity. A generator trained in the first stage then uses the 3D facial representation and the generated motion sequences to render high-quality animations.
JoyVASA's innovative approach allows for the animation of both human portraits and animal faces. Through decoupled facial representation and identity-independent motion generation, the system can seamlessly switch between different species. The model was trained on a hybrid dataset of private Chinese and public English data, enabling multilingual support.
JoyVASA's architecture offers several advantages over existing methods. The separation of static facial representation and dynamic motion allows for greater flexibility and control over the animation. Identity-independent motion generation simplifies the process and enables application to various characters. The use of diffusion transformers contributes to improved video quality and lip synchronization accuracy. Experimental results confirm the effectiveness of the approach and show promising results regarding the realism and fluidity of the animations.
The developers of JoyVASA plan to further improve the model's real-time performance and refine the control of facial expressions. This opens a wide range of application possibilities, including virtual avatars, animated films, video games, and interactive applications. The ability to animate static images with high accuracy and expressiveness could revolutionize how we interact with digital content. Especially for companies like Mindverse, which specialize in AI-powered content creation, JoyVASA offers a valuable tool for expanding their offerings and enhancing the customer experience. Integrating JoyVASA into Mindverse's all-in-one content platform could significantly simplify the creation of personalized and dynamic content and open up new possibilities for developing chatbots, voicebots, and AI search engines.
Bibliographie Cao, Xuyang, et al. "JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generation." arXiv preprint arXiv:2411.09209 (2024). jdh-algo/JoyVASA. GitHub. jdh-algo. JoyVASA. GitHub Pages. r/StableDiffusion. "JoyVASA Portrait and Animal Image Animation with." Reddit. aimodels.fyi. "JoyVASA: Portrait and Animal Image Animation Diffusion-Based." CSVisionPapers. "JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generation." X. fudan-generative-vision/hallo. GitHub. camenduru. "JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generation." X.