.png)
December 9, 2024
The world of image editing is experiencing rapid development, driven by advances in artificial intelligence. Text-based image editing, which allows users to modify images through simple text input, is at the center of this innovation. This technology leverages the extensive capabilities of multi-step, diffusion-based text-to-image models. However, these methods often reach their limits when it comes to the speed requirements for real-time applications and use on mobile devices. The multi-step inversion and sampling process is often too computationally intensive.
SwiftEdit addresses precisely this challenge. The new image editing tool enables text-guided image editing in real-time – in just 0.23 seconds. This speed is achieved through two innovative core components: a single-step inversion framework that enables image reconstruction in just one step, and a mask-guided editing technique with a novel attention scaling mechanism to perform local image edits.
The inversion of single-step diffusion models is complex. Conventional techniques such as DDIM inversion or null-text inversion are unsuitable for real-time editing. SwiftEdit therefore relies on a novel framework inspired by encoder-based GAN inversion methods. Unlike GAN inversion, which requires domain-specific networks and retraining, the SwiftEdit framework can be generalized to arbitrary input images. It utilizes SwiftBrushv2, a well-known single-step text-to-image model, valued for its speed, diversity, and quality. SwiftBrushv2 serves both as a generator and as the basis for the inversion network. The network is trained with a two-stage strategy using both synthetic and real data.
After the single-step inversion, an efficient mask-based editing technique is employed. SwiftEdit can either use a predefined editing mask or derive it directly from the trained inversion network and the text input. The mask is then used in a novel attention scaling mechanism to control the editing strength while preserving background elements. This results in high-quality editing results.
SwiftEdit is the first tool to combine diffusion-based single-step inversion with a single-step text-to-image generation model to enable real-time text-guided image editing. Although SwiftEdit is significantly faster compared to multi-step and fewer-step editing methods, it simultaneously achieves competitive editing quality. The single-step editing allows for significant time savings and opens up new possibilities for applications that require real-time editing.
- Single-step inversion and editing for maximum speed. - Mask-guided editing for precise changes. - Attention scaling to control editing strength. - Competitive editing quality with significantly reduced editing time. - Versatile for various editing scenarios.
SwiftEdit sets a new standard in text-based image editing with its innovative technology. The combination of speed and quality makes the tool a promising solution for a variety of applications, from professional image editing to creative applications on mobile devices. Developments in this area will continue to be followed with great interest.
Bibliographie: https://www.arxiv.org/abs/2412.04301 https://arxiv.org/html/2412.04301v1 https://swift-edit.github.io/ https://www.zhuanzhi.ai/paper/140b4a1d3bda9d6fa73b91b8959a62ba https://github.com/wangkai930418/awesome-diffusion-categorized https://www.reddit.com/r/ninjasaid13/comments/1h7rp5n/241204301_swiftedit_lightning_fast_textguided/ https://bohrium.dp.tech/paper/arxiv/2407.17850 https://www.researchgate.net/publication/386401833_Fast_High-Resolution_Image_Synthesis_with_Latent_Adversarial_Diffusion_Distillation https://arxiv-sanity-lite.com/?rank=pid&pid=2411.15034 https://arxiv-sanity-lite.com/?rank=pid&pid=2412.04301 ```