.png)
April 22, 2025
Google has released new versions of its Gemma 3 language models that significantly reduce memory requirements through quantization. This allows the models to run on commercially available graphics cards like the RTX 3090 and even on mobile devices. A decisive advancement that makes the use of powerful AI accessible to a wider audience.
The original Gemma 3 models were designed for high-performance systems with NVIDIA H100 graphics cards and BFloat16 precision, making them inaccessible to most users. The new variants, however, utilize a specialized training approach that enables efficient execution on consumer hardware without significant loss of quality.
At the heart of this development is quantization, a process that drastically reduces memory usage. Weights and activations are stored with fewer bits – often 8, 4, or even just 2 – instead of the usual 16 or 32. The result is smaller models that run faster, as lower precision numbers can be processed and transferred more quickly.
Google uses Quantization-Aware Training (QAT) for Gemma 3. This technique simulates reduced bit widths already during training. This allows the model to adapt to these limitations and minimizes the usual performance drop that occurs when running with lower precision.
The memory savings are substantial. The 27B model now only requires 14.1 GB of VRAM in int4 format instead of the original 54 GB. The 12B variant shrinks from 24 GB to 6.6 GB. Even the smaller models benefit: the 4B version requires 2.6 GB and the 1B model only 0.5 GB.
Google emphasizes the robustness of the models against quantization, which typically leads to a loss of quality. Updated benchmark results that substantiate this claim, however, have not yet been released.
The models are compatible with common inference engines and can be integrated into existing workflows. Native support is offered by Ollama, LM Studio, and MLX (for Apple Silicon), among others. Tools like llama.cpp and gemma.cpp support the quantized Gemma models in the GGUF format.
Beyond Google's official releases, the community is experimenting under the name "Gemmaverse" with variants that use post-training quantization to customize model size, speed, and quality.
The quantized Gemma 3 models are available on platforms like Hugging Face and Kaggle in various formats. This development opens up new possibilities for developers and users who want to benefit from powerful AI models without relying on expensive hardware. It remains to be seen how the "Gemmaverse" will evolve and what innovations will arise from the wider availability of this technology.
Sources: - https://the-decoder.com/gemma-3-27b-it-qat-q4_0-gguf-sounds-like-a-wi-fi-password-but-its-googles-leanest-ai-yet/ - https://huggingface.co/google/gemma-3-27b-it-qat-q4_0-gguf - https://the-decoder.com/googles-ai-overviews-are-quietly-draining-clicks-from-top-sites-new-data-shows/ - https://www.reddit.com/r/LocalLLaMA/comments/1jqnnfp/official_gemma_3_qat_checkpoints_3x_less_memory/ - https://www.threads.net/@testingcatalog/post/DHGEUSDNvbL/gemma-3-27b-is-also-now-available-on-ai-studio?hl=ko - https://news.ycombinator.com/item?id=43743337 - https://www.aitoolhunt.com/ai-daily-news - https://developers.googleblog.com/en/gemma-3-quantized-aware-trained-state-of-the-art-ai-to-consumer-gpus/ - https://github.com/ollama/ollama/issues/10121 - https://huggingface.co/google/gemma-3-27b-it-qat-q4_0-gguf/discussions/2