Quick Answer
Gemma 4 12B Q4_K_M is our all-round pick among the six builds tested on a desktop RTX 4090. It generated 99.7 tokens/s and used 8,628 MiB of VRAM at 32k. It also completed both coding tasks on the first attempt. Start with gemma-4-12b-it-Q4_K_M.gguf and a 32k context allocation. For the fastest output, Nemotron 3.5 Lightning reached 224.7 tokens/s, though it needed five attempts to make the snake game work. Gemma 4 26B-A4B is the faster Gemma option for long documents: it generated 194.1 tokens/s, with higher memory use than the 12B.
This comparison covers the desktop card with 24 GB of VRAM. The RTX 4090 Laptop GPU has 16 GB and needs a different memory budget. All speed figures below were measured in llama.cpp; Atomic Chat performance depends on its selected backend and settings.
The table below compares the exact GGUF builds tested. For a 32 GB card, see our RTX 5090 benchmark. Largest context loaded means a successful memory allocation, not a quality test across the entire window. The document checks are reported separately.
| Model | Tested quant | File size | VRAM at 32k | 4k prompt | Generation | Largest fp16 context loaded | Best for |
|---|---|---|---|---|---|---|---|
| Qwen3.8-27B | AD-Q4_K_M | 15.94 GiB | 18,222 MiB | 3,004 tok/s | 48.5 tok/s | 96k | Coding and general use |
| Nemotron 3.5 Lightning 30B-A3B | AD-IQ4_NL | 18.30 GiB | 18,484 MiB | 9,613 tok/s | 224.7 tok/s | 192k | Fast output |
| Gemma 4 26B-A4B it | Q4_K_M | 15.64 GiB | 17,572 MiB | 9,970 tok/s | 194.1 tok/s | 192k | Long documents and speed |
| Gemma 4 12B it | Q4_K_M | 6.87 GiB | 8,628 MiB | 7,028 tok/s | 99.7 tok/s | 192k | Best tested all-rounder |
| Muse Glimmer 30B | AD-Q4_K_M | 17.75 GiB | 18,260 MiB | 3,495 tok/s | 46.7 tok/s | 128k tested limit | Reasoning-focused alternative |
| Gemma 4 31B it | IQ4_XS | 15.59 GiB | 20,380 MiB | 3,165 tok/s | 48.7 tok/s | 64k | Dense Gemma alternative |
Best Local LLMs for RTX 4090
Qwen3.8-27B: Strong Reasoning Performance
Qwen3.8-27B is Alibaba's 27.32B-parameter dense model for coding, research, and everyday assistance. Its 64 language-model layers combine Gated DeltaNet with conventional attention: most layers handle sequences through the lighter linear-attention path, while regular attention layers help the model connect information across a prompt. The model accepts text, images, and video, and lets you switch thinking on or off or adjust its reasoning effort. Our speed and document tests used text with thinking off. The coding tests below used different thinking settings.
The original Qwen3.8-27B checkpoint scored 89.2 on GPQA Diamond, 61.7 on SWE-bench Pro, and 73.0 on Terminal-Bench 2.1, making it a useful model when your work moves between questions, documents, and code.
| Tested file | Result |
|---|---|
| Repository | AtomicChat/Qwen3.8-27B-GGUF |
| Exact GGUF | Qwen3.8-27B-AD-Q4_K_M.gguf |
| File size | 15.94 GiB |
| VRAM at 32k | 18,222 MiB |
| 4k prompt / generation | 3,004 / 48.5 tok/s |
| Largest fp16 allocation loaded | 96k; 22,381 MiB |
| 128k with q8_0 KV | Loaded; 21,091 MiB allocated |
Qwen supports a native 262k-token context, but the RTX 4090's memory is not enough to take full advantage of it. With the default fp16 KV cache, a 128k allocation failed to load on our system, but setting both cache types to q8_0 made it work at 128k without changing the model file.
Verdict: Choose Qwen for coding and general use at a moderate context. This GGUF completed both browser tasks on its first attempt. At 32k it used 18,222 MiB; reaching 128k required q8_0 KV cache.
NVIDIA Nemotron 3.5 Lightning: Fastest Generation
Nemotron 3.5 Lightning 30B-A3B mixes Mamba-2 layers, attention, and a mixture of experts. It stores about 30B parameters but activates roughly 3B for each token. The design targets quick text generation and long-running agent workflows, including tool calls and structured output. It is a text-only model, unlike the image-capable Qwen, Gemma, and Muse.
This Nemotron checkpoint supports up to a million tokens of context, though the largest allocation in our RTX 4090 sweep was 192k. NVIDIA’s BF16 baseline scored 75.44 on GPQA Diamond and 51.56 on SWE-bench Verified.
Choose Nemotron when generation speed and the wait for the first token are your priorities.
| Tested file | Result |
|---|---|
| Repository | AtomicChat/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF |
| Exact GGUF | NVIDIA-Nemotron-3.5-Lightning-30B-A3B-AD-IQ4_NL.gguf |
| File size | 18.30 GiB |
| VRAM at 32k | 18,484 MiB |
| 4k prompt / generation | 9,613 / 224.7 tok/s |
| 128k document | 19.2 s to first token; facts 4/4; summary topics 4/4 |
| Largest tested allocation loaded | 192k; 19,603 MiB |
That speed makes it especially useful for iterative work: asking follow-up questions, revising drafts, or running an agent that produces many short responses. The hybrid architecture also keeps its KV cache relatively compact as context grows, which is why its 192k allocation still fit with substantial VRAM headroom.
Verdict: Choose Nemotron for fast, text-only chats and long documents. Its five snake attempts show why generation speed alone does not predict how many corrections a task will need.
Gemma 4 26B-A4B it: Best Balance
Gemma 4 26B-A4B it is the mixture-of-experts model from Google's Gemma 4 model family. It routes each token through eight of its 128 experts plus a shared expert, activating roughly 3.8B parameters at a time. That gives it much of a larger model's capacity while doing less work to produce each token. It accepts text and images in the original release and supports a native 256k-token context. Gemma 4 also combines local sliding-window attention with periodic global attention, and the family supports configurable thinking, function calling, and system instructions.
| Tested file | Result |
|---|---|
| Repository | AtomicChat/gemma-4-26B-A4B-it-GGUF |
| Exact GGUF | gemma-4-26B-A4B-it-Q4_K_M.gguf |
| File size | 15.64 GiB |
| VRAM at 32k | 17,572 MiB |
| 4k prompt / generation | 9,970 / 194.1 tok/s |
| 128k document | 24.8 s to first token; facts 4/4; summary topics 4/4 |
| Largest tested allocation loaded | 192k; 20,931 MiB |
The MoE means that this model can generate tokens much faster than the Gemma 31B dense model, for example, at the cost of some accuracy.
Verdict: Pick Gemma 26B-A4B when you regularly work with large inputs and want fast responses without filling the RTX 4090's VRAM.
Gemma 4 12B it: Best All-Rounder
Gemma 4 12B it is our default pick for the tested workloads when free VRAM matters. Google's 11.95B-parameter dense checkpoint accepts text, images, audio, and video, and generates text. It projects visual and audio inputs into the language model without separate perception encoders. Our GGUF benchmark measured text only.
The 12B shares Gemma 4's mix of local and global attention, configurable thinking, system-role support, and native 256k context. The original checkpoint scored 78.8% on GPQA Diamond and 72.0% on LiveCodeBench v6.
The 12B model gives up some of the larger Gemmas' reasoning strength in exchange for a much smaller footprint, making it appealing when the GPU also needs to run other applications, or another model.
| Tested file | Result |
|---|---|
| Repository | AtomicChat/gemma-4-12B-it-GGUF |
| Exact GGUF | gemma-4-12b-it-Q4_K_M.gguf |
| File size | 6.87 GiB |
| VRAM at 32k | 8,628 MiB |
| 4k prompt / generation | 7,028 / 99.7 tok/s |
| 128k document | 37.2 s to first token; facts 4/4; summary topics 4/4 |
| Largest tested allocation loaded | 192k; 11,347 MiB |
For routine writing, document questions, and lightweight local automation, 12B is easy to keep loaded throughout the day. It completed every question in our long-document check without any issues.
Verdict: Start with Gemma 12B if you want memory headroom and a model that completed both of our coding tasks on its first attempt. These two tasks do not establish a general coding-quality ranking.
Muse Glimmer 30B: Agent-Focused Alternative
Muse Glimmer 30B is Meta's local agent model, distilled from the larger Muse Spark. Its dense language model is trained for multi-step reasoning, tool use, and recovering when an action fails. It has a separate perception encoder for screenshots, charts, and other images. Our GGUF speed and document tests used text only.
Muse Glimmer lets you set reasoning strength from low through xhigh, making it adaptable to the complexity of work you're currently doing. Benchmark performance is also strong: 83.5 on GPQA Diamond and 51.2 on SWE-bench Pro for the original checkpoint.
| Tested file | Result |
|---|---|
| Repository | AtomicChat/Muse-Glimmer-30B-GGUF |
| Exact GGUF | Muse-Glimmer-30B-AD-Q4_K_M.gguf |
| File size | 17.75 GiB |
| VRAM at 32k | 18,260 MiB |
| 4k prompt / generation | 3,495 / 46.7 tok/s |
| Long document | 41.7 s to first token; facts 4/4; summary topics 4/4 |
| Largest allocation loaded | 128k tested limit; 19,603 MiB |
Muse has an unusually economical tokenizer. The same text that with Gemma used up 130,557 tokens only used 107,856 tokens for Muse, so it can potentially hold more information within the same context window.
Verdict: Try Muse Glimmer if you want to evaluate Meta's agent-focused model locally. It completed both browser tasks on its first attempt, but this test did not measure autonomous tool use or agent reliability.
Gemma 4 31B it: Dense Gemma Alternative
Gemma 4 31B it is Google's 30.7B-parameter dense alternative to the 26B-A4B mixture-of-experts model. It features 60 layers that alternate local sliding-window and global attention, with a native 256k context. It handles text and images, supports configurable thinking, and can follow system instructions or produce function calls.
The dense design favors model capacity over speed on this GPU. The original 31B scored 84.3% on GPQA Diamond, 80.0% on LiveCodeBench v6, and 66.4% on MRCR v2's eight-needle 128k test. That last result is notably higher than the 26B-A4B model's 44.1%, making 31B interesting when retrieving information from large inputs matters more than output speed.
| Tested file | Result |
|---|---|
| Repository | AtomicChat/gemma-4-31B-it-GGUF |
| Exact GGUF | gemma-4-31B-it-IQ4_XS.gguf |
| File size | 15.59 GiB |
| VRAM at 32k | 20,380 MiB |
| 4k prompt / generation | 3,165 / 48.7 tok/s |
| Largest fp16 allocation loaded | 64k; 22,971 MiB |
| 128k with q8_0 KV | Loaded; 23,725 MiB allocated |
For this model, the memory cost of context is the limiting factor on an RTX 4090. The fp16 KV cache stopped it from loading at 96k, but at q8_0 we were able to load the same GGUF with 128k. However, that configuration left less than 1 GiB free on the card, so running it at 32k or 64k is more realistic.
Verdict: Choose Gemma 31B if you want the dense Gemma model and typically work at moderate context lengths.
How We Benchmarked
The speed, memory and document test machine used a desktop RTX 4090 with 24,564 MiB of VRAM, Ubuntu 24.04.3, NVIDIA driver 580.159.04, and the official CUDA 12.8 binary of llama.cpp b10988. We used full GPU offload, one parallel slot, and an fp16 KV cache for the main results. All benchmark prompts were text-only. We also checked that the binary contained native sm_89 kernels for the RTX 4090.
For the long-context test, every model received the same source document at roughly 8k, 32k, and 128k tokens. We placed facts near its beginning, middle, and end, added a question requiring facts from two places, and checked whether a summary retained four required topics: all models completed this test without issue.
We ran the two coding tasks on September 19 and 20, 2026, on a separate RTX 4090 host with NVIDIA driver 590.48.01. It used the same llama.cpp b10988 CUDA 12.8 build and exact GGUF filenames above, with full GPU offload, one slot, a 16,384-token context, and an 8,192-token output budget including reasoning. Sampling was temperature 1.0, top-p 0.95, top-k 64, and seed 42. Thinking was enabled except for Nemotron, whose reasoning had exhausted the output budget in our earlier RTX 5090 run.
Each task started with one generated answer per model. We opened the output in Chrome and sent concrete browser feedback back to the model when a repair was needed. The model made every correction; we did not edit its HTML. Total generation time sums the model's attempts and excludes browser testing and feedback time. The videos show the final working version.
Vendor benchmark scores above describe the original checkpoints. Our tested GGUFs use different quantization recipes, so speed and memory differences reflect both the model and the file. The document check and two browser tasks cover specific workloads; they do not establish an overall quality ranking.
Coding a Self-Playing Snake
We asked each model to build a snake game that plays itself, where the snake eats food and grows.
| Model | Attempts | Total generation time |
|---|---|---|
| Qwen3.8-27B | 1 | 118.4 s |
| Gemma 4 31B | 1 | 103.1 s |
| Gemma 4 26B-A4B | 3 | 99.7 s |
| Gemma 4 12B | 1 | 93.2 s |
| Muse Glimmer 30B | 1 | 171.0 s |
| Nemotron 3.5 Lightning | 5 | 57.5 s |
Every model eventually completed the test. Qwen3.8-27B, Gemma 4 31B, Gemma 4 12B, and Muse Glimmer made a working game on the first attempt, and Nemotron needed five attempts, but still had the shortest total generation time at 57.5 seconds.
Coding Five Bouncing Balls
We also asked each model to make a page with five balls affected by gravity and collisions.
| Model | Attempts | Total generation time |
|---|---|---|
| Qwen3.8-27B | 1 | 84.8 s |
| Gemma 4 31B | 1 | 84.9 s |
| Gemma 4 26B-A4B | 1 | 23.7 s |
| Gemma 4 12B | 1 | 60.5 s |
| Muse Glimmer 30B | 1 | 145.8 s |
| Nemotron 3.5 Lightning | 1 | 9.6 s |
All six models completed the five-ball simulation on the first attempt. Nemotron had the shortest generation time of just 9.6 seconds, followed by Gemma 4 26B-A4B at 23.7 seconds.
How to Run These Models in Atomic Chat
- Install Atomic Chat and select Find optimal backend so it can choose a compatible GPU runtime.
- Open Models and search for the repository linked in your chosen model's card above.
- Under Download Options, select the exact GGUF filename from that card. Download it and choose Use this model.
- Open a new chat with a 32k context allocation first. Increase context only when a task calls for it, and watch GPU memory use.
For a GGUF already on your computer, use Settings → Model Providers → llama.cpp → Import.
To reproduce the 128k q8_0 configuration directly in llama.cpp, run its server with your downloaded file and both KV cache types set to q8_0:
llama-server -m /path/to/model.gguf -ngl 999 -c 131072 --parallel 1 \ --cache-type-k q8_0 --cache-type-v q8_0
Check memory use on your own machine before relying on that allocation. If you are still choosing a model, download Atomic Chat and begin with the exact GGUF from the table rather than a similarly named quant.
FAQ
What is the best LLM to run on an RTX 4090?
Gemma 4 12B Q4_K_M is our all-round pick among the six tested builds. It used 8,628 MiB at 32k, generated 99.7 tokens/s, and completed both coding tasks on the first attempt. Start at 32k context and increase it when needed.
Is 24 GB VRAM enough for a local LLM?
Yes. All six files in this comparison loaded entirely on the GPU at 32k. Whether another model fits depends on its exact file size, KV cache, context allocation, and memory used by other applications.
Can an RTX 4090 run a 70B model?
A 70B model at four bits needs about 35 GB for weights alone before quantization overhead and runtime memory. It will not fit entirely in 24 GB at that precision. Partial CPU offload can make it run if you have enough system RAM, with lower generation speed.
What is the best coding LLM for RTX 4090?
Qwen3.8-27B has strong published coding results for its original checkpoint, and our AD-Q4_K_M file completed both browser tasks on the first attempt. Gemma 4 12B also passed both first time while using much less VRAM. These small tasks do not establish which model is best for a large codebase.
Is RTX 4090 faster than RTX 3090 for local LLMs?
Both desktop cards have 24 GB of VRAM. The speed difference depends on the model, quantization and inference backend. This article measures the RTX 4090; it does not establish a fixed speed advantage over the RTX 3090.
What can an RTX 4090 Laptop with 16 GB run?
An RTX 4090 Laptop GPU with 16 GB can run most 7B to 14B models in 4-bit quantization entirely in VRAM. Models in the 20B to 30B range generally require lower-bit quantization, shorter context windows, or partial CPU offload.
Is the RTX 4090 still good for local AI in 2026?
Yes. Its 24 GB of VRAM is enough for many current 12B to 32B models in 4-bit quantization, and its high memory bandwidth provides fast local inference. While newer 32 GB cards offer more room for large models and long contexts, the RTX 4090 is still a strong single-GPU platform for local AI.

