Best Local LLMs for RTX 5090: Benchmarks

This is a guide to the desktop RTX 5090. The RTX 5090 Laptop GPU has a different memory budget and is not part of this ranking.

Quick Answer

Qwen3.8-27B is our general-purpose pick for the RTX 5090. It generated 79.4 tokens/s and processed a 4k prompt at 5,100 tokens/s. If output speed matters most, Nemotron 3.5 Lightning reached 334.3 tokens/s in the tested AtomicChat build. Gemma 4 26B-A4B offers a strong balance of speed and long-document work: it generated 248.6 tokens/s and took 24.4 seconds to answer the first question after our 128k-token document. All six models answered the document questions correctly, so the useful differences were wait time and VRAM use. Every number on this page came off a real RTX 5090 running the model, not from a memory-bandwidth estimate.

Best Local LLMs for RTX 5090

1. Qwen3.8-27B

Qwen3.8-27B is our all-round pick for a desktop RTX 5090. The dense model combines conventional attention with Gated DeltaNet layers and supports a native 262k-token context. Its strong performance in coding benchmarks makes it a great starting point for development work.

Released in August 2026, it is the 27B dense member of the Qwen3.8 family. The original model accepts text, images, and video and has switchable thinking. Our GGUF tests used text with thinking off. For this card we recommend the cdiamond community file: 70.8% of its weights are NVFP4, with the rest in Q5_K, Q6_K, and Q8_0.

The results above are for the original Qwen model.

Verdict: Choose the exact cdiamond NVFP4 GGUF for Qwen3.8-27B on Blackwell. It matched the AD-Q4 file size and processed the prompt faster. The 128k document took 62.6 seconds to produce its first token and used 24,773 MiB with the tested AtomicChat AD-Q4 build; do not assume those long-context figures transfer unchanged to the cdiamond file.

2. Gemma 4 31B it

Gemma 4 31B it is Google's largest dense Gemma 4 model. It is a useful alternative if you prefer Gemma's output style, but it is also the most memory-hungry of our six at long context.

Gemma 4 31B uses local sliding-window and global attention across its layers. Unlike the 26B-A4B model below, it uses all its language-model weights to produce each token. The table below shows how it performs in benchmarks. These are official Google numbers, and Google doesn't specify a per-row thinking setting:

Our RTX 5090 test:

Verdict: The dense Gemma alternative works at 128k on the RTX 5090, but it leaves only 2,246 MiB of the card's 32,607 MiB free in our test. Choose a lighter model if long-document response time or room for other GPU tasks matters more. Among our six it also holds the best published long-context score, 66.4% on MRCR v2 at 128k.

3. Gemma 4 26B-A4B it

If you need more generation speed at the small cost of reasoning performance, and want to stick to the Gemma family, choose Gemma 4 26B-A4B it. Its mixture-of-experts design stores about 25.2B parameters but activates roughly 4B for each token, helping it generate much faster than the dense 31B model.

The smaller active parameter count reduces computation per generated token, which is what makes generation fast here. Its lighter memory use at long context comes from a different place: a smaller file plus the way its attention layers store the KV cache. In our setup, this model used 19,831 MiB at 128k against 30,361 MiB for Gemma 4 31B. And here's how this model performs in the benchmarks, according to Google's official measurements for the original full precision file:

The table below displays results during our RTX 5090 test:

Verdict: Choose Gemma 4 26B-A4B when you want fast generation and a 128k document without consuming most of the RTX 5090's VRAM. On Google's own MRCR v2 test at 128k it scores 44.1% against 66.4% for the dense 31B, so pick the 31B when the answer matters more than the wait.

4. Gemma 4 12B it

Gemma 4 12B it is the smallest model in our lineup of the best models to run on an RTX 5090. As the smallest file, it makes the most sense when you need an AI model to run alongside other apps that also need GPU memory. The GGUF file we tested occupied 8,758 MiB at a 32k allocation and 10,447 MiB with our 128k document.

This is a much lighter model than the two larger Gemmas we've tested, yet it completed all of the planted-fact questions and preserved all four required topics in our 128k summary.

And here are the results from our test run:

Verdict: Choose Gemma 4 12B when you need a capable AI model that generates tokens fast and offers great performance for the size, or when you need additional GPU headroom. In our test, Gemma 4 12B used less than half the 128k VRAM use of Qwen or Gemma 4 31B.

5. Muse Glimmer 30B

Muse Glimmer 30B is Meta's agentic model with a dedicated perception encoder. Meta built it for tool use, multi-step reasoning and failure recovery on consumer hardware, and lists OpenClaw and Hermes Agent among the scaffolds it works with. Both of those already run on top of Atomic Chat's local endpoint.

Muse Glimmer continues to use reasoning tokens even at a low reasoning setting. Give it enough output budget for both thought and final answer. It also found all four facts at 8k, 32k, and 128k. It uses a denser tokenizer, which encoded our 128k-class document in 107,856 tokens against 130,557 for the Gemma models. Here's how this model performs in the benchmarks, according to Meta:

And here's how it fared in our test on an RTX 5090 graphics card:

Verdict: Muse Glimmer is the pick when you run agents locally. On Meta's own numbers it reaches 76.0% on SWE-bench Verified and 51.7% on Terminal-Bench 2.1, well ahead of the other agentic candidate here. Its denser tokenizer also fits more text into the same context length.

6. NVIDIA Nemotron 3.5 Lightning 30B-A3B

NVIDIA Nemotron 3.5 Lightning 30B-A3B is a great pick to run on an RTX 5090 when you're optimizing for inference speed. Its hybrid Mamba-2, attention, and mixture-of-experts architecture stores about 32.9B parameters but activates roughly 3B per token, and allows the model to output tokens quickly, especially for its parameter count.

The architecture is built to reduce per-token work and handle long sequences. In our tested build, it generated 334.3 tokens/s and returned the first token after the 128k document in 14.9 seconds. Those figures are for ordinary generation without speculative decoding. Here's how the model performs in benchmarks, according to NVIDIA:

And here are the results of our RTX 5090 test:

Verdict: Choose Nemotron 3.5 Lightning for fast output and long-document response time. On published agentic benchmarks it trails Muse Glimmer, so pick it for speed rather than for task success.

How the Six Models Compare

Only GPQA Diamond is published for all six models, so the table below marks which numbers come from the same benchmark and which do not. Each vendor runs its own harness: Qwen scored SWE-bench Pro with the Claude Code harness at a 256k context, while Meta scored it with its own setup. Treat cross-vendor rows as indicative, not as a head-to-head.

Two comparisons here are like for like. On MRCR v2 at 128k the three Gemma models line up 66.4, 44.1 and 43.4, so the dense 31B is far better at finding things in a long document than its faster siblings. On SWE-bench Verified and Terminal-Bench 2.1, Muse Glimmer reaches 76.0 and 51.7 against 51.56 and 24.58 for Nemotron, so the two agentic candidates are not close.

Picking Context Over Parameters

The RTX 5090's 32 GB lets you spend memory on a bigger model or on a longer context, and the two compete for the same card. Our 128k measurements show how wide that spread is.

Gemma 4 12B is the clearest case for trading parameters for room. It held our full 128k document in 10,447 MiB, less than half of what Qwen3.8-27B needed and a third of Gemma 4 31B, and it still answered all four planted-fact questions. That leaves 22 GB for a second model, an image model, or the rest of your desktop.

The trade is real, though, and Google's own numbers price it. On MRCR v2 at 128k the 12B scores 43.4% against 66.4% for the dense 31B. A long context you can load is not the same as a long context the model reads well. Load the biggest model whose answers you trust, then give the remaining memory to context.

Recommended file type for Blackwell architecture

The RTX 5090 uses NVIDIA's Blackwell architecture and has FP4 tensor cores. In our llama.cpp test, the NVFP4 GGUF files processed 4k prompts 28 to 58% faster than the standard GGUF build of the same model. Generation improved too, by 2 to 10%. For a long input, faster prompt processing reduces the wait before generation starts.

The exact community NVFP4 files tested were Qwen3.8-27B-iMatrix-NVFP4-MTP.gguf, gemma-4-31B-it-NVFP4-turbo-NVFP4.gguf, and Nemotron-3.5-Lightning-30B-A3B-NVFP4.gguf, respectively.

• The Qwen file has 70.8% of its weights in NVFP4 and the rest in Q5_K/Q6_K/Q8_0.
• The Gemma and Nemotron files are 95.4% and 95.6% NVFP4 by weight.

These are measured results for these specific files on llama.cpp b10988. We did not isolate how much of the gain comes from the FP4 tensor cores and how much from the smaller weights. NVIDIA's own card for Nemotron lists the GeForce RTX 5090 compute path as still to be confirmed, so treat the numbers as what these builds did on our card rather than as a property of the format.

How we tested these models

The main throughput and document tests used a desktop RTX 5090 with 32,607 MiB of VRAM, Ubuntu 24.04, and the official CUDA 12.8 build of llama.cpp b10988. We used full GPU offload, one parallel slot, fp16 KV cache, and automatic flash-attention selection. Throughput comes from llama-bench -p 512,4096 -n 128 -r 2; the tables show the 4k-prompt and 128-token-generation results. The Gemma files here are AtomicChat GGUF builds, not the QAT files from the earlier run. Qwen AD-Q5 speed and the Muse long-document run used a second pod with a different driver, so those two rows are not strictly comparable with the rest.

One caveat on the throughput table: the six models do not all run the same quantization recipe. Three Gemma files are Q4_K_M, Muse Glimmer is AD-Q4_K_M, Nemotron is AD-IQ4_NL, and Qwen is the mixed cdiamond NVFP4 file. Part of the spread between rows is the build, not the model.

Coding a Self-Playing Snake

We asked each model to build a snake game that plays itself, eats food, and grows. This test used vLLM with NVFP4 builds: AtomicChat/gemma-4-12B-it-NVFP4, AtomicChat/gemma-4-26B-A4B-it-NVFP4 and AtomicChat/gemma-4-31B-it-NVFP4 for the Gemma models, and nvidia/Qwen3.8-27B-NVFP4, nvidia/Muse-Glimmer-30B-NVFP4 and nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 for the rest.

Every model eventually completed the test, creating a snake that chases food, grows, and increases its score as the game plays.

Gemma 4 26B-A4B and 31B were able to build the game on the first attempt, and the 26B-A4B was also quickest at 27.1 seconds. Nemotron needed five attempts, but its total generation time was 45.7 seconds. Its repair requests used thinking off after the initial response exhausted the output budget.

Coding Five Bouncing Balls

We also asked each model to make a page with five balls affected by gravity and collisions. This test used the same vLLM and NVFP4 setup as the snake task.

As in the snake test, only Gemma 4 26B-A4B and 31B worked on the first attempt. Despite needing three attempts, Nemotron had the shortest generation time at 31.8 seconds.

Finding facts in a 128k-Token Document

We gave each model the same source document at roughly 8k, 32k, and 128k tokens. We planted facts near the beginning, middle, and end, then asked a fourth question requiring facts from two separate places.

We also checked whether a summary retained four specified topics. The document's token count differs by tokenizer, even though the source text is the same.

All six passed this particular retrieval and summary test at every length, so it separates them on wait time rather than on accuracy. For a harder read of long-context quality, use the published MRCR v2 scores in the comparison table above. Two rows also did less work than the others: Muse Glimmer processed 107,856 tokens against 130,557 for the Gemma models, and its run used the second pod.

At this length, Gemma 4 31B used 30,361 MiB and Qwen3.8-27B AD-Q4 used 24,773 MiB. Both fit on our 32,607 MiB RTX 5090, but exceed the nominal 24 GB (24,576 MiB) VRAM budget of an RTX 4090 or 3090. The other four models used 10,447 to 19,831 MiB in this test, below 24 GB. These are measured RTX 5090 memory footprints; the table does not show their speed on older GPUs. Qwen's 128k result is for the AtomicChat AD-Q4 file, not the cdiamond NVFP4 file recommended above.

How to Run Offline AI Models in Atomic Chat

1. Install Atomic Chat. When choosing a backend, select Find optimal backend for your GPU.

2. For an AtomicChat-built model in the table, open Models, search the linked AtomicChat repository, choose the exact filename shown in its card under Download Options, and download it.

3. For our recommended Qwen build, download Qwen3.8-27B-iMatrix-NVFP4-MTP.gguf from cdiamond's repository. In Atomic Chat, go to Settings, then Model Providers, then llama.cpp, then Import, and select that local GGUF file.

4. Select Use this model, open a new chat, and start with a 32k context allocation. Increase the context when a task requires it, watching GPU memory use. If the imported NVFP4 file does not load, update or select a compatible llama.cpp backend.

Atomic Chat's own self-hosted LLM guide walks through the app's model download and import controls.

FAQ

What is the best LLM to run on an RTX 5090?

For a balance of speed and performance, Qwen3.8-27B is the best AI model to run on an RTX 5090. If your priority is the shortest task completion time or the highest generation speed, choose Nemotron 3.5 Lightning.

Is 32 GB VRAM enough for a local LLM?

Yes. 32 GB of VRAM is enough to run most 12B to 33B models in 4-bit quantization entirely on the GPU, with room for a long context. Larger models or very long context windows may need a smaller quantization or partial CPU offload.

Can an RTX 5090 run a 70B model?

Not at 4-bit. A 70B Q4 file generally exceeds the RTX 5090's 32 GB VRAM before context memory is added. You can reach a 70B at a lower bit rate or with part of the model on the CPU, but both cost quality or speed.

What is the best coding LLM for RTX 5090?

Qwen3.8-27B holds the best published coding scores in our lineup, at 61.7 on SWE-bench Pro and 90.3 on LiveCodeBench v6. In our own two coding tasks it was slower to a working result than the Gemma models: Gemma 4 26B-A4B built both the snake and the balls page on the first attempt, in 27.1 and 32.9 seconds, against two attempts and 97.5 and 88.5 seconds for Qwen. Pick Qwen for harder problems and Gemma 4 26B-A4B when you want a working answer quickly.

How much power and system RAM does an RTX 5090 setup need?

NVIDIA specifies 575 W total graphics power for the RTX 5090 and a 1,000 W system power requirement for the reference configuration. System RAM needs depend on file size, context, and whether any model layers run on the CPU. Our benchmark host had 117 GB of RAM; that is not a requirement for these GPU-resident files.