Best Local LLM for 8GB RAM or VRAM in 2026

If you have 8GB of RAM or VRAM, you can run a pretty powerful local AI model. In short, if you're looking for the best local large language model to run with 8GB, consider Qwen 3.5 9B for an 8GB GPU and Qwen 3.5 4B for a system with 8GB of RAM or unified memory.

Further down in the article, we'll cover more options you might want to prioritize for specific tasks, so keep reading to find the right model for your needs.

If you're wondering how to run these models, read our guide on running a local LLM.

What is the best local LLM to run on 8GB?

As a rule of thumb, 8GB of VRAM is enough for a quantized 7B–9B model, while a computer with 8GB of total RAM or unified memory is better suited to 3B–4B models.

For an 8GB GPU, we looked for models that stay at roughly 6GB or less in a 4-bit quantization, leaving some memory for the KV cache and runtime overhead. Models much above 9B usually leave too little headroom, especially once you increase the context length.

On an 8GB RAM or unified-memory system, the model needs to be smaller because the operating system and inference software are using the same memory pool. Here, models around 2–3.5GB quantized are a more practical target, which generally means 3B–4B parameters.

That leaves us with the following models:

Best Local LLMs for 8GB VRAM Systems

The best LLM for 8GB VRAM is a 7B–9B model that can run entirely on the graphics card, including the model weights and working context.

You can sometimes load a 12B or 14B model by offloading some layers to system RAM, but this comes with a major speed penalty because data has to move across the PCIe bus while the model is generating. In most cases, a smaller model that fits fully on the GPU will run several times faster and feel much better to use.

Qwen 3.5 9B — Best Overall

Qwen 3.5 9B is our top pick for an 8GB GPU. At Q4_K_M, the model is about 5.7GB, so it can stay fully in VRAM while leaving room for the context cache and inference overhead.

We recommend Q4_K_M for 8GB cards. Higher-quality Q4 and Q5 builds can approach 6–7GB, which leaves much less room for context. Q6 and Q8 are too large to be practical if you want the model fully offloaded to the GPU.

Qwen 3.5 9B uses 32 transformer layers and a hybrid architecture that mixes Gated DeltaNet layers with standard attention layers. Its official context length is 262,144 tokens, with support for extending beyond that using YaRN. You should not expect to use anything close to 262K on an 8GB GPU, though. The available VRAM will limit your practical context long before you reach the model's architectural maximum.

The model also supports image input, which gives it an advantage over text-only models in the same size range. Qwen reports strong results across reasoning, coding, instruction following, OCR, and general vision benchmarks.

The biggest gains over Qwen 3.5 4B are in harder reasoning and coding tasks. For example, the 9B model scores 81.7 vs. 76.2 on GPQA Diamond and 65.6 vs. 55.8 on LiveCodeBench v6. The gap is smaller on several vision benchmarks.

These are Qwen's own published benchmark results, so they are useful for comparing the two models but should not be treated as independent testing of the exact GGUF build you will run locally.

Qwen 3.5 9B pros

• Q4_K_M fits fully in 8GB of VRAM
• Strong reasoning and coding performance for a 9B model
• Native image input
• Strong OCR and document-understanding results
• Supports thinking and non-thinking modes
• Supports 201 languages and dialects according to Qwen

Qwen 3.5 9B cons

• 262K context is not realistic on an 8GB GPU
• Q5 leaves little VRAM for context
• Q6 and Q8 are too large for a sensible fully offloaded 8GB setup
• Image inputs use additional memory
• The benchmark figures above are vendor-reported

When to choose Qwen 3.5 9B: Choose it if you have an 8GB GPU and want the strongest Qwen 3.5 model you can run fully in VRAM. Use Q4_K_M on cards such as the RTX 3060 Ti, RTX 3070, RTX 4060, RTX 5060, or RX 7600.

GLM-4.6V-Flash

GLM-4.6V-Flash is a multimodal model from Z.ai built for text, images, documents, video, and visual reasoning. The official BF16 weights are about 20.6GB, so an 8GB GPU needs a community quantized build. We recommend Q4_K_M, which is roughly 6.2GB.

On an RTX 3070, Q4_K_M can perform competitively with Qwen 3.5 9B at short and medium context lengths, but at around 6.2GB, there is less VRAM left for the context cache, and performance drops quickly if longer prompts force even a few layers into system RAM.

For that reason, Qwen 3.5 9B is probably better as the general-purpose choice on an 8GB card, but you might run GLM-4.6V-Flash when image, document, or visual reasoning performance matters more than having extra room for context.

GLM-4.6V-Flash pros

• Strong image and document understanding
• Supports video input
• Good visual and quantitative reasoning
• Fast prompt processing at shorter context lengths
• MIT license

GLM-4.6V-Flash cons

• Q4_K_M uses about 6.2GB, leaving limited VRAM for context
• Long prompts can force partial CPU offload and sharply reduce generation speed
• The official weights are far too large for an 8GB GPU
• Practical 8GB deployment relies on community GGUF builds

When to choose GLM-4.6V-Flash: Choose it if you have an 8GB GPU and care mainly about images, PDFs, scanned documents, or other visual inputs. Use Q4_K_M, keep the context length modest, and make sure all model layers remain on the GPU.

DeepSeek-R1-0528-Qwen3-8B

DeepSeek-R1-0528-Qwen3-8B is an 8B reasoning model distilled from DeepSeek-R1-0528 into Qwen3 8B. DeepSeek trained it to preserve much of the larger R1 model's reasoning behavior in a size that can run locally on consumer hardware.

We recommend Q4_K_M for an 8GB GPU, where the model is roughly 5.2GB. That leaves more VRAM for context than Qwen 3.5 9B or GLM-4.6V-Flash, although reasoning sessions can still use that headroom quickly. The model may generate thousands of reasoning tokens before producing its final answer, and those tokens also occupy the KV cache.

The main reason to run it is math and difficult reasoning. DeepSeek reports a large improvement over the original Qwen3 8B on AIME 2024, along with strong results in logic and programming benchmarks. It is less useful for ordinary chat, rewriting, or short factual questions, where the extra reasoning often means a slower and much longer response than you need.

DeepSeek-R1-0528-Qwen3-8B pros

• Excellent mathematical reasoning for an 8B model
• Strong logic and programming performance
• Q4_K_M leaves around 2.8GB of an 8GB card free before runtime overhead
• Mature support in common local inference runtimes
• MIT license

DeepSeek-R1-0528-Qwen3-8B cons

• Reasoning traces can add thousands of tokens to otherwise simple answers
• Long reasoning sessions increase KV-cache use and reduce practical context headroom
• Text-only
• Slower and more verbose than necessary for routine chat, summarization, and writing

When to choose DeepSeek-R1-0528-Qwen3-8B: Choose it for math, difficult debugging, logic problems, and other tasks where reasoning quality matters more than response speed. For general chat and writing, Qwen 3.5 9B is the better default.

Qwen2.5-Coder 7B

Qwen2.5-Coder 7B Instruct is a 7B model from Alibaba built specifically for programming. It handles code generation, debugging, explanation, refactoring, and tests across more than 40 programming languages.

Go for Q4_K_M or Q5_K_M on an 8GB GPU. A typical Q4 build is around 4.7GB, leaving considerably more VRAM for context than the 9B models above. If your runtime and context requirements allow it, an 8GB card also has enough room to run a higher-quality Q5 quantization without offloading layers to system RAM.

The smaller size is particularly useful when you need to feed the model several source files, a long function, compiler output, or test failures. Its 32K native context can also be extended with YaRN, although the amount you can use still depends on your quantization and available VRAM.

Qwen2.5-Coder 7B pros

• Trained specifically for code generation, repair, explanation, and completion
• Q4 leaves roughly 3GB of an 8GB card available before runtime overhead
• Q5 is also practical on many 8GB cards
• Supports more than 40 programming languages
• Widely supported by local inference apps and coding tools

Qwen2.5-Coder 7B cons

• Older than the Qwen 3.5 generation
• Qwen 3.5 9B is stronger for many tasks that mix coding with general reasoning
• Text-only, so it cannot inspect screenshots, diagrams, or other visual inputs
• A 7B model is still limited for large repository-wide changes and long autonomous coding tasks

When to choose Qwen2.5-Coder 7B: Pick it if you mainly want a local coding model with more room for source code and context on an 8GB GPU.

Learn more about the best coding models in our guide to the best local LLMs for coding.

Best Local LLMs for 8GB RAM or Unified Memory Systems

On an 8GB Apple Silicon Mac, the CPU and GPU share the same memory pool, while on a Windows or Linux PC with that amount of total memory, model weights are partially offloaded to system RAM. In either case, we recommend keeping the quantized model itself to roughly 2–3.5GB so the rest of the system still has room to work. This means that if your computer has 8GB of total RAM or unified memory, stick to 3B–4B models at Q4. A 7B model may technically load, but it leaves too little memory for the operating system, the inference runtime, and context. The best small LLMs in this class come from the Qwen, Phi, Gemma, and SmolLM families.

Qwen 3.5 4B — Best Overall

Qwen 3.5 4B is our top pick for a computer with 8GB of RAM or unified memory. Builds at Q4 are about 3.4GB, which leaves around 4.5GB for the operating system, the runtime, and context.

Qwen 3.5 4B supports text and images, plus thinking mode, coding, tool use, and 201 languages. It has the same 262K native context limit as the 9B model.

In benchmarks, the 4B model is not far behind Qwen 3.5 9B:

Qwen 3.5 4B pros

• Q4 is about 3.4GB
• Strong text and vision support
• Supports thinking and non-thinking modes

Qwen 3.5 4B cons

• Behind the 9B model in coding and harder reasoning
• Long contexts can still push an 8GB system into swap
• Image inputs use additional memory
• The 262K context limit is not realistic on 8GB

When to choose Qwen 3.5 4B: Start here if you have 8GB of total RAM or unified memory. Drop to a 3B model if your computer starts swapping heavily or already uses a large share of its memory at idle.

Phi-4 Mini — Best for CPU Use and Compact Reasoning

Phi-4 Mini Instruct is Microsoft's compact 3.8B text model. It was trained with an emphasis on reasoning-rich synthetic data, instruction following, mathematics, logic, code, and function calling. A Q4 build weighs around 2.5GB, giving it one of the best memory margins on an 8GB system.

Phi-4 Mini is smaller and older than Qwen 3.5 4B, but its focused training can make it competitive on structured text and reasoning tasks.

*Qwen reports GPQA Diamond rather than standard GPQA, so the GPQA scores are not directly equivalent.

It's text-only, but that also means none of the memory budget goes to a vision encoder.

Phi-4 Mini pros

• Large memory margin on an 8GB computer
• Strong math, logic, and instruction following for its size
• Text-only design avoids vision overhead
• Q5 builds can still fit while leaving reasonable runtime memory

Phi-4 Mini cons

• No image understanding
• Less broad multilingual coverage than Qwen 3.5 or Gemma 3
• Can be overly formal on casual writing tasks

When to choose Phi-4 Mini: Choose it for CPU-only inference, structured Q&A, math, logic, code explanation, or an 8GB laptop that is also running several everyday applications.

Gemma 3 4B

Gemma 3 4B Instruct is Google's lightweight multimodal model. It accepts text and images, supports more than 140 languages, and has an official 128K input context. The common quantized build is approximately 3.3GB.

Gemma 3 4B pros

• Strong multilingual coverage
• Useful image, screenshot, and document understanding
• Quantization-aware-trained variants are available

Gemma 3 4B cons

• Vision inputs reduce the context headroom available on an 8GB machine
• More restrictive terms than Apache 2.0 or MIT models
• Qwen 3.5 4B is the stronger default for mixed workloads

When to choose Gemma 3 4B: Use it when multilingual work or image understanding is central to the workload, or when you already have a Gemma-compatible workflow and want a compact model with official quantized variants.

SmolLM3 3B

SmolLM3 3B is a small, fully open model from Hugging Face. It supports both thinking and non-thinking modes and has a 64K context window, which can be extended to 128K with YaRN.

A Q4 version takes up around 2GB, so SmolLM3 is a good option if you want to keep plenty of memory free for the operating system and other apps.

SmolLM3 3B pros

• Uses very little memory, leaving more RAM available for the OS and context
• Hugging Face publishes the full training recipe, rather than only the model weights
• Supports both thinking and direct-response modes
• Covers several major European languages

SmolLM3 3B cons

• Weaker at demanding reasoning and coding tasks than Qwen 3.5 4B
• Text-only
• Fewer ready-made packages and integrations than Qwen, Gemma, or Phi

When to choose SmolLM3 3B: If you want a lightweight offline assistant for writing, summarization, or general everyday use.

Managing Context With Limited Memory

You'll find that managing context length on a system with little memory is one of the most important things for a good local AI experience. Give the model too little and it becomes too forgetful; give it too much and it's going to crash.

How much RAM or VRAM the model needs depends on many factors, but the heaviest ones are:

• Model weights
• KV cache

The model weights take up a mostly fixed amount of memory once the model is loaded, but KV cache grows as the conversation or prompt gets longer. In general, a longer active context means a larger KV cache.

On an 8GB GPU, the model and its KV cache must share the available VRAM. If they no longer fit, some data may spill into system RAM. This can slow generation considerably because data has to move between the GPU and CPU over PCIe. If there's nowhere to offload that data, the model can crash.

Reducing KV-cache memory

There are three common ways to reduce the memory used by long context:

• Use a shorter active context
• Quantize the KV cache
• Use a model with a more memory-efficient attention architecture

For example, Atomic Chat, a local AI app we've built, supports TurboQuant, which compresses the KV cache while the model is running. TurboQuant is a mechanism created by Google — it reduces KV values to roughly three bits. In Google's long-context tests, this produced at least a sixfold reduction in KV-cache memory without measurable accuracy loss on the evaluated tasks.

On an 8GB system, that can be the difference between running out of KV-cache memory at 16K tokens and using the same model at roughly 64K. A 4-bit cache can hold about four times as many tokens as a conventional 16-bit cache in the same memory space; Atomic Chat's implementation supports 3- and 4-bit storage — up to 4.3× reduction.

What Models We Wouldn't Recommend for 8GB Systems

A few popular models look tempting on paper but don't leave enough memory to run well on an 8GB system.

Gemma 3 12B on 8GB VRAM

Some Gemma 3 12B quantized builds sit just above the capacity of an 8GB card before adding the KV cache and runtime buffers — this is tempting, but the model will most likely use partial CPU offload, which will be painfully slow.

Phi-4 14B

At Q4, you can theoretically run it on an 8GB GPU, but once you account for the memory you'll need for context and buffers, you'll very quickly need to offload it.

GPT-OSS 20B at IQ2

Unfortunately, 8GB of VRAM or RAM isn't enough to run even OpenAI's smallest 20B open-weights model — you can barely make it work with an extreme IQ2 quantization, but at that point it's not worth it. Even then, there is little room for context, and the low-bit quality loss undermines the benefit of loading a larger model. GPT-OSS 20B is much more compelling on 16GB or more.

FAQ

Quick answers to the most common questions about running a local LLM on 8GB of memory.

What is the best local LLM for 8GB VRAM in 2026?

Qwen 3.5 9B Q4. It gives you a strong mix of general reasoning, coding, multilingual performance, and image input, and it can fit entirely on many 8GB GPUs.

Can I run a 12B model on 8GB VRAM?

Yes, but you will probably need to keep part of it in system RAM. The weights may fit at a low enough quantization, but the KV cache and runtime buffers also consume VRAM. For example, Atomic Chat and llama.cpp can split the model between GPU and system memory. That avoids an out-of-memory error, but it can also hurt generation speed because data has to move across PCIe.

For regular use, a fully GPU-resident 8B or 9B model is often a better trade.

Can I run Llama 3.1 8B on 8GB RAM?

Barely, at Q4. On a machine with only 8GB of system RAM, the model takes up most of the available memory once the OS is running. That leaves little room for context or anything else.

What context length can I use with 8GB VRAM?

Qwen 3.5 9B, for example, may reach 16K or 32K while staying fully offloaded, particularly with a quantized KV cache.

What quantization should I use for 8GB VRAM?

Q4_K_M. For an 8B–9B model, it gives you a good chance of fitting the full model on the GPU without giving up too much quality. Q5_K_M can make sense for a 7B model if you do not need a large context.

How much VRAM does a 9B model use with llama.cpp?

A 9B Q4_K_M GGUF is usually around 5–6GB for the weights alone. llama.cpp also needs memory for the KV cache, compute buffers, and other runtime allocations. A running session can therefore land around 6–7GB or higher, depending mainly on context length and KV-cache precision. Check memory usage while the model is running rather than judging fit from the GGUF file size.

Bottom line

• If you have a graphics card with 8GB VRAM, you can run 7B–9B models at Q4 with full GPU offload. In that class, consider Qwen 3.5 9B.
• If your system has 8GB RAM or unified memory, 7B–9B models might be too slow, so consider 3B–4B models. In this range, the best one is Qwen 3.5 4B.
• A model that fits entirely in fast memory generally provides higher inference speed than a larger model using CPU offload or swap.
• Account for the KV cache as well as model weights when choosing a model. Memory use increases with context length, and when memory is limited you can quickly run out of it. You can negate that with TurboQuant in Atomic Chat, which reduces this runtime memory cost, allowing you to run models with a longer context than you normally would be able to locally.