EXL3 Quantization Compared with GGUF on Quality, Speed and VRAM

In this article we'll compare LLM performance across two tested stacks: EXL3 quantization served through ExLlamaV3 and TabbyAPI, and GGUF models served through llama.cpp. This is not a test of the file containers in isolation: the EXL3 tensors in this benchmark are stored in safetensors files, while GGUF is a container that supports multiple quantization schemes.

TL;DR

In our EXL3 vs GGUF benchmark we found that EXL3 delivered better quality for a given amount of quantized model weights and much faster prompt processing on an RTX 5090; however, GGUF loaded and decoded faster, with a wider hardware support.

• EXL3 4.00 bpw was closer to the BF16 model at a lower quantized-weight size: 5.79 GiB and 0.0248 mean KLD, compared with 6.62 GiB and 0.0398 for GGUF Q4_K_M.
• EXL3 4.00 bpw was 7.79 GiB after its high-precision embeddings were included, almost identical to the 7.84 GiB GGUF Q5_K_M file, which had the lower KLD at 0.0172.
• ExLlamaV3 processed a 2,048-token prompt up to 2.5× faster, while llama.cpp decoded 15-21% faster in the two file-size-matched pairs.
• GGUF is easier to deploy broadly because llama.cpp can use CPUs and several GPU backends, including mixed CPU/GPU execution. The tested EXL3 path used ExLlamaV3 and TabbyAPI on CUDA.
• In this test, EXL3's current mul1 codebook reduced KLD and improved decode throughput relative to the legacy MCG build; prefill results were similar.

What Is EXL3 Quantization?

EXL3 quantization is a low-bit quantization scheme based on ideas from QTIP, which uses trellis-coded quantization and incoherence processing to preserve more information at low bitrates. It is part of the ExLlama ecosystem and is designed to compress model weights below the usual 4-bit level while keeping quality loss relatively small. Instead of forcing every weight into the same fixed bit width, EXL3 can use more flexible bit allocation, which helps squeeze a model into less VRAM. Compared to GGUF, EXL3 is optimized around ExLlama's GPU inference stack and newer quantization techniques.

How EXL3 Differs from EXL2 and GGUF

EXL2 is the older ExLlama quantization format, designed around mixed-precision weight quantization and optimized specifically for fast NVIDIA GPU inference through ExLlamaV2.

EXL3, introduced with ExLlamaV3, changes the underlying method substantially: it is a streamlined variant of QTIP, using procedural codebooks and trellis-based vector quantization rather than simply being a refined EXL2. EXL3 is intended to preserve model quality better at very low bitrates, and the quantizer can target an average bitrate such as 2.0, 3.0, or 4.0 bits per weight. EXL3 also retains more of the original Hugging Face tensor structure than EXL2, which should make integration with other runtimes easier.

GGUF is a model file format used primarily by llama.cpp, containing weights, model metadata, tokenizer information, and related data. GGUF supports many quantization schemes such as Q4_K, Q5_K and newer importance-weighted/i-quant variants. GGUF models can run through llama.cpp across CPUs, NVIDIA/AMD/Apple GPUs and heterogeneous CPU+GPU configurations. EXL3, by contrast, is much more GPU-focused and tightly optimized around the ExLlamaV3 runtime.

What BPW Means for Model Size and VRAM

BPW means “bits per weight.” It tells you, on average, how many bits are used to store each model parameter after quantization. A model stored in FP16 uses about 16 BPW, while a quantized model might use 8, 4, 3, or even ~2 BPW. Lower BPW means smaller model files and lower VRAM usage, but usually also increases the risk of quality loss.

For example, consider the EXL3 4.00 bpw Gemma 4 model in our test. Its quantized layers and output head occupied 5.79 GiB, while the complete model file was 7.79 GiB, a 2.00 GiB difference.

EXL3 vs GGUF Benchmarks

So how do EXL3 recipes compare with GGUF recipes in practice? To answer this question, we benchmarked google/gemma-4-12B-it. EXL3 files came from turboderp/gemma-4-12B-it-exl3, while the GGUF files came from unsloth/gemma-4-12b-it-GGUF.

Because each format required its own runtime, the speed results compare the EXL3 + ExLlamaV3/TabbyAPI and GGUF + llama.cpp stacks rather than the containers alone.

The speed test used batch size 1 with a 16k FP16 cache. Speculative and multi-token decoding were off, as was prefix caching.

Each configuration received one warm-up followed by five measured runs at 2,048 and 8,192 input tokens, with 512 output tokens, and we'll document the median and prefill range.

How We Made the Quality Test Comparable

Quality was measured with one qbench.py harness for the BF16 reference and both quantized backends. Each backend received the same token IDs, and every quant was compared with cached logits from the same BF16 run. That avoids a common problem in cross-format tests, where EXL3 perplexity comes from one harness and GGUF perplexity from another.

Quality at a Similar Model Size

Kullback-Leibler divergence, or KLD, measures how far each quantized model's token distribution moved from BF16. Lower is better. Perplexity, or PPL, is also lower-is-better, but KLD is the more direct measure here because every row is compared with the same reference logits.

EXL3 weights exclude high-precision embeddings; use File for total storage comparisons.

At a similar complete file size, the results were mixed rather than uniformly favoring one format.

Looking only at the compared quantized weights, EXL3 4.00 bpw used 5.79 GiB and returned 0.02479 KLD, while GGUF Q4_K_M used 6.62 GiB and returned 0.03985. That comparison excludes EXL3's high-precision embeddings. By complete file size, EXL3 4.00 bpw was 7.79 GiB, nearly matching GGUF Q5_K_M at 7.84 GiB; Q5_K_M had the lower KLD relative to BF16, at 0.01719 versus 0.02479.

At about 9.1 GiB, EXL3 5.00 bpw and GGUF Q6_K were effectively tied at 0.00946 and 0.00918 KLD, respectively.

Prompt Processing and Generation Speed

Prompt processing measures how quickly the model reads and processes your existing input/context before it starts replying in tok/s, this is called the prefill phase. Generation speed is how quickly the model produces new output tokens after that, also measured in tok/s.

2,048-token input and 512-token output:

8,192-token input and 512-token output:

For the two complete-file-size-matched pairs, EXL3 prefill was 2.27× and 2.21× faster at 2k, and 1.27× and 1.39× faster at 8k, respectively. At 8k, EXL3 4.00 bpw reached the first token 0.42 seconds sooner than Q5_K_M, while EXL3 5.00 bpw did so 0.62 seconds sooner than Q6_K.

llama.cpp led decode in both file-size-matched pairs. Q5_K_M reached 130.1 tokens per second against 113.1 for EXL3 4.00 bpw, while Q6_K reached 117.4 against 98.8 for EXL3 5.00 bpw. The Q6_K advantage was similar at 8k input: 115.1 against 95.3 tokens per second. The faster prefill and slower decode can cancel out over a full request. At 2k input, the median rates imply about 5.0 seconds for EXL3 4.00 bpw to deliver 512 tokens and about 5.1 seconds for Q5_K_M.

Different hardware or server configuration can influence these results, so take them with a pinch of salt.

VRAM Usage and Context Length

GGUF models loaded in four to six seconds, while EXL3 models took eight to nine seconds. The smallest configurations ended up using almost the same amount of VRAM despite their different file sizes: EXL3 3.00 bpw used 7.37 GiB after loading, compared with 7.34 GiB for IQ4_XS.

At roughly the same 7.8 GiB file size, EXL3 4.00 bpw used less VRAM than Q5_K_M. It occupied 8.71 GiB after loading, versus 9.24 GiB for Q5_K_M, a difference of 0.53 GiB. Peak usage was much closer: 9.60 GiB for EXL3 and 9.74 GiB for Q5_K_M.

The same pattern appears around the 9.1 GiB file size. EXL3 5.00 bpw used 9.89 GiB after loading and peaked at 10.76 GiB, while Q6_K used 10.52 GiB and peaked at 11.02 GiB. That gives EXL3 a 0.63 GiB advantage in loaded VRAM and a 0.26 GiB advantage at peak usage.

How to Run EXL3 Models with ExLlamaV3

This section describes the tested TabbyAPI setup used for EXL3 models on Linux.

Tested environment

The configuration was tested with:

• Linux
• Python 3.12
• NVIDIA GPU
• CUDA 12.8
• PyTorch 2.8.0
• ExLlamaV3 1.4.9

Before installation, check the ExLlamaV3 releases and select a wheel that matches the installed Python, CUDA, PyTorch, operating system, and CPU architecture.

Install ExLlamaV3 and TabbyAPI

Install huggingface_hub with transfer support, install the matching ExLlamaV3 wheel, then clone and install TabbyAPI:

Python 3.12 with venv support is required. On Ubuntu, install python3.12-venv if it is missing, then create and activate the environment below. Run the remaining installation commands inside that environment.

python3.12 -m venv .venv
source .venv/bin/activate

pip install 'huggingface_hub[hf_transfer]'

pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128

pip install https://github.com/turboderp-org/exllamav3/releases/download/v1.4.9/exllamav3-1.4.9+cu128.torch2.8.0-cp312-cp312-linux_x86_64.whl

git clone https://github.com/theroyallab/tabbyAPI
cd tabbyAPI
git checkout de76ff88
pip install -e .

The wheel shown above is specifically for Python 3.12, CUDA 12.8, PyTorch 2.8.0, Linux, and x86-64. Use a different wheel if the local environment differs.

Download an EXL3 model

Download the required EXL3 branch from Hugging Face. For this repository, the branch name identifies the quantization bitrate and recipe.

The following example downloads the 4.00bpw_mul1 variant:

hf download turboderp/gemma-4-12B-it-exl3 \
  --revision 4.00bpw_mul1 \
  --local-dir models/exl3/4.00bpw_mul1

Configure TabbyAPI

From the cloned tabbyAPI repository, create config.yml in its root. The following configuration uses a 16,384-token sequence length and an FP16 cache:

network:
  host: 127.0.0.1
  port: 5000
  disable_auth: false

model:
  model_dir: /absolute/path/to/models/exl3
  model_name: 4.00bpw_mul1
  max_seq_len: 16384
  cache_size: 16384
  cache_mode: FP16

Set model_dir to the absolute path containing the downloaded EXL3 model directories. model_name must match the directory containing the model to load.

Start TabbyAPI

Run TabbyAPI from its repository directory:

python main.py --config config.yml

By default, the configuration above exposes the API at:

http://127.0.0.1:5000

Test the API

On first start, TabbyAPI creates api_tokens.yml in its repository directory. Copy the api_key value, then send a request to the OpenAI-compatible chat completions endpoint:

curl http://127.0.0.1:5000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Explain bits per weight in two sentences."
      }
    ],
    "max_tokens": 120,
    "temperature": 0
  }'

A successful request should return a JSON response containing the generated assistant message.

Security and versioning

Keep disable_auth: false, even when TabbyAPI is bound to the loopback interface (127.0.0.1): web pages open on the same computer can also reach a local API. Use the generated API key for requests, and do not expose the instance directly to a network.

TabbyAPI is developed as a rolling-release project. Pin known-working dependency versions for reproducible deployments, and validate updates before using them in an existing environment.

Can EXL3 Run Models Larger Than GPU Memory?

Yes, but only in limited cases. ExLlamaV3 1.4.9 includes experimental CPU offload for supported mixture-of-experts (MoE) models. When you’re using a compatible model, routed experts from selected block-sparse MoE layers can be placed in system RAM instead of VRAM. This reduces the amount of GPU memory required for those expert weights.

The current CPU-offload documentation applies only to supported MoE layers. You’re limited to mul1 experts with K ≤ 8 and compatible expert biases.

However, you cannot use this layer offload to split every dense model between system RAM and VRAM in the same way you can with llama.cpp. In our dense Gemma 4 12B benchmark, the entire model remained on the GPU. If you’re trying to run a dense model that does not fit entirely in VRAM, GGUF with llama.cpp is a more suitable runtime for that.

Which Format Should You Choose?

EXL3, if the model fits entirely on the GPU and VRAM is the main constraint, especially for workloads that repeatedly process long prompts. In the two complete-file-size-matched pairs, EXL3 used slightly less peak VRAM and handled prompts faster, with lower time to first token. Quality near the same file size was mixed rather than uniformly better.

GGUF, for CPU inference or mixed CPU/GPU offload. It also has broader app support and faster generation in our llama.cpp test. GGUF is simpler to distribute as one file, and its recipes let you trade quality against memory in smaller steps.

You can run GGUF models in Atomic Chat, an offline AI app we've built, which allows you to easily download and run offline AI models, set up agentic workflows, or chat with models directly through a built-in interface.

Frequently Asked Questions

Is EXL3 Always Better Than GGUF?

No, in our Gemma 4 12B tests, EXL3 preserved more model quality per byte of quantized weights and processed prompts faster, but GGUF decoded tokens faster and supports a much broader range of hardware and applications. There are pros and cons to each format.

Can You Convert a GGUF Model to EXL3?

Technically, you could build a conversion workflow around already-quantized weights, but you should not use a GGUF file as the source for a high-quality EXL3 quantization. EXL3 quantization is intended to start from the original Hugging Face model weights. If the GGUF file is already quantized, requantizing it cannot recover the original weights and may add error.

Which Apps Support EXL3?

Your main options are ExLlamaV3 itself, TabbyAPI, and applications that integrate an ExLlamaV3-compatible backend. ExLlamaV3 is the reference implementation for the EXL3 format. It can load EXL3 models directly and also provides a Transformers integration. Model compatibility still depends on whether the architecture is supported by the current ExLlamaV3 release. TabbyAPI is the official and recommended API server for ExLlamaV3. It provides an OpenAI-compatible API and supports EXL3 models through the ExLlamaV3 backend.