NVFP4 is an NVIDIA-proprietary 4-bit floating-point data format designed specifically to accelerate inference and training in Large Language Models (LLMs). This data format compresses the model size to a quarter of its 16-bit size, while keeping accuracy close to the original.
If you run models locally, you need to know about NVFP4, as it allows you to install and run much larger models much faster than you could have otherwise, but you need the right hardware to do this.
Quick answer
Large language models are becoming too large to fit comfortably in GPU memory. Running them faster therefore depends on storing their numerical weights in fewer bits without losing much accuracy. NVFP4 is NVIDIA's solution to that problem: a 4-bit floating-point number format (E2M1) for storing a model's weights at very low precision. It makes a model roughly 3.5 times smaller than FP16 and 1.8 times smaller than FP8, and can speed up inference by up to about 3 times over FP8, while typically keeping accuracy loss on large language models under 1 percent.
What is NVFP4?
NVFP4 is a 4-bit floating-point data format developed by NVIDIA. It is used to store the numerical values inside a neural network, particularly the model's weights and the intermediate values it produces while running. The "NV" in the name refers to NVIDIA, the company that created it. NVFP4 was first introduced in June 2025, and over 2025 and 2026 NVIDIA added support for it to the major inference engines — vLLM, SGLang, and TensorRT-LLM.
NVFP4 was designed for NVIDIA's Blackwell GPU architecture, whose fifth-generation Tensor Cores can perform 4-bit floating-point math directly, so a model stored in NVFP4 both takes less memory and runs faster.
How the NVFP4 format works: quantization and scaling
The defining challenge of a 4-bit floating-point format is that four bits simply cannot represent enough numbers to store a modern language model directly. An NVFP4 value can represent only 15 distinct numbers, ranging from 0 to ±6:
0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6
A large language model, however, contains billions of weights whose values vary enormously in magnitude. Some are very close to zero, while others are hundreds or thousands of times larger. If every weight had to fit directly into the fixed range of E2M1, the rounding error would be so large that the model would lose much of its accuracy.
Instead, NVFP4 separates the information needed to reconstruct a weight into two parts. The first is the 4-bit value itself, chosen from the 15 values available in the E2M1 format. The second is one or more scaling factors shared with neighbouring weights. Rather than storing the full value directly, the format stores a small relative value together with enough information to recover its original magnitude.
When the model runs, Blackwell's FP4 Tensor Cores combine the stored 4-bit values with their scaling factors to reconstruct an approximation of the original weights. Because this reconstruction is performed directly in hardware, the additional scaling introduces little overhead while allowing the model to retain most of its original accuracy.
This combination of low-precision values and shared scaling factors is what distinguishes NVFP4 from a simple 4-bit floating-point format. The scaling system effectively extends the useful numerical range without increasing the storage required for every individual weight.
NVFP4 Benefits
Reducing weight precision from sixteen bits to four bits decreases the amount of memory required to store and process a model. This has three main effects:
- Smaller models. An NVFP4 model is approximately 3.5× smaller than its FP16 equivalent and about 1.8× smaller than FP8.
- Higher inference throughput. Smaller weights reduce memory traffic, allowing more of the model to be processed each second. On supported hardware, inference throughput can be up to three times higher than FP8.
- Minimal accuracy loss. Large models typically retain more than 99% of their original accuracy after quantization.
Smaller models
The most immediate effect of NVFP4 is lower memory usage. Compared to FP16, the format reduces weight storage by roughly 3.5×, and by about 1.8× compared to FP8.
For example, if you take a 70-billion-parameter model, this determines which hardware can load it entirely in VRAM:
- FP16: ~140 GB of weights — requires at least two 80 GB data-center GPUs.
- FP8: ~70 GB — exceeds the memory capacity of current consumer GPUs.
- NVFP4: ~39 GB of weights — fits on two RTX 5090s (32 GB each) or a single B200/B300.
The reduction is even more significant for smaller models.
A 32B model stored in NVFP4 fits on a single RTX 5090, whereas its FP16 equivalent does not.
NVFP4 can also reduce KV cache storage by up to 50% compared to FP8, increasing the maximum context length that fits within the same amount of VRAM.
Faster inference
Large language model inference is often limited by memory bandwidth rather than arithmetic throughput. Because NVFP4 stores fewer bits per weight, less data must be transferred between memory and the GPU's compute units, increasing throughput.
On Blackwell GPUs, which include native FP4 tensor cores, NVIDIA reports up to 3× higher inference throughput than FP8 and up to 4× greater matrix multiplication throughput than BF16.
Lower memory traffic also reduces latency and energy consumption per generated token, making NVFP4 a key part of Blackwell's efficiency improvements.
Minimal accuracy loss
Quantization reduces numerical precision, but NVFP4 is designed to minimize its effect on model quality.
NVIDIA reports less than 1% accuracy loss when converting large models from FP8 to NVFP4, with 95–99% of full-precision accuracy retained depending on model size. Results published for DeepSeek-R1 show only small differences across seven evaluation benchmarks:
| Benchmark | FP8 | NVFP4 |
|---|---|---|
| MMLU-Pro | 85 | 84 |
| GPQA Diamond | 81 | 80 |
| AIME 2024 | 89 | 91 |
| LiveCodeBench | within ~1% | within ~1% |
The slightly higher AIME 2024 score is generally attributed to normal variation introduced by quantization rather than a systematic improvement.
Accuracy retention also depends on model size: models with around 70 billion parameters or more typically retain about 99% of their original accuracy, while models in the 7B–14B range more commonly retain 95–98%. Larger models are generally more tolerant of quantization because they contain greater representational redundancy.
Who can use NVFP4?
NVFP4 requires dedicated FP4 tensor cores and is supported only on NVIDIA's Blackwell architecture. Current Blackwell hardware with NVFP4 support includes:
- Data center: B200, B300, GB200, GB300, and GB300 NVL72 systems
- Workstation: RTX PRO 6000 Blackwell
- Consumer: GeForce RTX 50-series GPUs, including the RTX 5090
- Desktop AI: DGX Spark
Previous architectures, including Hopper (H100, H200), Ampere (A100), Ada Lovelace (RTX 40-series), and earlier generations, do not support NVFP4. Although these GPUs can often load NVFP4-quantized models by converting the weights to a supported format, they cannot execute FP4 operations natively and therefore do not receive the memory bandwidth or throughput advantages provided by Blackwell's FP4 tensor cores.
How to run NVFP4 models
NVIDIA publishes official NVFP4 checkpoints on Hugging Face under the nvidia/ namespace. Examples include:
nvidia/DeepSeek-R1-NVFP4nvidia/DeepSeek-V4-Flash-NVFP4nvidia/Qwen3-30B-A3B-NVFP4nvidia/Qwen3.5-397B-A17B-NVFP4nvidia/Qwen3.6-27B-NVFP4nvidia/Gemma-4-31B-IT-NVFP4nvidia/Kimi-K2.6-NVFP4nvidia/GLM-5.2-NVFP4nvidia/Llama-3.1-8B-Instruct-NVFP4
Additional models are released over time, you can see the latest checkpoints by searching Hugging Face for NVFP4. If an NVFP4 version of a model is unavailable, you can also create one from the original weights using NVIDIA Model Optimizer or LLM Compressor, both of which quantize a model using a calibration dataset.
NVFP4 support in vLLM, SGLang, TensorRT-LLM, and ComfyUI
Running an NVFP4 model also requires an inference engine with native FP4 support. Current options include:
- vLLM — NVFP4 support for production and local model serving
- SGLang — high-throughput NVFP4 inference
- TensorRT-LLM — NVIDIA's reference runtime for NVFP4
- ComfyUI — NVFP4 image and video generation models such as FLUX and LTX
However, if you're using Atomic Chat, you don't need to set up anything manually to run NVFP4 models. Atomic Chat includes built-in support for NVFP4 models and features an integrated Hugging Face browser, so you can search, download, and launch NVFP4 models directly from within the app in a single click.
And if you want to check the latest supported NVFP4 models, you can also browse the current list on Hugging Face at huggingface.co/models?search=NVFP4.
NVFP4 vs MXFP4 vs FP8 (and GGUF)
Once a model is available in NVFP4, the remaining question is whether it is the most suitable format for a given system. This depends primarily on what type of hardware you have available and what balance between memory usage, performance, and accuracy you want to achieve. Consider the table below:
| Attribute | NVFP4 | MXFP4 | FP8 |
|---|---|---|---|
| Bits per value | 4 | 4 | 8 |
| Block size | 16 | 32 | — |
| Scale format | E4M3 (8-bit float) + FP32 | E8M0 (power-of-two) | Per-tensor |
| Scale overhead | 0.5 bits/weight | 0.25 bits/weight | — |
| Developed by | NVIDIA | Open Compute Project (OCP) | Industry standard |
| Hardware | Blackwell only | Multiple accelerators | Hopper, Blackwell, and others |
| Typical accuracy | ~95–99% of full precision | Slightly lower | ~99%+ |
| Best suited for | Maximum efficiency on Blackwell | Portable 4-bit deployments | Higher precision with modest compression |
NVFP4 vs FP8:FP8 stores each value in eight bits instead of four. It typically preserves nearly all of the original model accuracy, but requires roughly twice as much memory as NVFP4. FP8 is often preferred when a model is particularly sensitive to quantization or when memory capacity is less constrained.
MXFP4 is an open 4-bit floating-point format standardized by the Open Compute Project. Unlike NVFP4, it is not tied to a single hardware vendor and is designed to run across multiple accelerator architectures. Its coarser scaling scheme generally results in slightly lower accuracy than NVFP4. In NVIDIA's pretraining experiments, models trained in MXFP4 required approximately 36% more training tokens to reach the same quality as equivalent NVFP4 models.
GGUF uses a different family of quantization formats, including variants such as Q4_K_M, developed for the llama.cpp ecosystem. GGUF is designed for broad compatibility across CPUs and GPUs from multiple vendors. On Blackwell GPUs, NVFP4 generally provides higher throughput and better accuracy than similarly sized GGUF quantizations because it is executed directly on dedicated FP4 tensor cores. On CPUs and pre-Blackwell GPUs, however, GGUF is usually the more practical choice.
PTQ vs QAT
Another important factor to understand when choosing a model or creating a checkpoint is how the model was converted to NVFP4. You'll see that on Hugging Face, checkpoints are labeled by one of two methods.
- Post-training quantization (PTQ) converts an already-trained model to NVFP4 in a single pass, using a small calibration dataset and no retraining. It is fast and requires no access to the original training pipeline.
- Quantization-aware training (QAT) builds the 4-bit arithmetic into training itself, so the model learns to compensate for the lower precision. It costs far more compute but recovers additional accuracy.
Why NVFP4 is more accurate
The accuracy advantage in the comparison above comes from two design decisions inside the format:
- How large its blocks are.
- How precisely it stores each block's scale factor.
Recall that NVFP4 groups weights into blocks of 16 and gives each block its own scale.
MXFP4 uses blocks of 32 — twice as large. A smaller block means each scale factor is responsible for fewer weights and can fit them more tightly, so a single unusually large weight distorts 16 of its neighbors rather than 32.
The NVFP4 scale factor itself is also very precise. NVFP4 stores it as a true 8-bit floating-point value (E4M3), which can take fine-grained, fractional steps. MXFP4 stores it as a power-of-two value (E8M0), which can only scale by whole doublings — ×2, ×4, ×8 — and therefore often overshoots the ideal scale.
Taken together, NVIDIA reports that the finer blocks and finer scaling produce about 88 percent lower quantization error than power-of-two scaling, which is why NVFP4 holds accuracy better than MXFP4 at the same 4-bit budget.
Frequently asked questions
Is NVFP4 better than MXFP4?
For accuracy, generally yes. NVFP4's 16-value blocks and 8-bit E4M3 scale factor recover more accuracy than MXFP4's 32-value blocks and power-of-two scale. MXFP4 is the more portable format as an open standard, but on Blackwell hardware NVFP4 is the stronger choice where quality is the priority.
Does NVFP4 work on the H100 or RTX 40-series?
No. NVFP4 requires Blackwell GPUs such as the B200, B300, RTX 50-series, RTX PRO 6000, and DGX Spark. Hopper (H100, H200) and Ada (RTX 40-series) lack the FP4 tensor cores it depends on.
How much accuracy does NVFP4 lose?
Typically under 1 percent for large models, recovering 95–99 percent of full-precision accuracy overall. Larger models lose the least; small models in the 7B–14B range lose somewhat more.
What is the value range of NVFP4 (is it -8 to 7.5)?
The "-8 to 7.5" range comes from 4-bit integer formats and does not apply to NVFP4. As a floating-point format, a single NVFP4 value (E2M1) spans the grid 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6, with a maximum magnitude of ±6. Two-level scaling then maps this grid onto the model's actual, much wider range of weights, so the effective range depends on the scale factors applied.
What NVFP4 models are available to download?
NVIDIA hosts many on Hugging Face under the nvidia/ namespace, including DeepSeek-R1 and V4, Qwen3, Gemma 4, Kimi K2, GLM-5, and Llama 3.1 variants. Searching Hugging Face for "NVFP4" returns the current list.
Is NVFP4 suitable for local use?
On a Blackwell GPU, yes. It allows larger models to fit in less VRAM with minimal quality loss, which suits running models on a single desktop or workstation card.
