What four-bit quantization saves—and what it changes

The same model name can appear on a download larger than fifteen GiB and another smaller than five. “Four-bit” explains the change in weight representation, but leaves the deployment questions open: what happens to GPU cost, answer quality, speed and the number of requests the service can handle?

Quantizing weights can make local execution practical and leave more memory for active requests. It changes the numerical representation, however, and its benefit depends on the conversion method and execution path. Keep parameter count, weight precision and total serving memory separate. The language-model guide connects exact artifacts with their compatible software and memory assumptions.

What changes when weights become four-bit?

Weights are learned numerical values. Common source checkpoints store them in BF16 or FP16, with sixteen bits per value. Quantization maps a larger set of possible values into a smaller representation. Four bits provide sixteen codes; scales and sometimes other metadata determine what those codes represent in a group of weights. The Transformers quantization guide explains this distinction between codes and reconstructed values.

Think of measuring components with a less precise instrument: nearby measurements may become identical. The resulting difference matters when it changes the finished product. Quantization methods try to distribute approximation error so that it has less effect on useful model behavior; sophisticated methods do more than round every weight by one uniform rule.

Ordinary weight quantization preserves the layers and parameter count. An 8B model stays an 8B model. Pruning and distillation are different interventions. Expanding a quantized weight back into a sixteen-bit container does not restore the information lost during conversion; comparison with the original requires the original checkpoint.

Four-bit storage does not describe every computation

Weights, intermediate activations and KV cache have separate precision choices. W4A16 describes four-bit weights and sixteen-bit activations for the covered operations; accumulation depends on the kernel. A file named Q4 can still use FP16 KV. The bitsandbytes configuration explicitly separates storage from compute dtype.

The GGUF specification and llama.cpp quantization guide describe the container and presets. GPTQ and AWQ describe different error-reduction methods. QLoRA introduces NF4 in its adaptation workflow, while NVIDIA’s FP4 explanation distinguishes NVFP4 from MXFP4. These labels are not interchangeable names for one file type.

The first saving is in stored weights

A pure sixteen-to-four-bit conversion would reduce raw weight storage to one quarter. Real files also carry scales, metadata and tensors at other precisions, and the parameter count in a model name is rounded. Use the published file sizes when planning storage and downloads.

For Qwen3-8B, the BF16 row below sums the weight files in the pinned original repository. The other rows come from its pinned official GGUF repository. A GiB is 2³⁰ bytes; reductions are relative to BF16 weight files, not measured VRAM.

Here the BF16 package is about 3.26 times the Q4_K_M package size. Download volume and stored compressed weights fall accordingly. Loading time also depends on conversion and runtime preparation. A particularly valuable change occurs when the whole model now fits on one GPU, removing repeated RAM transfers or the need to split weights across cards.

Total serving memory falls by a different percentage

Weights are only one allocation. KV cache, workspace, buffers and request management still consume memory. Quantizing weights does not automatically quantize the context state.

Qwen3-8B’s configuration has 36 layers, 8 KV heads and head dimension 128. With two-byte KV values and 32,768 retained tokens, raw cache for one request is:

2 × 36 × 8 × 128 × 2 × 32,768 bytes = 4.5 GiB

For this planning example, use weight-file size as an approximation of resident weights and reserve an assumed 2 GiB for workspace. Every active request retains the full 32,768 tokens; there is no shared prefix or KV quantization.

The single-request total falls by approximately 49%, rather than the 69% reduction in weight files. Four long requests exceed a nominal 24 GB card’s budget even with Q4 weights. Adjusting context, active-request count, replicas or KV precision addresses different parts of this problem. vLLM’s quantized KV documentation describes that separate optimization; the 24 versus 48 GB article develops the hardware implications.

These numbers are memory arithmetic. They do not measure simultaneous users or establish acceptable latency, and actual engine workspace can differ from the assumed reserve.

Does four-bit mean faster?

Smaller weights can reduce memory traffic, which helps when token generation is limited by memory bandwidth. A suitable kernel must consume that representation efficiently. Unpacking, rescaling or rearranging weights can erode the gain; some paths fuse reconstruction into the computation. Marlin is an example of kernel work aimed at making four-bit weights useful for speed. Its results belong to its tested kernel and workload, not automatically to an entire application.

Reading the prompt, or prefill, differs from generating successive tokens, or decode. Long inputs and larger batches change weight reuse and the amount of computation. Long contexts also increase KV traffic. A method that improves one user’s decode rate may not improve time to first token or total service throughput by the same factor.

Record input and output lengths, active requests and engine configuration. Compare first-token latency, continuation speed and successfully completed requests separately. W4A16 may read compressed weights but perform arithmetic at higher precision; native low-precision hardware instructions are another route. The vLLM hardware support matrix illustrates why model, numeric format, kernel and engine version must be matched.

Quality changes are task-specific

There is no universal four-bit quality-loss percentage. Architecture, model size, quantization method, affected tensors and the task all influence the result. Evaluating Quantized Large Language Models examines weights, activations and KV across several families and finds uneven sensitivity. It does not establish one loss figure for every subsequent model or conversion.

Two variants may sound equally fluent while differing on an identifier, a contractual exception or a tool argument. The useful comparison therefore scores task completion: field values for extraction, support from the document for RAG, executed tests for code, and correct tool/argument selection for agents. Valid JSON establishes syntax, not correct values. Different wording alone is not deterioration, especially with stochastic generation.

Evaluate the language actually used by the service. English results or a multilingual average do not answer how a particular quantized checkpoint handles local names, number formats or specialist terminology. Keep representative difficult examples alongside common ones, so that rare expensive errors remain visible.

A larger four-bit model or a smaller higher-precision one?

Quantization can put a larger model within the same memory budget. That helps when the smaller candidate lacks the required capability and the quantized larger one retains it. But more parameters can still mean more work per token, and compression does not erase that computational difference.

Compare two deployable configurations on the same task and hardware, rather than comparing file sizes alone. If a specialist or small model already passes the quality threshold, spare memory can support more requests or other pipeline components. If the task requires stronger reasoning, a larger quantized model may be worthwhile. The model-size guide explains how to form that comparison without assuming a universal size ranking.

Trained quantization and file conversion differ

ParetoQ reports HellaSwag scores of 53.3 for MobileLLM-350M BF16 and 53.5 for its 4-bit variant; reported Wiki perplexity falls from 10.5 to 10.3. These are trained and fine-tuned ParetoQ variants. The result does not show that converting an arbitrary checkpoint to 4-bit GGUF improves its quality. Compare the training recipe and quantization method alongside architecture and bit width; the study is not a measured memory or speed guarantee for a desktop runtime.

PTQ, QAT and QLoRA solve different problems

Post-training quantization, or PTQ, converts an already trained model. It may use a small calibration collection to select a better numerical approximation; GPTQ and AWQ are examples. Calibration data are not equivalent to teaching the model the organization’s knowledge. If a suitable published artifact exists, a deployer need not repeat the conversion.

Quantization-aware training, or QAT, includes the effect of reduced precision during training or adaptation so that parameters can adjust. PyTorch’s QAT explanation presents an implementation and evaluation. Simulating quantization error does not mean every training operation and optimizer state is four-bit.

QLoRA keeps quantized base weights frozen and trains a limited set of adapters. It targets cheaper adaptation, rather than full training of all parameters in four-bit arithmetic. Decide first whether the remaining error requires training or whether retrieval and better inputs address it; the RAG and tuning comparison separates those cases.

Choose a variant that removes the actual constraint

Start with a credible artifact whose base checkpoint, quantization, publisher, license and execution path are identifiable. Its tokenizer and chat template must match the model. If converting again, use an appropriate original checkpoint: successive conversions from one lossy representation to another can carry earlier errors forward.

The catalog identifies artifacts, while the software table covers execution. A controlled quality comparison should hold inputs and generation settings steady; a final system comparison can tune each route separately if it reports those differences.

The economic benefit comes from using fewer resources for accepted work, or completing more accepted work with the same resources. Conversion effort, maintaining artifacts, engineering time and failed answers also contribute. A smaller file is valuable when the released memory or reduced traffic produces a usable improvement while preserving the task’s quality requirement.