LoRA vs. QLoRA — Parameter-Efficient Fine-Tuning

Clearing up the hierarchy, before we go on

It’s worth being precise here, because this is a point where it’s easy to get confused: LoRA and QLoRA are fine-tuning — not separate alternatives to it, but a family of strategies inside it.

Fine-tuning (adjust a pretrained model with gradient descent and your own data)
│
├── Full fine-tuning
│   → updates ALL of the model's weights (Chapter 4)
│
└── PEFT — Parameter-Efficient Fine-Tuning
    → updates only a small fraction of new parameters
    │
    ├── LoRA (low-rank matrices)
    └── QLoRA (LoRA + 4-bit quantization)

“Fine-tuning” is the name of the general process. “Full” and “PEFT” are two strategies for doing it, with very different trade-offs of cost, memory, and overfitting risk — not two separate categories where one “is” fine-tuning and the other is not.

The problem it solves, in concrete numbers

We already calculated the cost of full fine-tuning in the previous chapter: simultaneous memory for weights + gradients + optimizer state of every parameter — 692MB to train a model whose weights alone weigh 231MB, and hundreds of gigabytes or more for models at real scale. LoRA attacks exactly this root of the problem: if you only train a tiny fraction of parameters, you only need gradients and optimizer state for that fraction — the rest of the model stays frozen, with no gradient to compute and no state to keep.

A useful analogy for the overfitting risk it avoids, on top of the memory savings: imagine a class that studied math all year. Right before the exam, the teacher emphasizes long division a lot. On the exam, many students get so obsessed with that that they neglect other equally important equations. This is what overfitting can do to an LLM in full fine-tuning — emphasize the new so much that the rest of what the model already knew how to do well gets neglected.

How LoRA works, with a numerical example

Instead of training every parameter, LoRA freezes them and trains, instead, a low-rank adaptation matrix — called that because it contains far fewer parameters than the original matrix it is adapting.

The savings with real numbers: imagine a typical projection matrix inside an attention block (remembering Chapter 4 of Part I), of size 4096×4096 — that is, 16,777,216 parameters in that single matrix. Fully fine-tuning it means computing and storing gradients and optimizer states for those ~16.7 million numbers.

LoRA, instead, replaces it with two much smaller matrices: one 4096×8 and another 8×4096 (where 8 is the chosen “rank” — a hyperparameter you control). Those two matrices together have 4096×8 + 8×4096 = 65,536 parameters. That is 256 times fewer parameters to train, in that matrix alone — and the same savings replicate in every matrix of the model where you apply LoRA.

Once trained, those two small matrices are multiplied together (recovering a 4096×4096 matrix, the same size as the original) and added to the original frozen weights — the model applies the new learning without the original matrix having been touched directly at any moment during training.

Advantages of LoRA:

Far fewer parameters to train (256x fewer in our example, in a single matrix)
Lower overfitting risk, because almost all of the original knowledge stays frozen and intact
More agile training and a much smaller memory footprint
Flexible adapters — you can apply LoRA only to certain matrices in the model (for example, only to the attention projections) and leave the rest with no adapter at all

How QLoRA works

QLoRA is an extension of LoRA with an additional advantage: it uses even less memory, thanks to quantization (the “Q” in the name).

What is quantization? (a compression technique)

Quantization is, in essence, a model-compression technique. It reduces the amount of memory occupied by the numbers that represent the model’s weights.

A simple analogy: imagine you are measuring distances with a ruler.

Using float32 (32 bits per number) is like having a ruler with 1-millimeter marks. It is very precise, but you need a lot of space to store every measurement.
Using int4 (4 bits per number) is like having a ruler with 1-centimeter marks. You lose precision (you cannot measure millimeter differences), but the ruler is much smaller and easier to carry.

In real memory terms, per model parameter:

Format	Bits	Bytes per parameter
float32	32 bits	4 bytes
float16	16 bits	2 bytes
int8	8 bits	1 byte
NF4	4 bits	0.5 bytes

With a 7-billion-parameter model, that table is not abstract: its weights alone, not counting metadata or runtime memory, occupy 28GB in float32; the ideal 4-bit representation occupies barely 3.5GB. In a real implementation, NF4 also requires quantization constants and metadata, but the difference in scale is still the one that can let you load the model where it would not fit before.

Why QLoRA quantizes in order to train, not only to serve

We already saw quantization applied to inference in the Inference optimization chapter of Part III — there the goal was to cheapen the cost of serving an already-trained model. Here the problem is different: QLoRA quantizes the frozen base model specifically so it fits in memory during training — not so it can be served afterward. They are two mathematically related problems, but with different goals.

During the forward and backward pass, QLoRA dequantizes the needed blocks to a higher-precision compute format and propagates the gradients to the LoRA adapters. The quantized weights of the base model stay frozen: they receive no updates.

Why NF4 is not “just int4 by another name”

Using the same ruler analogy: int4 is like a ruler with marks spaced uniformly every centimeter, no matter where you measure. The weights of a trained neural network, however, are not distributed uniformly — they tend to concentrate near zero, following approximately a normal (bell-curve) distribution.

NF4 is like a smarter ruler: it places its marks where you are actually going to measure — closer together in the regions with the highest concentration of weights, and farther apart where there are few values. The QLoRA paper itself describes it as “information-theoretically optimal for normally distributed weights”. It is not simply int4 by another name: it is a 4-bit data type designed to take advantage of the specific statistical distribution of normalized weights.

Quantizing still introduces rounding error; NF4 does not magically preserve every original value. Its advantage is assigning the 16 representable values in a way that reduces that damage under the distribution it was designed for.

The real tools behind this

bitsandbytes: the library that implements NF4 and that QLoRA uses directly — it handles quantization of the frozen weights and the arithmetic needed for training to work on top of them.
GPTQ: a post-training quantization technique (for inference, not for this training case) that uses a second-order information approximation of the loss function to decide how to quantize the weights with the least possible damage to accuracy.
AWQ (Activation-aware Weight Quantization): identifies the most relevant weights from the model’s real activations with calibration data and adjusts the quantization to protect them better.

GPTQ and AWQ are techniques you will encounter much more in the inference context (which is why they are also mentioned in Part III) — for the efficient training that is this chapter’s topic, NF4 via bitsandbytes is specifically what makes QLoRA possible.

The real result: what the original paper showed

The original QLoRA paper, published by Dettmers et al. in 2023, showed that it was possible to fine-tune a 65-billion-parameter model on a single 48GB GPU, preserving 16-bit fine-tuning performance on the evaluated tasks. The resulting family of models was called Guanaco.

That result did not come from swapping float16 for NF4 alone. QLoRA combined three innovations:

NF4: the 4-bit format designed for weights with an approximately normal distribution.
Double Quantization: also quantizes the constants used to quantize the weights, reducing the extra memory those metadata require.
Paged Optimizers: uses paged memory to handle usage spikes during training and avoid out-of-memory errors.

The 48GB figure corresponds to the paper’s concrete experiment; it does not mean that any 65B model, dataset, or modern configuration will automatically fit in that memory. Sequence length, batch, adapters, and the implementation also matter.

Advantages of QLoRA:

Requires less memory than LoRA without quantization
Keeps LoRA’s low overfitting risk
In the original experiments it preserved 16-bit fine-tuning performance on the evaluated tasks
Lets you fine-tune models that, without quantization, would not fit on a single GPU

Which should you choose?

LoRA is effective and can be used on its own. QLoRA adds extra memory efficiency and introduces quantization error in the frozen weights. The main practical consideration is your available GPU memory: if the base model does not fit at conventional LoRA precision, QLoRA can make training viable. If both options fit, the decision should be made with evals on the real task — do not assume in advance that one will always be more accurate than the other.