Fine-tuning a language model, continuing its training on your own data, used to mean updating every one of its billions of weights. That requires enormous GPU memory, produces a full-size copy of the model for every variant, and is out of reach for most teams. LoRA, short for Low-Rank Adaptation, changed that. It lets you fine-tune a large model by training a tiny set of extra weights, often well under 1% of the original, while leaving the base model untouched.
This explainer covers what LoRA does, why it works, how QLoRA makes it even cheaper, the settings that matter, and when LoRA is (and is not) the right tool.
The problem LoRA solves
Full fine-tuning updates all parameters. For a model with billions of weights, you need memory not only for the weights but also for gradients and optimiser state, typically several times the model size. And each fine-tuned version is a complete copy: ten customer-specific variants means ten full models to store and serve.
Hu et al. (2021), "LoRA: Low-Rank Adaptation of Large Language Models," proposed a different approach based on a hypothesis: the change needed to adapt a pre-trained model to a new task is simple, in a specific mathematical sense. It has low "intrinsic rank." If that is true, you do not need to learn a full-size update for each weight matrix; you can learn a compact one.
How LoRA works
Inside a transformer, much of the computation is multiplication by large weight matrices. Take one such matrix W with dimensions *d × d*, for example 4,096 × 4,096, about 16.8 million numbers.
Instead of learning a full update ΔW of the same size, LoRA learns two thin matrices:
- A, of size *r × d*
- B, of size *d × r*
where *r*, the rank, is small: 8, 16 or 64. Their product B × A has the full *d × d* shape but is described by far fewer numbers. During fine-tuning, the original W is frozen and only A and B are trained. The layer computes:
output = W·x + (B·A)·x × (alpha / r)
d = 4096, r = 8:
full update ΔW: 4096 × 4096 = 16,777,216 parameters
LoRA (A and B): 2 × 4096 × 8 = 65,536 parameters (≈ 0.4%)B is initialised to zero, so at the start of training the model behaves exactly like the original. The scaling factor alpha / r controls how strongly the adapter influences the output.
Why this is so useful
- Much less memory to train. No gradients or optimiser state for the frozen base weights. The LoRA paper reported reducing trainable parameters for GPT-3 175B by a factor of about 10,000 and GPU memory requirements by about three times compared with full fine-tuning.
- Tiny adapters. A LoRA adapter is often a few megabytes to a few hundred megabytes, versus many gigabytes for a full model.
- Many variants, one base. Store and serve one base model with many small adapters, swapping or combining them per request.
- No inference slowdown when merged. After training, B·A can be added into W, producing a normal model with no extra computation.
- Comparable quality to full fine-tuning on many tasks, as reported in the original paper and much follow-up work, though not every task.
QLoRA: LoRA on a quantized base
Dettmers et al. (2023), "QLoRA: Efficient Finetuning of Quantized LLMs," combined LoRA with a 4-bit quantized base model. The frozen base weights are stored in a 4-bit format called NormalFloat (NF4), while the LoRA adapters train in higher precision. Additional tricks (double quantization of the scaling constants, and paged optimisers that handle memory spikes) cut memory further. The paper showed fine-tuning of a 65-billion-parameter model on a single 48 GB GPU.
For builders, QLoRA means fine-tuning 7B to 13B models is practical on a single consumer or cloud GPU. For more on what quantization costs in quality, see quantization without tears.
The settings that matter
| Setting | What it controls | Common starting point |
|---|---|---|
| Rank (r) | Capacity of the adapter | 8–16 for narrow tasks; 32–64 for broader ones |
| Alpha | Scale of the adapter's effect | Often set to r or 2 × r |
| Target modules | Which weight matrices get adapters | Attention projections; adding MLP layers often helps |
| Learning rate | Speed of training | Higher than full fine-tuning, e.g. around 1e-4 to 2e-4 |
| Dropout | Regularisation on adapter inputs | 0 to 0.1 |
Higher rank adds capacity but also more parameters and risk of overfitting. Applying adapters to more layers usually helps more than increasing rank on a few. As always, measure on a held-out eval set rather than relying on training loss.
What LoRA is good for, and what it is not
Good fits:
- teaching a consistent format, style or tone;
- narrow tasks with clear inputs and outputs: classification, extraction, domain-specific responses;
- adapting a small model to replace a large prompted one; see small models, big jobs;
- distilling a large model's behaviour into a smaller one; see distillation explained for builders;
- preference tuning on a budget; see preference tuning: RLHF and DPO.
Poor fits:
- adding lots of new factual knowledge. Fine-tuning is unreliable for teaching facts and can increase confident errors on topics the model half-learned. Use retrieval for knowledge.
- fixing problems a better prompt would fix. Try prompting and few-shot examples first; fine-tuning adds a training and maintenance burden.
- tasks where you lack good data. A few thousand clean, representative examples beat a hundred thousand noisy ones.
Serving many adapters
One of LoRA's practical advantages appears at serving time. Because adapters are small and the base model is shared, a single deployment can serve many fine-tuned variants: one per customer, per language or per task. Modern inference servers can load multiple adapters alongside one base model and apply the right adapter per request, batching requests for different adapters together.
This changes the economics of customisation. Instead of choosing between one generic model and a handful of expensive dedicated deployments, you can maintain dozens of specialised adapters at little extra cost. The trade-offs are operational: versioning adapters together with the base model they were trained on, re-training adapters when you upgrade the base model, and evaluating each adapter separately.
A minimal workflow
- Define the task and build an eval set before training.
- Collect and clean training examples in the exact format you will use at inference.
- Train a LoRA adapter on a base model, starting from rank 8 to 16 on attention and MLP layers.
- Evaluate against the base model with your best prompt, and against a larger model, on the held-out set.
- Inspect failures, improve data (usually the biggest lever), and retrain.
- Merge the adapter or serve it alongside the base model, and keep versions of data, adapter and eval results together.
Frequently asked questions
What is LoRA in machine learning?
LoRA (Low-Rank Adaptation) is a fine-tuning method that freezes a pre-trained model's weights and trains small low-rank matrices added to selected layers. It reduces the number of trainable parameters and memory needed, while often matching full fine-tuning quality.
What is the difference between LoRA and QLoRA?
QLoRA applies LoRA on top of a base model quantized to 4 bits, which cuts memory requirements further. The adapters themselves still train in higher precision.
What rank should I use for LoRA?
Ranks of 8 to 16 work well for many narrow tasks; broader tasks may benefit from 32 to 64. Applying adapters to more layers often helps more than raising the rank. Test on a held-out eval set.
Can LoRA teach a model new facts?
Not reliably. Fine-tuning is better at teaching behaviour, format and style than at adding knowledge. For facts that change or must be accurate, use retrieval.
