Common causes of out-of-memory errors when training AI models

Learn why GPU out-of-memory errors occur during AI model training, their common causes, and practical fixes to reduce VRAM and RAM usage without upgrading hardware.

Common causes of out-of-memory errors when training AI models

I am sure when you are training your AI model you have seen the “out-of-memory” error at least once. But here is one thing that is kept on hanging while this error. It tells you exactly what happened: ‘Your GPU hit its memory limit’, but what's left hanging is “WHY?”, “WHERE?”, “WHAT?”. 

It fires on the allocation that broke the camel's back, not on the one that caused the problem. So the line number in the traceback is rarely the real culprit. From a business point of view as well, knowing all the answers is really important for you. 

Keep reading and you will know it. I have covered everything here in a quick to the point manner.

What does an out-of-memory error actually mean during training?

It means the training job asked for more memory than the device had free at that moment. Most of the time that device is the GPU, and the memory is VRAM. But not always, and that difference matters more than people expect, which I will come back to below.

The error appears at the point where memory peaks. A job can run fine for hours and then crash on one unusually long batch.

Why does training need so much more memory than the model's size?

Because the model's weights are only one of four things being stored. During training the GPU also holds gradients, optimizer states and activations.

Here is the usual accounting for mixed-precision training with the Adam optimizer:

What is stored

Rough size per parameter

What it scales with

Weights (fp16 or bf16)

2 bytes

Model size

Gradients

2 bytes

Number of trainable parameters

fp32 master copy of weights

4 bytes

Model size

Adam momentum and variance

8 bytes

Number of trainable parameters

Activations

Varies a lot

Batch size, sequence length, model depth

That adds up to roughly 16 bytes per parameter before a single activation is counted. Some frameworks count a little differently and land closer to 18, so treat 16 as a floor.

Can a 7B model be fully fine-tuned on one 80GB GPU?

Not with plain mixed-precision Adam. Seven billion parameters at 16 bytes each is about 112GB, and that is before activations. A smaller 1B model already needs around 16GB just to exist in the training loop.

That is why full fine-tuning at this size needs sharding across GPUs, an 8-bit optimizer, or a method like LoRA that trains far fewer parameters.

Now that we cleared up some foundation, let's get to the main agenda here. The causes for the memory error so you can avoid it happening in your next AI model training.

  1. Batch size or sequence length too big

 

This is the most common cause, and it hits activations. Activations are the intermediate values the model saves during the forward pass so it can compute gradients later.

  • Double your batch size and you roughly double the activations

  • Longer sequences make attention memory grow much faster, because standard attention scales with the square of the sequence length

  • Deeper and wider models store more of them at every layer

The fixes are a smaller micro-batch with gradient accumulation to keep your effective batch size, gradient checkpointing, and an efficient attention implementation such as FlashAttention. Gradient checkpointing recomputes activations instead of storing them, and it typically makes training about 20 to 30 percent slower.

  1. Optimizer eating your memory

Adam keeps two extra values for every trainable parameter, and they sit in higher precision than the model itself. For many training runs, optimizer states are the largest single block of memory after the weights.

An 8-bit optimizer cuts this sharply for a small precision cost. LoRA helps too, since fewer trainable parameters means fewer gradients and fewer optimizer states.

There is one catch worth knowing. LoRA and quantization shrink the parts that scale with parameters. They do nothing for activations. If you are still out of memory after switching to LoRA, the batch size or sequence length is probably your real problem.

  1. Accidentally holding on to old tensors

A leak looks like memory that climbs a little every step until the job dies hours in. The classic cause is keeping tensors that still carry their computation graph.

  • Adding loss to a running total instead of loss.item() keeps every step's graph alive

  • Storing predictions for later without detaching them has the same effect

  • Running evaluation without turning off gradient tracking uses far more memory than it needs to

If your memory use grows steadily over time instead of jumping, look for this first.

  1. Fragmentation, or another process

Sometimes the GPU has free memory, just not in one piece large enough for the allocation. A good sign is when reserved memory is far above allocated memory. In PyTorch, setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True often helps.

Also run nvidia-smi. A forgotten notebook kernel or an old process can quietly hold gigabytes that your new job needs.

  1. RAM or VRAM that ran out

This is the ram vs vram mix-up, and it trips up many people. RAM is the computer's main memory, and VRAM is the GPU's own memory. They run out in different ways and need different fixes.

What you see

Which memory

Usual cause

CUDA out of memory. Tried to allocate...

VRAM

Batch size, activations, model states

Process just says Killed, with no traceback

System RAM

The operating system's out-of-memory killer stepped in

MemoryError in Python

System RAM

A dataset loaded fully into memory

On the system RAM side, the usual suspects are loading an entire dataset into memory at once and running many data loader workers that each hold their own copy of the data. Lower the worker count, stream the data from disk, and check memory with free -h or your system monitor.


How do you fix an out-of-memory error, in the right order?


Start with the cheap changes and move down the list only if you need to.

Fix

What it shrinks

Trade-off

Smaller micro-batch plus gradient accumulation

Activations

More steps, slightly slower

Mixed precision (bf16)

Weights, gradients and activations

Needs a little numerical care

Gradient checkpointing

Activations

Roughly 20 to 30 percent slower

Shorter sequences or efficient attention

Activations

May change what the model sees

8-bit optimizer

Optimizer states

Tiny precision loss

LoRA or QLoRA

Gradients, optimizer states and weights

Does not reduce activations

Sharding across GPUs, or a bigger card

Everything

Setup effort, or more cost

Before changing anything, confirm which memory ran out. Fixing VRAM when RAM was the problem wastes an afternoon.


Conclusion

Out-of-memory errors feel random, but they have a short list of causes: model states, activations, leaks, fragmentation, and the wrong memory entirely. Work out which one you are facing before you change settings. Most jobs that crash can be rescued with a smaller micro-batch, gradient accumulation and checkpointing, without buying anything bigger.


Frequently asked questions

Does torch.cuda.empty_cache() fix out-of-memory errors? 

Rarely. It releases cached blocks PyTorch is holding but not using, which can help with fragmentation. It cannot help when your model, gradients and activations genuinely do not fit.

Will adding more GPUs fix an out-of-memory error? 

Only if you shard the model. Plain data parallel training keeps a full copy of the model on every GPU, so each card needs the same memory as before. Techniques that split optimizer states and weights across GPUs, such as ZeRO and FSDP, are what reduce the per-card load.

How can I estimate memory before I start? 

Take about 16 bytes per trainable parameter for weights, gradients and optimizer states, then add activations. After that, run a few steps with a small batch and read the peak memory figure, rather than trusting any formula completely.