How to Fix CUDA Out of Memory Errors: Causes and Effective Solutions

You launch your training script or load a large model, and a few seconds later your screen fills with a red error: “RuntimeError: CUDA out of memory.” It is one of the most common problems in deep learning, and it hits beginners and experienced engineers alike. The good news is that it is almost always fixable without buying new hardware.

This guide explains what the error really means, why it happens, and exactly how to fix it, step by step. Whether you are training a neural network, fine-tuning a language model, or simply running a large model for inference, you will find a solution that fits your setup.

How to fix CUDA out of memory error guide cover image — A complete guide to fixing the CUDA out of memory error

What Does “CUDA Out of Memory” Mean?

CUDA is the technology that lets frameworks such as PyTorch and TensorFlow run calculations on an NVIDIA graphics card. Every graphics card has a fixed amount of dedicated memory, called VRAM. When your program asks for more VRAM than is currently free, CUDA cannot hand it over and the program stops with an out of memory error.

A typical PyTorch message looks like this:

RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB
(GPU 0; 8.00 GiB total capacity; 5.90 GiB already allocated;
0 bytes free; 6.20 GiB reserved in total by PyTorch)

Read the numbers carefully, because they tell you a lot. “Total capacity” is the size of your card, “already allocated” is memory used by your tensors, and “reserved” is memory PyTorch has set aside in its cache. If reserved is much larger than allocated, fragmentation may be part of your problem.

Why Does the Error Happen?

GPU memory is consumed by several things at the same time, not just the model itself. During training, the main consumers are:

Model weights: the parameters of the network.
Gradients: one value per parameter, created during the backward pass.
Optimizer states: optimizers such as Adam store extra values for every parameter, which makes them surprisingly expensive.
Activations: intermediate results saved for the backward pass. These grow with batch size, sequence length, and image resolution.
CUDA overhead and cache: the CUDA context, temporary buffers, and memory that PyTorch keeps reserved for reuse.

Diagram showing where GPU memory goes during training: weights, gradients, optimizer states and activations — With FP32 weights and Adam, each parameter costs roughly 16 bytes before activations are counted

As a quick example, a model with one billion parameters trained in full FP32 precision with Adam needs around 16 GB just for weights, gradients, and optimizer states, before a single activation is stored. That is why large models run out of memory so quickly on consumer cards.

Step 1: Diagnose the Problem First

Before changing anything, find out what is using your memory. Open a terminal and run:

nvidia-smi

This shows total memory, used memory, and every process that is holding the GPU. You will often discover that an old Python session, a Jupyter notebook, or another application is quietly occupying several gigabytes. Inside PyTorch you can also check your own usage:

import torch

print(torch.cuda.memory_allocated() / 1024**3, "GB allocated")
print(torch.cuda.memory_reserved() / 1024**3, "GB reserved")
print(torch.cuda.memory_summary())

If the numbers are higher than expected, a leak or a leftover process is likely. If they look normal, your model or batch is simply too large for the card.

Six step order for fixing CUDA out of memory errors, from freeing memory to upgrading the GPU — Start with the easy fixes and move down the list only if you need to

Fix 1: Free Up GPU Memory

The simplest fix is to make sure nothing else is using the card. Close other notebooks, stop background training jobs, and shut down programs that use the GPU, such as video editors or games. Browsers with hardware acceleration can also take a few hundred megabytes.

Inside a long running session, you can release cached memory that is no longer needed:

import gc, torch

del model, optimizer      # remove references you no longer need
gc.collect()
torch.cuda.empty_cache()

Keep in mind that empty_cache() only returns memory that is not in use. It cannot free tensors that your code still references.

Fix 2: Reduce the Batch Size

This is the most effective and most common solution. Activation memory grows roughly in proportion to batch size, so cutting the batch size in half can cut that part of memory nearly in half as well. If a batch of 32 fails, try 16, then 8, then 4. Smaller batches do not make your model worse as long as you adjust the learning rate sensibly or use gradient accumulation, described below.

Fix 3: Use Mixed Precision Training

Mixed precision stores most values in 16-bit instead of 32-bit numbers. This reduces activation memory significantly and also speeds up training on modern NVIDIA cards. In PyTorch it takes only a few lines:

scaler = torch.cuda.amp.GradScaler()

for inputs, labels in loader:
    optimizer.zero_grad()
    with torch.autocast(device_type="cuda", dtype=torch.float16):
        outputs = model(inputs.cuda())
        loss = criterion(outputs, labels.cuda())
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

If your card supports it, BF16 is often more stable than FP16 and does not need the gradient scaler.

Fix 4: Use Gradient Accumulation

Gradient accumulation lets you train with a large effective batch size while only keeping a small batch in memory at a time. You run several small forward and backward passes, add up the gradients, and only then update the weights.

accum_steps = 4
optimizer.zero_grad()

for i, (inputs, labels) in enumerate(loader):
    loss = criterion(model(inputs.cuda()), labels.cuda())
    (loss / accum_steps).backward()
    if (i + 1) % accum_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

With a batch size of 4 and 4 accumulation steps, you get the same effective batch size as 16, at a fraction of the memory cost.

Fix 5: Turn On Gradient Checkpointing

Gradient checkpointing trades a little speed for a lot of memory. Instead of storing every activation, the framework stores only some of them and recomputes the rest during the backward pass. Training becomes roughly 20 to 30 percent slower in many cases, but activation memory can drop dramatically. Many libraries expose this as a single option, for example enabling it on a transformer model with one method call or one setting in the training configuration.

Bar chart comparing approximate memory savings from batch size, mixed precision, checkpointing, 8-bit optimizers and quantization — Approximate savings vary by model and settings, but this shows where the biggest wins usually are

Fix 6: Shrink the Model or the Optimizer

If the model itself is too big, there are several ways to reduce its footprint:

Use a smaller model variant. A base or small version of the same architecture often performs well enough for your task.
Use an 8-bit optimizer. Optimizer states can take as much memory as the weights themselves, and 8-bit versions reduce this sharply.
Quantize the model. Loading a model in 8-bit or 4-bit precision can cut weight memory by roughly half or by about three quarters compared with 16-bit.
Use parameter-efficient fine-tuning. Methods such as LoRA train only a small set of extra parameters, so gradients and optimizer states shrink enormously. Combined with 4-bit quantization, this is how many people fine-tune large language models on a single consumer card.
Freeze layers. Layers that do not receive gradients need no gradient or optimizer memory.

Fix 7: Reduce Input Size

Long text sequences and high resolution images are expensive. Attention in transformer models grows quickly with sequence length, so cutting the maximum length from 2048 to 1024 tokens can make a big difference. For vision models, lowering the image resolution or using smaller crops has the same effect. Always test whether the smaller input still gives acceptable results for your project.

Fix 8: Fix Memory Leaks in Your Code

Sometimes the error appears after many successful iterations, which usually points to a leak. The most common causes are easy to avoid:

Accumulating tensors that still carry gradient history. When you add up losses for logging, use loss.item() instead of the tensor itself.
Running evaluation without disabling gradients. Wrap validation and inference code in torch.no_grad() or torch.inference_mode().
Keeping references to old outputs in lists or global variables.

model.eval()
with torch.no_grad():
    for inputs, labels in val_loader:
        outputs = model(inputs.cuda())
        total_loss += criterion(outputs, labels.cuda()).item()

Fix 9: Reduce Memory Fragmentation

Even when enough total memory is free, PyTorch can fail if that memory is split into small pieces and no single block is large enough. If the error message shows that reserved memory is much larger than allocated memory, try changing the allocator behavior with an environment variable before starting your script:

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

On older PyTorch versions, the setting max_split_size_mb:128 is commonly used for the same purpose. Check which options your installed version supports.

Fix 10: Offload or Distribute the Work

When one card is not enough, you can spread the load. Offloading moves optimizer states or parameters to system RAM, which is slower but allows much larger models. Data parallel and sharded training methods split memory across several GPUs, so each card only holds a portion of the weights, gradients, and optimizer states. Libraries built for large model training offer ready-made configurations, so you rarely need to write this logic yourself.

Fixing CUDA Out of Memory When Running LLMs

If you are running a language model for inference rather than training it, the biggest memory users are the model weights and the KV cache, which grows with the length of the conversation. These steps help most:

Load a quantized version of the model, such as 4-bit or 8-bit.
Lower the maximum context length.
Reduce the number of requests processed at the same time.
Use half precision rather than full 32-bit precision.
Offload some layers to the CPU if your tool supports it.

Quick Reference Table

Fix	Memory saved	Effort	Trade-off
Close other GPU apps	Varies	Very easy	None
Lower batch size	High	Very easy	Slower epochs
Mixed precision	Medium to high	Easy	Rare numeric issues
Gradient accumulation	High	Easy	Slightly slower
Gradient checkpointing	High	Easy	20 to 30% slower
8-bit optimizer	High	Medium	Small accuracy risk
Quantization	Very high	Medium	Some quality loss
Offloading	Very high	Medium to hard	Much slower

How to Prevent CUDA Out of Memory in the Future

Estimate memory needs before you start. Count parameters, multiply by the bytes needed for your precision and optimizer, and leave room for activations.
Begin with a small batch size and increase it gradually while watching memory use.
Monitor the GPU during training with nvidia-smi or a monitoring tool.
Restart your notebook kernel regularly, because leftover objects pile up over long sessions.
Leave a safety margin of a few hundred megabytes, since drivers and the display can use some memory too.

When Is It Time to Upgrade Your GPU?

Software fixes can take you a long way, but there is a point where more VRAM is the sensible answer. If you regularly work with larger models, long context windows, or high resolution data, the amount of VRAM matters more than almost any other specification of a graphics card. As a rough guide:

6 to 8 GB: fine for learning, small models, and quantized inference of small language models.
12 to 16 GB: a comfortable range for fine-tuning mid-size models with efficient methods and running larger quantized models.
24 GB and above: suited to serious fine-tuning, larger language models, and demanding image or video work.

When comparing cards, look at VRAM capacity and memory bandwidth first, then compute performance. A slightly slower card with more memory will often let you run workloads that a faster card with less memory cannot run at all. Renting cloud GPUs by the hour is another option if you only need extra memory occasionally.

Frequently Asked Questions

Why do I get CUDA out of memory even though nvidia-smi shows free memory?

The memory may be fragmented, or the single allocation your program requested may be larger than any free block. Another process may also have grabbed memory between the moment you checked and the moment the error occurred. Lowering the batch size or adjusting the allocator setting usually helps.

Does torch.cuda.empty_cache() fix the error?

Only partly. It releases cached blocks that PyTorch is not currently using, but it does not free tensors that are still referenced. It is useful between experiments, not as a cure for a model that is too large.

Can I fix CUDA out of memory without reducing the batch size?

Yes. Mixed precision, gradient checkpointing, 8-bit optimizers, quantization, and shorter input lengths all lower memory use while keeping your batch size, or your effective batch size through gradient accumulation.

Is CUDA out of memory the same as running out of system RAM?

No. This error refers only to the dedicated memory on your graphics card. System RAM is separate, although offloading techniques can use it as extra space at the cost of speed.

Why does the error appear after several epochs?

This points to a memory leak or growing allocations, such as storing loss tensors with their gradient history or very uneven batch sizes. Use .item() when logging, and check that your evaluation code runs without gradients.

Final Thoughts

A CUDA out of memory error looks frightening, but it is really just a signal that your workload needs more memory than your card can provide at that moment. Start by checking what is using your GPU, then work down the list: lower the batch size, enable mixed precision, add gradient accumulation and checkpointing, and shrink the model or optimizer if needed. For most projects, one or two of these changes are enough to get training or inference running smoothly again.

If you keep running into the limit even after optimizing, that is a clear sign your projects have outgrown your current card, and choosing a GPU with more VRAM will give you the biggest improvement you can make.