How to Set Up Multi-GPU Training with PyTorch

Training modern deep learning models on a single GPU quickly hits a wall: batch sizes shrink, epochs stretch into days, and experimentation slows to a crawl. Setting up PyTorch multi GPU training correctly can cut training time almost linearly with the number of GPUs – but only if you use the right parallelisation strategy, the right launcher, and AI Hosting with proper GPU interconnects.

This guide walks through the complete distributed training setup on a dedicated multi-GPU server: from verifying the driver stack and NCCL, to converting a single-GPU script to DistributedDataParallel (DDP), launching with torchrun, and tuning batch size and mixed precision for maximum throughput.

When Multi-GPU Training Is Worth It

Before learning how to use multiple GPUs in PyTorch, confirm that your workload actually benefits from it. PyTorch multi GPU training pays off when:

Related ReadHow to Fine-Tune an LLM with LoRA: Step-by-Step Guide

→

• Your model or batch no longer fits in VRAM. If you are training transformers, diffusion models, or large CNNs and constantly fighting CUDA out of memory errors, a second GPU either doubles effective batch size (data parallelism) or lets you shard the model itself.
• Epoch time is the bottleneck for iteration speed. If a single training run takes 20+ hours on one GPU, near-linear DDP scaling on 2 GPUs brings it down to ~10–11 hours – meaning two experiments per day instead of one.
• You train regularly, not once. For teams running continuous fine-tuning, retraining pipelines, or hyperparameter sweeps, the fixed monthly cost of a multi-GPU dedicated server is amortised across every run – with no per-hour cloud billing anxiety.

When it is not worth it: small models (<50M parameters) with small datasets, inference-only workloads, or pipelines bottlenecked by data loading rather than compute. Fix the input pipeline first – multi-GPU training amplifies a data-loading bottleneck, it does not remove it.

DataParallel vs DistributedDataParallel (and why DDP wins)

PyTorch ships two data-parallel APIs, and the DDP vs DataParallel choice is settled: use DistributedDataParallel for everything except quick prototypes.

	nn.DataParallel (DP)	DistributedDataParallel (DDP)
Process model	Single process, multi-thread	One process per GPU
Python GIL impact	Severe – threads contend for the GIL	None – separate interpreters
Gradient sync	Gather to GPU 0, then scatter	Ring all-reduce via NCCL, overlapped with backward pass
GPU 0 memory imbalance	Yes – GPU 0 holds outputs and gradients	No – symmetric load
Multi-node support	No	Yes
Typical 2-GPU scaling	1.4–1.6x	1.85–1.95x

DataParallel replicates the model across GPUs inside a single Python process. Every iteration, inputs are scattered, outputs gathered back to GPU 0, and gradients reduced on GPU 0 – creating both a GIL contention problem and a memory hotspot on the first GPU.

DDP instead spawns one process per GPU. Each process owns a full model replica, and gradients are synchronised with an all-reduce operation over NCCL that overlaps with backpropagation. The result is near-linear scaling and the same code path whether you run 2 GPUs on one server or 16 GPUs across nodes. Even the official PyTorch documentation recommends DDP over DP for single-machine multi-GPU training.

Step 1: Provision a Multi-GPU Dedicated Server (2x V100S 32 GB, EUR 983/mo)

A reliable distributed training setup starts with hardware where you control the full stack – drivers, CUDA version, NCCL, and the interconnect topology. Shared cloud instances often hide topology details and suffer from noisy neighbours; a dedicated server gives you deterministic performance.

A strong baseline configuration for 2-GPU DDP training:

• 2x NVIDIA V100S 32 GB 64 GB total VRAM, NVLink-capable, excellent FP16 Tensor Core throughput for mixed-precision training
• Modern multi-core CPU (16+ cores) – data loading and augmentation workers must keep both GPUs fed
• 128 GB+ RAM – dataset caching and prefetch buffers
• NVMe storage – sequential read speed directly affects DataLoader throughput
• Root access -install the exact CUDA / cuDNN / NCCL versions your framework requires

At EUR 983/month, a 2x V100S 32 GB dedicated GPU server typically undercuts equivalent on-demand cloud GPU instances running more than ~6–8 hours per day, while eliminating egress fees and spot-instance interruptions. For teams that prefer infrastructure tailored to ML workloads out of the box, purpose-built AI Hosting environments come with the GPU driver stack, monitoring, and network configuration prepared for distributed workloads.

Best practice: whatever provider you choose, insist on redundancy essentials – RAID or replicated storage for checkpoints, off-server backup of training artifacts, and out-of-band access (IPMI/iLO) so a hung NCCL job never requires a support ticket to power-cycle.

Step 2: Verify GPU Topology, NCCL and Driver Stack

Before touching Python, validate the foundation. This is the NCCL setup PyTorch relies on for all inter-GPU communication.

1. Check drivers and visibility:

bash

nvidia-smi

Both GPUs should appear with the expected VRAM and a driver version matching your target CUDA release.

2. Inspect the interconnect topology:

bash

nvidia-smi topo -m

Look at the link matrix between GPU0 and GPU1:

• NV1/NV2 -NVLink: ideal, 25–50 GB/s per link
• PIX/PXB – PCIe through a switch: fine for 2 GPUs
• SYS – traversing the CPU/QPI path: expect slower all-reduce; consider gradient accumulation to reduce sync frequency

3. Confirm PyTorch sees CUDA and NCCL:

python

import torch
print(torch.cuda.device_count())        # 2
print(torch.cuda.nccl.version())        # e.g. (2, 20, 5)
print(torch.cuda.get_device_name(0))    # Tesla V100S-PCIE-32GB

4. Run the NCCL sanity test (from NVIDIA’s nccl-tests repo):

bash

./build/all_reduce_perf -b 8 -e 256M -f 2 -g 2

Healthy NVLink pairs show bus bandwidth in the tens of GB/s. If the test hangs, check NCCL_DEBUG=INFO output – the most common culprits are mismatched CUDA/NCCL versions or IOMMU/ACS settings on the host.

Recommended environment variables for debugging:

bash

export NCCL_DEBUG=INFO          # verbose NCCL logging while validating
export NCCL_P2P_LEVEL=NVL       # prefer NVLink when available

Disable NCCL_DEBUG in production runs – the logging overhead is measurable.

Step 3: Convert Your Training Script to DDP

This PyTorch DistributedDataParallel tutorial section covers the four changes needed to convert a standard single-GPU training loop.

1. Initialise the process group. Each process learns its rank and device from environment variables that torchrun injects:

python

import os
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

def setup():
    dist.init_process_group(backend="nccl")
    local_rank = int(os.environ["LOCAL_RANK"])
    torch.cuda.set_device(local_rank)
    return local_rank

2. Wrap the model:

python

local_rank = setup()
model = MyModel().to(local_rank)
model = DDP(model, device_ids=[local_rank])

3. Shard the dataset with DistributedSampler so each GPU sees a unique slice of every epoch:

python

from torch.utils.data import DataLoader
from torch.utils.data.distributed import DistributedSampler

sampler = DistributedSampler(train_dataset, shuffle=True)
loader = DataLoader(
    train_dataset,
    batch_size=64,          # per-GPU batch size
    sampler=sampler,
    num_workers=8,
    pin_memory=True,
)

4. The training loop – call sampler.set_epoch() for correct shuffling, and restrict logging/checkpointing to rank 0:

python

for epoch in range(epochs):
    sampler.set_epoch(epoch)
    for x, y in loader:
        x, y = x.to(local_rank, non_blocking=True), y.to(local_rank, non_blocking=True)
        optimizer.zero_grad(set_to_none=True)
        loss = criterion(model(x), y)
        loss.backward()          # gradients all-reduced automatically
        optimizer.step()

    if dist.get_rank() == 0:
        torch.save(model.module.state_dict(), f"ckpt_epoch{epoch}.pt")

dist.destroy_process_group()

Key details that prevent subtle bugs:

• Save model.module.state_dict(), not model.state_dict(), to strip the DDP wrapper prefix.
• Checkpoint only on rank 0 to avoid file corruption from concurrent writes.
• Use set_to_none=True in zero_grad – it is faster and reduces memory fragmentation.

Step 4: Launch Jobs with torchrun

torchrun is the modern launcher that replaced torch.distributed.launch. It spawns one process per GPU and sets RANK, LOCAL_RANK, WORLD_SIZE, and rendezvous variables automatically.

A minimal torchrun multi GPU example for a single server with 2 GPUs:

bash

torchrun --standalone --nproc_per_node=2 train.py --epochs 50 --lr 3e-4

• --standalone – single-node mode, no external rendezvous endpoint needed
• --nproc_per_node=2 – one process per GPU

For multi-node scaling later, the same script works with rendezvous flags:

bash

torchrun --nnodes=2 --nproc_per_node=2 \
  --rdzv_backend=c10d --rdzv_endpoint=10.0.0.10:29500 \
  train.py

Operational best practices:

• Run training inside tmux or as a systemd service so SSH disconnects never kill a job.
• Add --max_restarts=3 to let torchrun’s elastic agent recover from transient failures.
• Log per-rank output to separate files (--log_dir) – debugging a hang is far easier when you can diff rank logs.

Step 5: Tune Batch Size, Mixed Precision and Gradient Accumulation

Multi GPU batch size semantics trip up nearly everyone in PyTorch multi GPU training: with DDP, the batch_size in your DataLoader is per GPU. Two GPUs at batch_size=64 give a global batch of 128.

Practical tuning rules:

• Scale the learning rate with the global batch. The linear scaling rule (Goyal et al.) – if you double the global batch, double the LR – combined with a short warmup (500–1,000 steps) preserves convergence in most cases.
• Enable mixed precision (AMP). V100S Tensor Cores roughly double FP16 throughput and halve activation memory:

python

scaler = torch.cuda.amp.GradScaler()

with torch.cuda.amp.autocast():
    loss = criterion(model(x), y)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()

• Use gradient accumulation when the target global batch exceeds VRAM. Wrap non-final micro-steps in model.no_sync() to skip redundant all-reduce operations:

python

for i, (x, y) in enumerate(loader):
    is_update_step = (i + 1) % accum_steps == 0
    ctx = model.no_sync() if not is_update_step else contextlib.nullcontext()
    with ctx, torch.cuda.amp.autocast():
        loss = criterion(model(x), y) / accum_steps
    scaler.scale(loss).backward()
    if is_update_step:
        scaler.step(optimizer)
        scaler.update()
        optimizer.zero_grad(set_to_none=True)

• Find the memory ceiling empirically: increase per-GPU batch size until OOM, then step back ~10% to leave headroom for fragmentation and occasional long sequences.

Step 6: Monitor Utilisation and Find Bottlenecks

A distributed training setup is only as good as its observability. Track three layers:

GPU layer:

bash

watch -n 1 nvidia-smi
# or richer:
nvidia-smi dmon -s pucvmet

Both GPUs should sit at 90%+ utilisation with balanced memory. A pattern of utilisation spikes followed by idle gaps means the GPUs are starving – usually a DataLoader problem (increase num_workers, enable pin_memory, move datasets to NVMe).

Framework layer: the PyTorch Profiler with TensorBoard reveals whether time goes to compute, communication, or data loading:

python

with torch.profiler.profile(
    activities=[torch.profiler.ProfilerActivity.CUDA],
    schedule=torch.profiler.schedule(wait=1, warmup=2, active=5),
    on_trace_ready=torch.profiler.tensorboard_trace_handler("./tb_logs"),
) as prof:
    for step, batch in enumerate(loader):
        train_step(batch)
        prof.step()

Host layer: export DCGM or nvidia-smi metrics into Prometheus/Grafana and alert on GPU temperature, ECC errors, and utilisation drops. Long training runs fail silently more often than loudly – a job that “runs” at 15% utilisation for a weekend wastes real money.

Teams that would rather focus on models than on Grafana dashboards can offload the monitoring, patching, and incident-response layer to Managed Services, keeping 24/7 eyes on the hardware while your team owns the training code.

Scaling Beyond Two GPUs: A100 80 GB and H100 Dedicated Servers

One of the biggest advantages of a correct PyTorch multi GPU training setup is portability: the DDP code above scales unchanged to larger configurations – only the launcher arguments change. When 2x V100S stops being enough:

• A100 80 GB – 2.5x the VRAM per GPU, third-generation Tensor Cores with BF16 and TF32, and Multi-Instance GPU (MIG) partitioning for mixed training/inference workloads. The jump to 80 GB per card often eliminates gradient accumulation entirely for mid-size LLM fine-tuning.
• H100 – transformer engine with FP8 support delivering 3–6x training throughput on transformer architectures versus A100, plus NVLink 4 at 900 GB/s aggregate bandwidth – critical once all-reduce volume grows with model size.
• Multi-node clusters – beyond 8 GPUs, node-to-node networking (100–400 Gbps, ideally RDMA-capable) becomes the scaling factor; this is where controlling the physical network on Dedicated Servers beats virtualised cloud networking.

The same principles apply to latency-sensitive production inference in regulated verticals: operators running real-time recommendation or risk models within iGaming Hosting environments benefit from the same dedicated-hardware determinism, with compliance-oriented infrastructure controls layered on top.

Rent a Dedicated Multi-GPU Server

Skip the procurement cycle and start PyTorch multi GPU training this week:

• LLM Hosting (2x V100S 32 GB GPU Server — EUR 983/mo) — NVLink-ready, NVMe storage, full root access, ideal for DDP workloads up to mid-size models.
• Machine Learning Servers — custom multi-GPU configurations (V100S, A100, H100), pre-validated driver and NCCL stacks, and optional managed monitoring.

Both options include the redundancy fundamentals serious training pipelines require: backup-ready storage, network redundancy, and 24/7 infrastructure support.

FAQ