Setting up a GPU stack correctly is the first hurdle for any AI or machine learning project. Whether you are fine-tuning models on LLM Hosting infrastructure or configuring your own workstation, a broken driver or mismatched CUDA version will stop your workloads before they start. This guide walks through the full process — from a clean driver install to a verified, production-ready CUDA environment.

Below you will learn how to install CUDA on Ubuntu step by step: removing conflicting drivers, installing the recommended NVIDIA driver, setting up the CUDA Toolkit and cuDNN, verifying everything works, and fixing the most common errors along the way. The instructions apply to Ubuntu 22.04 and 24.04 LTS.

Before You Start: Check GPU, Kernel and Ubuntu Version

Before touching drivers, confirm what hardware and OS you are working with. Mismatched kernel headers and unsupported GPUs cause most failed installations.

Related ReadHow to Set Up Multi-GPU Training with PyTorch

→

1. Identify your GPU:

lspci | grep -i nvidia

You should see your card listed (e.g., NVIDIA Corporation GA102 [GeForce RTX 3090] or a data-center card like an A100/H100).

2. Check your Ubuntu version and kernel:

lsb_release -a
uname -r

Note the kernel version — you will need matching headers. If you plan to install GPU driver Ubuntu 24.04 style (i.e., on the latest LTS), make sure your system is fully updated first, since 24.04 ships with a newer kernel (6.8+) that requires driver 545 or later.

3. Install build tools and kernel headers:

sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential linux-headers-$(uname -r) dkms

4. Check Secure Boot status:

mokutil --sb-state

If Secure Boot is enabled, unsigned kernel modules will be blocked. Either disable it in BIOS/UEFI or be prepared to enroll a MOK key during driver installation (covered in Troubleshooting).

Step 1: Remove Conflicting Drivers and Blacklist Nouveau

Ubuntu ships with the open-source Nouveau driver, which conflicts with NVIDIA’s proprietary driver. Old or partial NVIDIA installations (especially runfile installs) also cause version mismatches.

Remove any existing NVIDIA packages:

sudo apt purge -y 'nvidia-*' 'libnvidia-*'
sudo apt autoremove -y

Blacklist Nouveau:

sudo tee /etc/modprobe.d/blacklist-nouveau.conf <<EOF
blacklist nouveau
options nouveau modeset=0
EOF
sudo update-initramfs -u
sudo reboot

After the reboot, confirm Nouveau is not loaded:

lsmod | grep nouveau

Empty output means you are ready to proceed.

Step 2: Install the Recommended GPU Driver

The safest way to install NVIDIA drivers Ubuntu offers is the official ubuntu-drivers tool, which selects a tested, signed driver for your exact GPU.

Check what Ubuntu recommends:

ubuntu-drivers devices

You will see output like driver : nvidia-driver-550 - distro non-free recommended.

Install the recommended driver automatically:

sudo ubuntu-drivers autoinstall

Or install a specific version (recommended for servers, where you want deterministic builds):

sudo apt install -y nvidia-driver-550-server
sudo reboot

Use the -server variants on headless machines — they exclude desktop components and receive longer support. For data-center GPUs (A100, H100, L40S), always match the driver branch to the CUDA version your framework requires (check NVIDIA’s CUDA compatibility matrix).

After rebooting, run nvidia-smi. If you see your GPU listed with a driver version, the driver layer is working.

Step 3: Install the CUDA Toolkit (apt repo vs runfile)

There are two ways to handle Ubuntu CUDA Toolkit installation: NVIDIA’s apt repository or the standalone runfile. For most users — and virtually all production servers — the apt repository is the better choice.

	apt repository	runfile (.run)
Updates	Managed via apt upgrade	Manual
Driver handling	Uses existing driver	Can overwrite your driver (risk)
Multiple CUDA versions	Yes, side-by-side	Yes, but manual management
Best for	Servers, production	Air-gapped systems, exotic setups

Install via the apt repository (recommended):

# Ubuntu 24.04 (use ubuntu2204 in the URL for 22.04)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install -y cuda-toolkit-12-6

Note: install cuda-toolkit-12-6 rather than the cuda meta-package. The cuda package pulls in its own driver and can replace the one you just installed. Installing only the toolkit keeps driver and toolkit decoupled — this is the core of how to install CUDA on Ubuntu without breaking your driver.

Runfile method (only if you have a specific reason):

wget https://developer.download.nvidia.com/compute/cuda/12.6.0/local_installers/cuda_12.6.0_560.28.03_linux.run
sudo sh cuda_12.6.0_560.28.03_linux.run --toolkit --silent

The --toolkit flag skips the bundled driver, avoiding conflicts with the apt-installed one.

Step 4: Install cuDNN

Deep learning frameworks (PyTorch, TensorFlow, JAX) rely on cuDNN for optimized neural network primitives. The easiest way to install cuDNN Ubuntu supports is the same NVIDIA apt repository you added in Step 3:

sudo apt update
sudo apt install -y cudnn9-cuda-12

This installs cuDNN 9 built against CUDA 12.x. Verify the installed version:

dpkg -l | grep cudnn

Important compatibility note: if you install PyTorch via pip with bundled CUDA wheels (pip install torch), it ships its own CUDA and cuDNN libraries — the system-level cuDNN matters mainly for TensorFlow, custom C++/CUDA builds, and inference servers like TensorRT or Triton.

Step 5: Verify with nvidia-smi and a CUDA Sample

Now verify CUDA installation end to end — driver, toolkit, and runtime.

1. Driver check:

nvidia-smi

You should see the GPU model, driver version, and a “CUDA Version” field. Note: the CUDA version shown by nvidia-smi is the maximum version the driver supports, not necessarily what is installed.

2. Toolkit check — the reliable way to check CUDA version Linux-side:

nvcc --version

This reports the actual installed toolkit version (e.g., release 12.6). If nvcc is not found, your PATH is not set yet — see Step 6.

3. Compile and run a CUDA sample:

git clone https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples/Samples/1_Utilities/deviceQuery
make
./deviceQuery

A successful run ends with Result = PASS and prints your GPU’s compute capability, memory, and SM count. This confirms the entire stack — kernel module, driver, toolkit, and runtime — is functional.

4. Optional framework check:

python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

Step 6: Set PATH, LD_LIBRARY_PATH and Pin Driver Versions

CUDA installs to /usr/local/cuda-12.6 with a /usr/local/cuda symlink. Add it to your environment:

echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

For system-wide configuration on multi-user servers, use /etc/profile.d/cuda.sh instead of per-user .bashrc files.

Pin driver and CUDA versions in production. Unattended upgrades can pull a new driver that breaks the driver/library match and takes down every GPU workload on the host:

sudo apt-mark hold nvidia-driver-550-server cuda-toolkit-12-6 cudnn9-cuda-12

Best practice for production GPU nodes: pin versions, test upgrades on a staging server first, monitor GPU health with nvidia-smi exporters (e.g., DCGM + Prometheus), and document the exact driver/CUDA/cuDNN combination in your infrastructure runbook.

Troubleshooting: Driver/Library Mismatch, No Devices Found, Secure Boot

“Failed to initialize NVML: Driver/library mismatch” The kernel module and userspace libraries are different versions — usually after an unattended driver upgrade without a reboot. Fix:

sudo reboot

If it persists, purge and reinstall the driver (Step 1 → Step 2), then hold the packages as shown in Step 6.

nvidia-smi not working / “command not found” / “No devices were found” When nvidia-smi is not working, check in this order:

1. Module loaded? lsmod | grep nvidia — if empty, run sudo modprobe nvidia and check dmesg | grep -i nvidia for errors.
2. DKMS build failed? dkms status — a missing build usually means missing kernel headers: sudo apt install linux-headers-$(uname -r) then sudo dkms autoinstall.
3. GPU visible on the PCI bus? lspci | grep -i nvidia — if nothing appears on a VM or dedicated server, the GPU passthrough is misconfigured at the host level; contact your provider.

Secure Boot blocking the module If dmesg shows module verification failed, Secure Boot is rejecting the unsigned module. Options:

• Disable Secure Boot in BIOS/UEFI (simplest for dedicated servers), or
• Enroll a MOK key: during apt driver installation Ubuntu prompts you to set a password; on the next boot, select “Enroll MOK” in the blue MOK Manager screen and enter that password.

CUDA version confusion If nvidia-smi shows CUDA 12.6 but nvcc --version shows 12.2 — that is normal. The driver reports maximum supported CUDA; nvcc reports the installed toolkit. Frameworks care about the toolkit/runtime version.

Skipping the Setup: Pre-Installed CUDA Stacks on a Dedicated GPU Server

Now you know how to install CUDA on Ubuntu — but everything above takes 30–60 minutes on a clean system, and considerably longer when Secure Boot, kernel upgrades, or driver mismatches get involved. On production infrastructure, you also carry the ongoing burden: validating driver updates, rebuilding DKMS modules after kernel patches, and keeping CUDA/cuDNN aligned with your framework versions across every node.

For teams running AI workloads, a dedicated GPU server with a pre-installed, tested driver + CUDA + cuDNN stack removes that entire maintenance layer. You get root access to bare-metal GPU hardware with the environment already verified — nvidia-smi, deviceQuery, and framework checks all passing on delivery — plus a provider responsible for keeping the host firmware, kernel, and driver combination stable.

Get a Dedicated GPU Server with CUDA Ready to Go

HostingB2B provisions dedicated GPU servers with your choice of driver branch, CUDA toolkit, and cuDNN version pre-installed and verified — so your team starts training and serving models on day one instead of debugging kernel modules.

• Machine Learning Hosting — bare-metal and high-performance compute environments optimized for AI model training and inference, featuring enterprise-grade hardware, full root access, and a custom ML framework and CUDA stack configured to your specification.
• AI Hosting — infrastructure optimized for training, fine-tuning, and inference workloads, with scalable configurations and 24/7 support.

FAQ