Fine-tuning turns a general-purpose model like Llama 3 into a specialist that understands your domain, your tone, and your data. The problem: full fine-tuning of even a 7B model can demand 100+ GB of VRAM. LoRA (Low-Rank Adaptation) changes the economics entirely — you train small adapter matrices instead of all model weights, cutting memory requirements by up to 90%.

This lora fine tuning tutorial walks through the complete process: hardware, environment setup, dataset preparation, hyperparameters, training, and deployment. If you have been searching for how to fine tune llm workflows that actually run on affordable hardware, this guide covers everything from a single-GPU setup to multi-GPU H100 clusters.

What You Need Before Fine-Tuning (dataset, VRAM, drivers)

Before you fine tune llm step by step, make sure three prerequisites are in place:

• A training dataset. Quality beats quantity. 1,000–10,000 high-quality instruction/response pairs typically outperform 100,000 noisy examples. Formats: JSONL with instruction/output fields, or chat-formatted conversations.
• Sufficient VRAM. The single biggest constraint. LoRA reduces requirements dramatically, but you still need to fit the base model weights, optimizer states, and activations in GPU memory (see the table below).
• A working CUDA stack. NVIDIA driver ≥ 535, CUDA Toolkit 12.x, and matching PyTorch builds. Version mismatches between driver, CUDA, and PyTorch are the #1 cause of failed training runs.

You will also want fast NVMe storage (datasets and checkpoints grow quickly) and at least 64 GB of system RAM for data loading and tokenization.

LoRA vs QLoRA vs Full Fine-Tuning: Which to Use

Method	What it trains	Memory footprint	Quality	Best for
Full fine-tuning	All model weights	Very high (16 bytes/param with Adam)	Highest ceiling	Large budgets, major domain shifts
LoRA	Low-rank adapter matrices (~0.1–1% of params)	Base model in FP16/BF16 + tiny adapters	Near full FT quality	Most production use cases
QLoRA	LoRA adapters on a 4-bit quantized base	Lowest — 4-bit base + adapters	Slightly below LoRA	Limited VRAM, large models

Choosing the right method is the first real decision in how to fine tune llm projects, because it determines your hardware budget for everything that follows.

Practical recommendation:

• Start with LoRA if your GPU fits the model in 16-bit.
• Use QLoRA when VRAM is tight — this qlora fine tuning guide approach lets you train a 70B model on a single 48–80 GB GPU.
• Reserve full fine-tuning for cases where LoRA plateaus: heavy domain adaptation (legal, medical, low-resource languages) or when you need to modify deep model behavior.

VRAM Requirements by Model Size

Understanding gpu requirements for fine tuning llm workloads prevents expensive trial and error. Approximate figures for a batch size of 1–4 with gradient checkpointing enabled:

Model size	Full fine-tuning	LoRA (16-bit base)	QLoRA (4-bit base)
7B–8B	~120 GB (multi-GPU)	~18–24 GB	~8–10 GB
13B	~240 GB (multi-GPU)	~32–40 GB	~12–16 GB
70B	~1.2 TB (8× H100)	~160 GB (2× 80 GB)	~46–48 GB

Key takeaways:

• A single 24 GB card (RTX 4090 / A5000) handles LoRA on 7B–8B models comfortably.
• QLoRA on a 48 GB or larger GPU covers models up to 70B.
• Full fine-tuning at any serious scale requires dedicated multi-GPU infrastructure.

Step 1: Provision a Dedicated GPU Server

Cloud GPU spot instances get interrupted mid-training; shared GPU platforms throttle unpredictably. For multi-hour or multi-day training runs, a dedicated server is the reliable option.

A strong entry point is the NVIDIA DGX Spark — a Grace Blackwell-based dedicated AI machine with 128 GB of unified memory (per NVIDIA’s specifications), available from EUR 326/month. The unified architecture means the GPU can address the full memory pool rather than a fixed 24–48 GB of discrete VRAM, which removes the out-of-memory ceiling that stops LoRA and QLoRA runs on consumer cards.

Two honest caveats to set expectations:

• Capacity vs. throughput. 128 GB of capacity fits QLoRA on 70B-class models and LoRA on mid-size (13B–30B) models, but unified LPDDR5X memory has lower bandwidth than HBM on data-center GPUs — so a 70B run that fits will train slower than on an H100. For 7B–13B fine-tuning, iteration, and inference, the DGX Spark is well matched; for time-critical 70B training, benchmark first or go straight to H100-class hardware.
• Validate against your workload. Actual memory headroom depends on sequence length, batch size, and gradient checkpointing settings — run a short trial with your real dataset before committing to a long training run.

Why dedicated hardware matters for training:

• No interruptions — training runs of 12–72 hours complete without preemption.
• Consistent throughput — no noisy neighbors competing for GPU cycles.
• Data control — your training dataset never leaves hardware you control (critical for regulated industries; more on this below).

You can deploy a DGX Spark dedicated server at /nvidia-dgx-spark/ with root access and your choice of OS.

Step 2: Install CUDA, PyTorch, Transformers and PEFT

With the server provisioned, set up the software stack. This peft lora setup takes about 15 minutes on Ubuntu 22.04/24.04.

Version note: the Hugging Face ecosystem moves fast, and the SFTTrainer API in particular has changed between TRL releases. The code in this guide targets the pinned versions below — pin yours the same way, because an unpinned pip install months later may pull incompatible releases. Check pytorch.org/get-started for the currently recommended PyTorch/CUDA pairing before installing.

bash

# 1. Verify the NVIDIA driver
nvidia-smi   # driver >= 550 recommended; the CUDA version it reports must be >= your toolkit build

# 2. Create an isolated environment
python3 -m venv ~/ft-env && source ~/ft-env/bin/activate

# 3. Install PyTorch (CUDA 12.4 build shown; verify current pairing on pytorch.org)
pip install torch --index-url https://download.pytorch.org/whl/cu124

# 4. Install the fine-tuning stack with pinned versions
pip install "transformers==4.48.*" "datasets==3.2.*" "peft==0.14.*" \
            "accelerate==1.3.*" "bitsandbytes==0.45.*" "trl==0.14.*"

Component roles:

• transformers — model loading and tokenization
• peft — LoRA/QLoRA adapter implementation from Hugging Face
• bitsandbytes — 4-bit quantization for QLoRA
• trl — the SFTTrainer class that simplifies supervised fine-tuning
• accelerate — multi-GPU and mixed-precision orchestration

Verify GPU visibility and BF16 support before proceeding — the training config below depends on it:

import torch
print(torch.cuda.is_available(), torch.cuda.get_device_name(0))
print("BF16 supported:", torch.cuda.is_bf16_supported())

BF16 requires Ampere-generation GPUs or newer (A100, RTX 30/40-series, H100, Grace Blackwell). On older cards (T4, V100, RTX 20-series), set fp16=True instead of bf16=True in the training arguments — FP16 works but is more prone to loss instability, so watch for NaN losses and lower the learning rate if they appear.

Step 3: Prepare and Format Your Training Dataset

To fine tune llm on your own data, convert it into a consistent instruction format. For Llama 3.1 (and other Llama-3-family models), use the model’s chat template — TRL applies it automatically when the dataset uses the messages schema:

{"messages": [
  {"role": "system", "content": "You are a support assistant for a hosting company."},
  {"role": "user", "content": "How do I point my domain to a new server?"},
  {"role": "assistant", "content": "Update the A record in your DNS zone to the new server IP..."}
]}

Dataset preparation checklist:

• Deduplicate — repeated examples cause overfitting on those samples.
• Clean PII — strip customer emails, IPs, and credentials before training.
• Split — hold out 5–10% as an evaluation set to detect overfitting.
• Check token lengths — truncate or filter examples exceeding your max_seq_length (2048–4096 is typical).

Load and split with the datasets library:

from datasets import load_dataset
dataset = load_dataset("json", data_files="train.jsonl", split="train")
dataset = dataset.train_test_split(test_size=0.05)

Step 4: Configure LoRA Hyperparameters (rank, alpha, dropout, target modules)

This is the core of learning how to fine tune llama 3 with LoRA. The four parameters that matter most:

from peft import LoraConfig

lora_config = LoraConfig(
    r=16,                    # rank
    lora_alpha=32,           # scaling factor
    lora_dropout=0.05,       # regularization
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    task_type="CAUSAL_LM",
)

• Rank (r): the dimensionality of the adapter matrices. r=8–16 suits most tasks; increase to 32–64 for complex domain adaptation. Higher rank = more trainable parameters = more VRAM and higher overfitting risk.
• Alpha: scales adapter influence. The common convention is alpha = 2 × r.
• Dropout: 0.05–0.1 for small datasets to reduce overfitting; 0 for very large datasets.
• Target modules: the list above covers all attention and MLP projections in Llama-family architectures. The QLoRA paper (Dettmers et al., 2023) reported that adapting all linear layers, not just attention, was necessary to match full fine-tuning quality in their experiments — so this is a reasonable default, though the attention-only configuration trains faster and is worth benchmarking on your own task. Module names differ between architectures; use model.named_modules() to inspect them for non-Llama models.

For QLoRA, add 4-bit loading when initializing the model:

from transformers import BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

Step 5: Run Training and Monitor Loss

Launch training with TRL’s SFTTrainer. We use Llama 3.1 8B Instruct as the base model — a widely supported choice with a 128k context window (it is a gated repository, so accept the license on Hugging Face and run huggingface-cli login first). The same code works for any Llama-family checkpoint; if you standardize on a different model, only the model ID and target_modules inspection change.

The example below targets TRL 0.14 as pinned in Step 2. SFTTrainer arguments have been renamed across TRL releases (e.g., tokenizer → processing_class, evaluation_strategy → eval_strategy), so if you upgrade TRL, re-validate against its release notes rather than assuming the snippet still runs.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig

model_id = "meta-llama/Llama-3.1-8B-Instruct"

use_bf16 = torch.cuda.is_bf16_supported()

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if use_bf16 else torch.float16,
    device_map="auto",
    # quantization_config=bnb_config,  # uncomment for QLoRA (Step 4)
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

trainer = SFTTrainer(
    model=model,
    processing_class=tokenizer,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    peft_config=lora_config,
    args=SFTConfig(
        output_dir="./llama31-lora",
        num_train_epochs=3,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        bf16=use_bf16,
        fp16=not use_bf16,   # fallback for pre-Ampere GPUs
        logging_steps=10,
        eval_strategy="steps",
        eval_steps=100,
        save_strategy="epoch",
        gradient_checkpointing=True,
    ),
)
trainer.train()

What to monitor:

• Training loss should decline steadily. A flat curve suggests the learning rate is too low or the data is inconsistent.
• Eval loss rising while training loss falls = overfitting. Stop early, lower epochs, or increase dropout.
• GPU utilization — watch nvidia-smi or nvtop. If utilization sits below ~80%, increase batch size or check for data-loading bottlenecks.
• Loss spikes usually indicate corrupted samples or a learning rate that is too aggressive; try 1e-4.

Typical LoRA training times on dedicated hardware: an 8B model on 10k examples completes 3 epochs in roughly 2–4 hours. Once you understand how to fine tune llm training loops at this scale, scaling up to larger models is mostly a matter of VRAM and time.

Step 6: Merge Adapters and Serve the Fine-Tuned Model

After training you have two options:

Option A — serve with adapters attached (flexible, allows hot-swapping multiple adapters on one base model; vLLM supports this natively via --enable-lora):

from peft import PeftModel
model = PeftModel.from_pretrained(base_model, "./llama31-lora")

Option B — merge for production (simpler deployment, zero inference overhead).

Important caveat for QLoRA: do not merge adapters into the 4-bit quantized model. Reload the base model in 16-bit first, attach the adapters, then merge — merging into a quantized base degrades quality.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.bfloat16,   # 16-bit, even if you trained with QLoRA
)
model = PeftModel.from_pretrained(base, "./llama31-lora")
merged = model.merge_and_unload()

merged.save_pretrained("./llama31-finetuned", safe_serialization=True)  # safetensors
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
tokenizer.save_pretrained("./llama31-finetuned")

Serve the merged model with vLLM for production-grade throughput. Pin the vLLM version and smoke-test the merged directory before wiring it into production — vLLM validates the model config on startup, and a mismatched config.json or missing tokenizer files will fail here rather than at request time:

pip install "vllm==0.7.*"   # pin; verify compatibility with your transformers version
vllm serve ./llama31-finetuned --max-model-len 4096

# Smoke test from another shell:
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "./llama31-finetuned", "messages": [{"role": "user", "content": "ping"}]}'

This exposes an OpenAI-compatible API endpoint on your server, so existing client code works with a one-line base-URL change. For production, put the endpoint behind a reverse proxy with TLS, add API-key authentication, and monitor latency and GPU memory with Prometheus + Grafana.

When to Move Up to Full Fine-Tuning on an H100 Dedicated Server

LoRA covers 80–90% of use cases, but consider full fine-tuning when:

• LoRA quality plateaus despite rank increases and data improvements.
• Deep domain shift — training on low-resource languages, specialized legal/medical corpora, or proprietary code bases where surface-level adaptation isn’t enough.
• Continued pre-training — feeding the model large volumes of raw domain text, not just instruction pairs.
• Latency-critical serving — a fully fine-tuned smaller model can replace a larger base model + adapter, cutting inference costs.

Full fine-tuning of a 7B–8B model requires multiple 80 GB GPUs with NVLink; 70B models need 8× H100 nodes with DeepSpeed ZeRO-3 or FSDP. Dedicated H100 servers with root access, NVLink interconnects, and no virtualization overhead are available at /ai-hosting/llm-hosting/ — sized for both multi-day training runs and high-throughput inference.

Keeping Training Data in the UK/EU (iGaming, fintech, forex compliance)

For regulated businesses, where you fine-tune matters as much as how:

• GDPR data residency — training datasets often contain customer interactions, transaction records, or player behavior data. Sending that data to US-based GPU clouds creates cross-border transfer obligations under GDPR Chapter V.
• iGaming licensing — regulators such as the MGA and UKGC expect documented control over where player data is processed. A dedicated server in a UK/EU data center gives you a clear, auditable answer.
• Fintech and forex — FCA and CySEC-regulated firms benefit from demonstrable data locality and single-tenant infrastructure when fine-tuning models on trade or KYC data.
• Single-tenant isolation — on a dedicated server, no other customer’s workload shares your GPU, memory, or storage, simplifying your security and audit posture compared to multi-tenant AI APIs.

General guidance, not legal advice — but as a rule: fine-tuning on dedicated UK/EU infrastructure eliminates the third-party data processor questions that arise when uploading regulated datasets to overseas AI platforms. Standard best practices still apply: encrypt datasets at rest, restrict SSH access with key-based auth and IP allowlists, keep off-site backups of checkpoints, and log all access to training data.

Start Fine-Tuning on a Dedicated DGX Spark Server

Knowing how to fine tune llm models is only half the equation — the other half is hardware that removes VRAM guesswork and keeps multi-hour runs uninterrupted. The NVIDIA DGX Spark with 128 GB unified memory (from EUR 326/mo) is a strong fit for LoRA and QLoRA fine-tuning of 7B–13B models, with the memory capacity to experiment on larger ones — on a single dedicated machine with root access, UK/EU data residency, and no preemption. When training speed on 70B-class models becomes the bottleneck, H100 dedicated servers are the next step.

• Deploy a AI Hosting →
• Explore LLM hosting and H100 options →

FAQ