Machine learning workloads are unlike anything else you will run in a data center. They saturate GPUs for days at a time, chew through terabytes of training data, and punish every bottleneck in your storage, power, and networking stack. Getting your AI infrastructure right from day one is the difference between a team that iterates on models daily and a team that spends its sprints fighting CUDA errors.

This guide walks you through how to set up a machine learning server end to end — from hardware selection and OS configuration to frameworks, inference engines, orchestration, security, and the buy-vs-rent decision. Each section ends with a practical checklist you can apply directly to your own machine learning server setup, whether you are deploying a single workstation-class node or a liquid-cooled multi-GPU production cluster.

---

Machine Learning Server Requirements (GPU, RAM, Storage, OS)

Before touching hardware, define your workload. Machine learning server requirements vary dramatically depending on whether you are fine-tuning a 7B-parameter LLM, training computer-vision models, or serving high-throughput inference for a SaaS product. Still, four components define every build.

Related ReadBuilding Smarter Hosting Solutions: The Power Trio

→

GPU: Generations and Alternatives

The GPU is the heart of any training server. Key parameters to evaluate:

• VRAM capacity — the single most important spec. Model weights, optimizer states, gradients, and activations must all fit in GPU memory. As a rule of thumb: 24 GB (RTX 4090/5090-class) handles mid-size models and LoRA fine-tuning; 48 GB (L40S, RTX 6000 Ada) covers larger fine-tunes; 80–192 GB per card is the standard for serious LLM work.
• Memory bandwidth — HBM-based data-center cards dramatically outperform GDDR consumer cards on large-batch training.
• Interconnect — for multi-GPU training, NVLink/NVSwitch or at minimum full PCIe 5.0 x16 per card. PCIe-bottlenecked GPUs waste money.
• Precision support — BF16 and FP8 are standard on Hopper; Blackwell adds native FP4, which roughly doubles inference throughput per watt for quantized models.

The NVIDIA landscape in 2026 spans three practical tiers:

Tier	Cards	Where they fit
Budget / rental sweet spot	A100 80 GB, L40S	Widely available on the bare-metal rental market at the lowest $/GPU-hour; still excellent for fine-tuning and mainstream training
Current mainstream	H100, H200 (141 GB HBM3e)	The default choice for new LLM training and high-throughput inference deployments; H200’s extra VRAM and bandwidth make it the best all-rounder
Flagship	B200, GB200 NVL72 (Blackwell)	Frontier-scale training and FP4 inference; delivers 2–3x Hopper performance but demands liquid cooling and extreme rack power density

Alternative accelerators are no longer exotic:

• AMD Instinct MI300X / MI325X — 192–256 GB HBM per card at a lower price per GB of memory than NVIDIA equivalents. ROCm has matured substantially: PyTorch and vLLM support it natively, making MI300X a genuinely competitive option for large-model inference and increasingly for training. The trade-off is a smaller ecosystem — validate your full stack on ROCm before committing.
• AWS Trainium / Inferentia — cost-efficient for training and inference respectively, but cloud-only and AWS-locked; relevant if you are already committed to that ecosystem (see the cloud vs on-premise section below).

GPU vs CPU for Machine Learning

A question every team asks at the start: GPU vs CPU for machine learning — do you actually need a GPU at all? The short answer:

Workload	Best fit
Deep learning training (CNNs, transformers, LLMs)	GPU — 10–100x faster than CPU
LLM/DL inference at scale	GPU (or specialized accelerators like Inferentia)
Classical ML (XGBoost, random forests, scikit-learn)	CPU is usually sufficient
Data preprocessing, ETL, feature engineering	CPU (many cores + RAM)
Small-batch, low-latency inference of small models	Modern CPUs can be cost-effective

GPUs excel at the massively parallel matrix operations that dominate neural networks. CPUs remain relevant for tabular ML, orchestration, and the data pipeline feeding the GPUs. In practice, a balanced machine learning server pairs both: enough CPU cores (typically 4–8 physical cores per GPU) to keep the accelerators fed, plus GPUs sized to the models you train.

RAM

System RAM should be at least 2x total GPU VRAM, ideally more. Data loaders, caching, and preprocessing pipelines are memory-hungry. Practical baselines:

• Entry training node: 64–128 GB
• Multi-GPU training server: 256 GB – 1 TB
• H200/B200-class nodes: 1–2 TB (matching the much larger aggregate VRAM)

Always use ECC RAM in production — a flipped bit in a week-long training run is an expensive way to learn this lesson.

Storage

ML training is an I/O-heavy discipline. A three-tier layout works well:

1. NVMe SSD (hot tier) — active datasets and checkpoints. PCIe 4.0/5.0 NVMe with 5–12 GB/s sequential reads; RAID 0/10 across multiple drives for large datasets.
2. SATA SSD or large NVMe (warm tier) — recent datasets, model registry.
3. HDD / object storage (cold tier) — archives, raw data lakes, backup targets.

Budget 2–4 TB NVMe minimum for a single-GPU node and 8–30 TB for multi-GPU training servers.

Operating System

Ubuntu LTS (22.04 / 24.04) is the de facto standard: NVIDIA and AMD publish drivers and toolkit packages for it first, and virtually every ML framework tests against it. Rocky Linux / RHEL are solid alternatives in enterprise environments. Windows Server is viable only for niche cases.

Checklist: Hardware, GPU Selection & Cooling

Use this list when specifying or ordering the machine:

• Defined workload class (training vs fine-tuning vs inference vs classical ML)
• GPU generation chosen deliberately: A100/L40S (budget rental), H100/H200 (mainstream), B200/GB200 (flagship), or MI300X (VRAM-heavy inference on ROCm)
• GPU VRAM sized to largest model + optimizer states (with 20–30% headroom)
• Multi-GPU nodes: NVLink/NVSwitch or full PCIe 5.0 x16 lanes per card confirmed
• CPU: 4–8 physical cores per GPU (AMD EPYC / Intel Xeon for server builds)
• RAM ≥ 2x total VRAM, ECC registered
• NVMe hot tier with ≥ 5 GB/s reads; capacity ≥ 2x active dataset size
• Redundant PSUs and out-of-band management (IPMI/iDRAC/iLO) for production nodes
• 10/25 GbE NICs minimum; 100–400 GbE or InfiniBand NDR/XDR for distributed training

Power & cooling — plan this before ordering GPUs, not after:

• Per-node power budget confirmed: a 4x A100 node draws 2.5–3 kW; an 8x H100/H200 (HGX) node draws 10–12 kW; a single B200 GPU alone has a TDP around 1,000 W
• Rack power density validated: typical air-cooled enterprise racks support 10–20 kW. Dense Hopper deployments push 30–40 kW per rack, and a GB200 NVL72 rack draws on the order of 120 kW — far beyond what conventional data-center infrastructure can deliver or cool
• Cooling strategy matched to density: air cooling remains fine for 1–4 GPU workstation/server builds and L40S-class cards. For dense H100/H200 and any Blackwell deployment, plan direct-to-chip liquid cooling (DLC) — including facility-side coolant distribution units (CDUs), leak detection, and rear-door heat exchangers where applicable
• Sustained-load thermal test passed (GPUs held at 100% for 24h+ without throttling)

This is a decisive on-premise consideration: if your facility cannot deliver the power density or liquid-cooling loop that modern accelerators require, retrofitting is often more expensive than the servers themselves. In that case, colocation or renting bare-metal GPU servers from a provider whose data center is already built for DLC and high-density racks is usually the pragmatic path (more on this in the buy-vs-rent section).

Checklist: OS, Drivers & CUDA Setup

This is where most machine learning server configuration problems are born. The dependency chain is strict: kernel → NVIDIA driver → CUDA toolkit → cuDNN → framework. A mismatch anywhere breaks everything above it. (On AMD hardware the equivalent chain is kernel → ROCm driver stack → ROCm libraries → framework, with the same discipline required.)

A proven sequence on Ubuntu:

1. Install Ubuntu Server LTS with a minimal package set; disable the Nouveau open-source driver.
2. Install the NVIDIA driver from the official CUDA repository (not the GUI “Additional Drivers” tool). Pin the driver branch and hold the package to prevent surprise upgrades.
3. Install the CUDA toolkit matching your framework’s supported version — check the PyTorch/TensorFlow compatibility matrix before installing, not after. Note that Blackwell GPUs require CUDA 12.8+ and recent driver branches.
4. Install cuDNN and NCCL (NCCL is mandatory for multi-GPU training; RCCL on AMD).
5. Verify: nvidia-smi shows all GPUs; nvcc --version matches expectations; a quick torch.cuda.is_available() sanity test passes.
6. Install the NVIDIA Container Toolkit so Docker/Podman containers get GPU access — containerized environments are strongly recommended over bare-metal Python installs.
7. Set GPU persistence mode (nvidia-smi -pm 1) and configure power/clock limits if thermals require it.

Checklist:

• Nouveau blacklisted, secure boot handled (MOK enrollment or disabled)
• NVIDIA driver (or ROCm stack) installed from official repo and version-pinned
• CUDA version matched to framework compatibility matrix (12.8+ for Blackwell)
• cuDNN + NCCL installed and verified
• NVIDIA Container Toolkit installed; docker run --gpus all tested
• Persistence mode enabled; monitoring of GPU temps/power/clocks in place
• Unattended upgrades excluded for driver/CUDA packages

Checklist: ML Frameworks, Inference Engines & Jupyter

With the foundation in place, the framework layer of your machine learning server setup should be isolated, reproducible, and multi-user friendly.

Environment strategy. Avoid installing frameworks into the system Python. Use one of:

• Docker containers (recommended for production) — start from NVIDIA NGC images (nvcr.io/nvidia/pytorch), which ship with tuned CUDA/cuDNN builds; AMD publishes equivalent ROCm images.
• Conda / Mamba — convenient for research teams; one environment per project.
• uv / venv + pip — lightweight, works well inside containers.

Training stack — install and verify:

• PyTorch with CUDA/ROCm build (torch.cuda.get_device_name() returns your GPU)
• TensorFlow (if required) — verify tf.config.list_physical_devices('GPU')
• JupyterLab / JupyterHub for multi-user notebook access — always behind authentication and TLS, never exposed raw on port 8888
• Experiment tracking: MLflow or Weights & Biases
• Common libraries: NumPy, pandas, scikit-learn, Hugging Face Transformers, FlashAttention where applicable

Inference stack — a first-class decision, not an afterthought. For LLM serving, the choice of inference engine and quantization format affects throughput more than stepping up a GPU tier — differences of 2–5x tokens/sec on identical hardware are common:

• Serving engine selected: vLLM (the de facto open-source standard; PagedAttention, continuous batching, broad model and hardware support including MI300X), TensorRT-LLM (maximum performance on NVIDIA, tighter integration effort), or TGI (Hugging Face ecosystem, simple deployment). NVIDIA Triton can front any of these as a unified serving layer
• Quantization strategy chosen: FP8 on Hopper/Blackwell for near-lossless 2x memory savings; INT4/AWQ/GPTQ to fit larger models into limited VRAM; FP4 on Blackwell for maximum inference throughput per watt
• Target latency (TTFT, tokens/sec) and concurrency benchmarked on the actual engine + quantization combo before sizing the fleet
• A “hello world” training run and a serving benchmark recorded as the node’s acceptance test

Document exact versions in a lockfile or Dockerfile. Reproducibility is a core requirement — you must be able to rebuild the identical environment six months later.

Checklist: Orchestration & Scaling Beyond One Server

A single well-configured node with Docker is manageable by hand. The moment your team grows, manual container juggling collapses — GPUs sit idle on one server while data scientists queue for another. Plan the orchestration layer against clear thresholds:

Tier 1 — single server, 1–4 users:

• Docker + NVIDIA Container Toolkit, one container per project
• GPU sharing via MIG partitioning (A100/H100/H200/B200 — hardware-isolated GPU slices) or time-slicing for lighter notebook workloads
• Optionally Slurm as a lightweight job queue if users compete for training slots

Tier 2 — 3+ nodes or 5+ data scientists:

• Kubernetes + NVIDIA GPU Operator as the foundation — the Operator automates driver, container-toolkit, and device-plugin lifecycle across the cluster, turning GPUs into schedulable resources
• Ray on top for distributed training and hyperparameter search, or Kubeflow for end-to-end ML pipelines (choose based on whether your bottleneck is distributed compute or workflow standardization)
• Fractional GPU allocation policy defined (MIG profiles per team, time-slicing for dev notebooks, whole GPUs reserved for training jobs)
• Namespace/quota model so one team cannot starve the cluster
• Autoscaling for inference workloads (KEDA or native HPA driven by queue depth / GPU utilization)

Tier 3 — multi-node distributed training:

• NCCL-tested interconnect: 100–400 GbE with RoCE, or InfiniBand NDR (400 Gbps) / XDR (800 Gbps) for Hopper- and Blackwell-class clusters
• Topology-aware scheduling so multi-GPU jobs land on NVLink-connected cards
• Shared high-throughput storage (parallel filesystem or high-performance NFS/object gateway) reachable from every node

The practical advice: even if you start at Tier 1, containerize everything from day one. A Dockerized workload moves to Kubernetes with minimal friction; a hand-installed Conda environment does not.

Checklist: Networking, Security & Data Compliance

ML servers concentrate two attractive targets: expensive compute and sensitive data. They are frequent victims of cryptomining intrusions precisely because exposed Jupyter instances are easy to find. Treat security as part of the initial machine learning server configuration, not an afterthought.

Networking:

• Management interface (IPMI) on an isolated VLAN, never internet-facing
• 10/25 GbE minimum to storage; RDMA (RoCE/InfiniBand) for multi-node training
• Private networking between GPU nodes and data storage

Access control:

• SSH key-only authentication; root login disabled; fail2ban or equivalent
• VPN or zero-trust access (WireGuard, Tailscale, or corporate ZTNA) — no ML service ports exposed publicly
• JupyterHub/MLflow/inference endpoints behind a reverse proxy with TLS and SSO/MFA
• Role-based access: separate users, no shared credentials, sudo audited

Data protection & compliance:

• Encryption at rest (LUKS/dm-crypt on data volumes) and TLS in transit
• Dataset access logging — who touched which data, when
• Backups: the 3-2-1 rule for datasets, checkpoints, and experiment metadata; restore procedures actually tested
• Regulatory mapping: GDPR for personal data, PCI DSS if payment data ever enters pipelines, sector rules (HIPAA, iGaming licensing requirements) as applicable — including data-residency constraints that may dictate where the server physically lives
• Anonymization/pseudonymization applied before data reaches training pipelines where feasible
• Incident response plan covering both data breach and compute-abuse scenarios
• Monitoring: GPU utilization anomalies (a sudden 100% load at 3 a.m. on an idle project is a red flag), Prometheus + Grafana + DCGM exporter

Note: compliance items above are general guidance, not legal advice — validate specific obligations with your compliance team or counsel.

Buy vs Rent a Machine Learning Server (Comparison)

Once the technical design is clear, the commercial question follows: buy vs rent machine learning server — which is right for you? The math depends almost entirely on utilization — and, increasingly, on whether your facility can physically host modern GPUs at all.

Factor	Buy (own hardware)	Rent (dedicated/bare-metal from a provider)
Upfront cost	High ($15k–$400k+ per node)	None — monthly OpEx
Cost at high utilization (24/7)	Lowest TCO over 3–5 years	Higher over long term, but predictable
Cost at low/spiky utilization	Poor — idle GPUs burn capital	Efficient — scale down when idle
Hardware refresh	Your problem; GPU generations now turn over every 12–18 months	Provider upgrades generations
Power & cooling	Your facility must support 10–40+ kW racks; DLC retrofit for Hopper/Blackwell density is a major capital project	Included — provider’s DC is built for high density and liquid cooling
Deployment speed	Weeks–months (procurement, delivery, facility work)	Days or hours
Customization	Total control	High (bare metal) to moderate
Failure risk	You carry replacement cost	SLA-backed replacement

Rules of thumb:

• If your GPUs will run at >60–70% utilization for 2+ years and your facility already supports the required power density and cooling, buying typically wins on raw TCO.
• If workloads are project-based, spiky, or experimental — or if hosting H200/B200-class hardware would require facility upgrades — renting dedicated GPU servers avoids stranded capital and infrastructure projects.
• The accelerating GPU release cadence strengthens the rental case: hardware you buy today competes against Blackwell-generation rentals within its depreciation window.
• A popular hybrid: rent first, buy later — validate real utilization on rented infrastructure for 6–12 months, then purchase only the baseline capacity you have proven you need.

For most B2B teams without an existing high-density data-center operation, renting bare-metal GPU servers hits the sweet spot: dedicated performance and full root control without capital expenditure, liquid-cooling infrastructure, or hardware-obsolescence risk.

Cloud vs On-Premise: Which Fits Your Workload (Comparison)

Related but distinct from buy-vs-rent is the cloud vs on-premise machine learning server decision — hyperscaler cloud GPU instances versus infrastructure you control (owned or rented bare metal in a data center).

Criterion	Cloud (AWS/GCP/Azure GPU instances)	On-premise / dedicated bare metal
Elasticity	Excellent — spin up 100 GPUs for a day	Fixed capacity; scaling takes planning
Cost per GPU-hour (sustained)	Highest; egress fees add up	2–5x cheaper at steady utilization
Performance consistency	Variable (virtualization, noisy neighbors)	Full, predictable bare-metal performance
Latest hardware access	Fast (H200/B200 instances, plus proprietary chips like Trainium/Inferentia and TPUs)	Depends on procurement or provider refresh cycles
Data gravity	Moving multi-TB datasets in/out is slow and costly	Data lives next to compute
Compliance & data residency	Depends on region availability and shared-responsibility model	Full control over location and isolation — often decisive in regulated sectors (fintech, iGaming, healthcare)
Ops burden	Minimal infra ops, but cloud-cost governance needed	Requires (or is delegated to a provider’s) ops expertise
Vendor lock-in	Meaningful — especially with proprietary accelerators (Trainium/Inferentia code paths are AWS-only) and managed services	Low — standard Linux stack, portable containers

Decision guide:

• Choose cloud for bursty experimentation, short-lived large-scale training runs, and teams that need dozens of GPUs today and zero next month. Proprietary accelerators (Trainium for training, Inferentia for inference) can undercut GPU pricing meaningfully — at the cost of AWS lock-in and a separate compilation toolchain (Neuron SDK).
• Choose on-premise or dedicated bare metal for sustained training pipelines, latency-sensitive or high-throughput inference, large stationary datasets, and workloads with strict data-residency or compliance constraints.
• Hybrid is increasingly the default: steady-state training and inference on dedicated servers, with cloud burst capacity for peaks. Design your tooling (containers, object storage, IaC, Kubernetes) to be portable from day one so the choice stays reversible.

Recommended Server Specs by Workload

Concrete machine learning server specs to anchor your planning:

1. Entry / Prototyping & Development — NVIDIA DGX Spark

For the entry and development tier, a purpose-built AI system like the NVIDIA DGX Spark has largely displaced self-assembled workstation builds. It packs the Grace Blackwell architecture into a compact node with the full NVIDIA AI software stack pre-configured — and is available as a hosted dedicated server, removing procurement entirely:

Component	Specification
Superchip	NVIDIA GB10 Grace Blackwell — Blackwell GPU with 5th-gen Tensor Cores
AI performance	Up to 1 PetaFLOP at FP4 precision
CPU	20-core Arm (10x Cortex-X925 + 10x Cortex-A725)
Memory	128 GB LPDDR5x unified system memory, 273 GB/s bandwidth
Storage	Up to 4 TB NVMe M.2 with self-encryption (SED)
Network	10 GbE RJ-45 + ConnectX-7 Smart NIC (200 GbE QSFP for dual-node clustering)
OS	NVIDIA Base OS (Ubuntu 24.04) with the AI stack preinstalled
Model capacity	Local prototyping, fine-tuning, and inference of models up to 200B parameters (up to 405B in a dual-Spark configuration via ConnectX-7)

• Fits: LLM prototyping and fine-tuning, inference development, data science, edge-AI testing before production deployment — with seamless model export to larger DGX-class or cloud infrastructure when you outgrow it
• Why it wins at this tier: unified memory means no VRAM-juggling between GPU and system RAM; the pre-optimized software stack eliminates the entire driver/CUDA setup chapter above; and as a hosted server it comes with power, cooling, and maintenance handled by the provider

2. Fine-Tuning & Mid-Size Training

• GPU: 2x L40S (48 GB each) or 2x A100 80 GB (excellent value on the rental market)
• CPU: 32 cores (EPYC 9004 / Xeon Scalable)
• RAM: 256 GB ECC
• Storage: 4 TB NVMe RAID + 20 TB warm tier
• Network: 25 GbE
• Cooling: air, with validated sustained-load airflow
• Fits: LoRA/QLoRA fine-tuning of 7–70B models, computer vision training, production inference for mid-size SaaS

3. Serious LLM Training / Multi-GPU

• GPU: 4–8x H200 141 GB (current mainstream default) or H100 80 GB with NVLink/NVSwitch; 8x MI300X 192 GB as the ROCm alternative for memory-bound workloads
• CPU: 64–128 cores dual-socket
• RAM: 1–2 TB ECC
• Storage: 15–30 TB NVMe (parallel or RAID 10), object storage backend
• Network: 400 GbE / InfiniBand NDR for multi-node scaling
• Cooling: direct-to-chip liquid cooling strongly recommended at 8-GPU density
• Fits: pre-training and full fine-tuning of large models, distributed training, high-throughput inference clusters

4. Frontier / Blackwell-Class

• GPU: 8x B200 180 GB (HGX B200) or rack-scale GB200 NVL72
• RAM: 2 TB+; Grace-CPU coherent memory in GB200 configurations
• Network: InfiniBand NDR/XDR (400–800 Gbps), rail-optimized topology
• Cooling & power: DLC mandatory; ~120 kW per NVL72 rack — practical only in purpose-built facilities, which for most organizations means colocation or a specialized bare-metal provider
• Fits: frontier-scale training, massive FP4 inference fleets

5. Inference-Optimized Node

• GPU: 2–4x L40S (cost-efficient), MIG-partitioned H100/H200 (multi-tenant serving), or MI300X (largest models on a single card thanks to 192 GB VRAM)
• CPU: 32 cores
• RAM: 128–256 GB
• Storage: 2 TB NVMe (model weights + KV-cache spillover)
• Software: vLLM or TensorRT-LLM behind Triton; FP8 quantization by default, INT4/AWQ where VRAM is the constraint, FP4 on Blackwell hardware
• Fits: low-latency LLM serving with continuous batching and Kubernetes autoscaling — remember that engine and quantization choice here moves throughput more than the GPU tier does

FAQ

1. How much does it cost to set up a machine learning server? Entry-level single-GPU builds start around $3,000–$7,000. A fine-tuning server with dual professional GPUs runs $15,000–$40,000; 8x H200 nodes range from $250,000 upward, and Blackwell rack-scale systems reach into the millions — before counting the liquid-cooling and power infrastructure they require. Renting equivalent dedicated bare-metal servers typically costs from a few hundred to several thousand dollars per month with no upfront capital, which is often the smarter path until utilization is proven.

2. Do I really need a GPU, or is a CPU enough? It depends on the workload. For deep learning (neural networks, transformers, LLMs), a GPU is effectively mandatory — training that takes hours on a GPU can take weeks on a CPU. For classical ML (gradient boosting, random forests, regression on tabular data), a well-specced multi-core CPU server is usually sufficient and more cost-effective.

3. Which GPU generation should I choose in 2026 — A100, H100/H200, or Blackwell? Match the generation to your budget and facility. A100 and L40S remain the best value on the bare-metal rental market for fine-tuning and mainstream training. H200 is the current mainstream default for new LLM deployments thanks to its 141 GB of HBM3e. B200/GB200 delivers 2–3x Hopper performance and native FP4 inference, but requires liquid cooling and rack power densities (up to ~120 kW) that only purpose-built facilities support. AMD MI300X is worth evaluating for inference of very large models — 192 GB per card, with mature vLLM/PyTorch support on ROCm.

4. Is it better to rent a machine learning server or use the public cloud? Cloud GPU instances excel at short bursts and elastic experimentation but become expensive at sustained utilization, and data egress fees add up; proprietary cloud chips (Trainium/Inferentia) cut costs further but lock you into one vendor’s toolchain. Rented dedicated (bare-metal) servers deliver 2–5x lower cost per GPU-hour at steady load, consistent performance without virtualization overhead, and full control over data residency — often decisive for regulated industries. Many teams run a hybrid: dedicated servers for baseline load, cloud for peaks.

5. What software do I need for LLM inference, beyond just PyTorch? A dedicated serving engine is essential: vLLM (open-source standard, runs on NVIDIA and AMD), TensorRT-LLM (peak NVIDIA performance), or TGI (simplest Hugging Face-native option), typically fronted by Triton Inference Server. Combine it with quantization — FP8 on Hopper/Blackwell for near-lossless memory savings, INT4/AWQ/GPTQ to fit larger models, FP4 on Blackwell for maximum throughput. The engine + quantization combination often changes throughput by 2–5x on identical hardware, so benchmark it before sizing your GPU fleet.