AMD Instinct MI300X Accelerator

Overview#

MI300X is the GPU AMD bet the data centre on, and the part that put ROCm into hyperscaler production. Announced at AMD's Advancing AI event in December 2023 and shipping in volume from Q1 2024, it packages eight CDNA 3 compute chiplets (XCDs, 38 Compute Units each, 304 CUs total) and four memory I/O dies (IODs) on a single 3D-stacked CoWoS package, with eight HBM3 stacks around the perimeter totalling 192 GB at 5.3 TB/s. No 2024-era NVIDIA part — H100, H200 (141 GB) or B100 (180 GB) — matched that per-GPU memory ceiling.

The pitch is straightforward and the procurement reality matches it. Inference of dense LLMs at 70B-180B parameters fits on a single MI300X where it would require tensor parallelism across two-to-four H100s, simplifying serving topology, eliminating an entire class of NCCL collective overhead and reducing tail latency. AMD priced the part as a credible second source for large-model inference, and through 2024-2025 it shipped in volume to Microsoft (Azure ND MI300X), Meta (production llama serving), Oracle (OCI BM.GPU.MI300X.8) and a long tail of Tier-1/Tier-2 neoclouds. By 2026, MI300X is the established multi-vendor anchor in serious enterprise GPU procurement — the part that lets a CIO say 'not single-vendor' without buying a science project.

The honest trade-off is software. ROCm has closed most of the CUDA gap through 2024-2026 — PyTorch, vLLM, SGLang, Hugging Face Transformers, Flash Attention 2/3, paged-KV attention and FP8 GEMM all run with parity or near-parity to CUDA — but specific paths (TensorRT-LLM, certain custom Triton kernels, some bleeding-edge quantisation schemes) remain CUDA-exclusive. The MI300X case is strongest for inference of widely-supported open-weight models on ROCm-compatible serving stacks; it weakens for teams whose production path is tied to TensorRT-LLM engines or NVIDIA's TAO / ModelOpt toolchain.

This entry is the reference for teams operating MI300X alongside or instead of NVIDIA Hopper/Blackwell: full spec sheet, the sizing tables we use on InferenceBench, the ROCm software-stack baseline, the operational issues that are MI300X-specific (driver pinning, RCCL replacing NCCL, xGMI topology vs NVLink-fabric expectations), and where the part fits in the Yobitel multi-vendor compute strategy. Yobitel NeoCloud offers MI300X OAM capacity in UK and EU regions with NCSC OFFICIAL alignment as the deliberate multi-vendor anchor alongside the NVIDIA H-series fleet, and Yobibyte schedules ROCm-compatible serving workloads onto MI300X pools whenever residency and software stack permit. This entry helps you decide when MI300X is the right pick for your workload and how to size and price it on Yobitel NeoCloud or your own cluster.

How it works: CDNA 3 chiplet architecture and the Matrix Core engine#

MI300X breaks the GPU into chiplets. Eight XCD dies — each containing 38 CDNA 3 Compute Units, 304 CUs in total — sit on top of four IO dies (IODs) that provide HBM3 memory controllers, AMD Infinity Cache (256 MB), and the inter-chiplet Infinity Fabric. All of this is 3D-stacked on a CoWoS organic substrate with eight HBM3 stacks (24 GB each, 192 GB total) around the perimeter. The chiplet approach is what gave AMD time-to-market in 2023-2024: smaller dies yield better than monolithic 800 mm² parts, and the same XCDs are reused in MI300A (the APU variant pairing GPU chiplets with Zen 4 CPU chiplets) without redesigning the silicon.

Each CU contains 64 stream processors organised across SIMD units, four AMD Matrix Cores (the equivalent of NVIDIA Tensor cores), 64 KB of LDS (Local Data Share, the AMD equivalent of CUDA shared memory) and dedicated wavefront schedulers. Matrix Cores execute mixed-precision dot products natively for FP64, FP32, TF32, BF16, FP16, FP8 (E4M3 and E5M2, matching NVIDIA's convention), INT8 and INT4. The Matrix Core engine on CDNA 3 supports 2:4 structured sparsity for inference workloads, with the same throughput-doubling pattern as Tensor cores.

The 256 MB Infinity Cache on the IODs is the secret sauce for memory-bound serving. It absorbs reused KV-cache reads and weight-tensor tiles, dramatically reducing HBM bandwidth pressure on long-context decode. Combined with the 5.3 TB/s HBM3 ceiling, MI300X frequently sustains higher effective bandwidth on real LLM serving than the headline number suggests — the 'serving win' on memory-bound shapes is wider than a pure bandwidth comparison would predict.

The trade-off is software complexity at chiplet boundaries. Crossing XCD-to-XCD boundaries adds latency and bandwidth constraints that the runtime must be aware of; ROCm libraries were tuned through 2024-2025 to hide most of this, but specific kernel patterns (small-batch, irregular memory access, cross-XCD reductions) can still hit chiplet-edge bottlenecks. The MI300X case is strongest on regular, batchable workloads (LLM serving with paged KV, batched training) and weakest on small-batch latency-critical inference.

• Eight XCDs (CDNA 3 compute dies) + four IODs (memory/IO dies) on a 3D-stacked CoWoS package.
• 304 Compute Units total (38 CUs per XCD); 1,216 Matrix Cores (4 per CU); 19,456 stream processors.
• Eight HBM3 stacks (24 GB each) totalling 192 GB at 5.3 TB/s.
• 256 MB AMD Infinity Cache on the IODs; bandwidth amplifier for memory-bound serving.
• Native FP8 E4M3 / E5M2, BF16, FP16, TF32, FP64; 2:4 structured sparsity supported.
• xGMI (Infinity Fabric over PCIe): 128 GB/s per link, 7 links per GPU on an OAM baseboard = 896 GB/s aggregate.
• OAM form factor at 750 W TDP; air-cooled chassis viable up to 8 GPUs per node.

Reference: full specification sheet#

Authoritative figures for the MI300X OAM SKU at standard 750 W TDP. AMD's 'Matrix' throughput numbers are roughly comparable to NVIDIA's 'Tensor' numbers but use different sparsity conventions — confirm whether published figures are dense or sparse before comparing.

Workload pattern A: Llama 3.1 70B BF16 on a single MI300X#

The pattern that defined MI300X's serving value proposition. Llama 3.1 70B in BF16 occupies roughly 140 GB of weight memory — fits on one MI300X with 52 GB headroom for KV cache, working activations and Infinity Cache pressure. On H100 SXM5 this requires TP=4 across four cards (80 GB each is not enough for weights alone); on MI300X it runs as a single-card replica, eliminating NCCL/RCCL collectives and halving the inter-GPU traffic budget. Throughput at 1,200-1,500 output TPS sustained is competitive with H100 TP=4 at materially lower cost.

# 70B BF16 on 1x MI300X with vLLM + ROCm 6.3
HIP_VISIBLE_DEVICES=0 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 1 \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 32 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --host 0.0.0.0 --port 8000

# Optional: pre-quantise to FP8 with AMD Quark for ~1.6x throughput
quark-quantize \
  --model meta-llama/Meta-Llama-3.1-70B-Instruct \
  --output ./Llama-3.1-70B-Instruct-FP8-AMD \
  --quant-format fp8 --calib-dataset wikitext-2

HIP_VISIBLE_DEVICES=0 vllm serve ./Llama-3.1-70B-Instruct-FP8-AMD \
  --tensor-parallel-size 1 \
  --quantization fp8 --kv-cache-dtype fp8_e5m2 \
  --max-model-len 131072 --max-num-seqs 64 \
  --gpu-memory-utilization 0.92

Workload pattern B: 8x MI300X training cluster#

Multi-GPU training on an 8x MI300X OAM baseboard with RCCL (the ROCm replacement for NCCL) handling collectives. Same PyTorch / Megatron-LM patterns as on NVIDIA — the code changes are minimal — but with `HIP_VISIBLE_DEVICES` instead of `CUDA_VISIBLE_DEVICES`, RCCL instead of NCCL, and Flash Attention's ROCm port (`flash-attn` with the `ROCM=1` build flag). Training a 13B model on 100B tokens completes in roughly 6-9 days on 8x MI300X — between H100 SXM5 and H100 PCIe Gen5 in practice on FLOPS-bound runs.

• RCCL is API-compatible with NCCL; PyTorch keeps the backend name 'nccl' for source compatibility.
• Flash Attention 2 is fully ported to ROCm; Flash Attention 3 ports landed in ROCm 6.3 (production-acceptable from late 2025).
• Megatron-Core supports ROCm with the same TP/PP/DP semantics as on NVIDIA; rebuild from source against ROCm 6.3.
• MFU on training: 50-60 % typical on MI300X for 7B-13B BF16, slightly below H100's 60-65 % at iso-precision.

# train_13b.py — 13B BF16 pretraining on 8x MI300X with PyTorch + Megatron-LM
# Deps: pip install "torch==2.4.0+rocm6.3" megatron-core flash-attn

import os, torch
import torch.distributed as dist
import torch.nn.parallel as ddp
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

os.environ.setdefault("NCCL_DEBUG", "INFO")
# RCCL is API-compatible with NCCL; PyTorch uses 'nccl' as the backend name on ROCm too.
dist.init_process_group(backend="nccl")
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)  # On ROCm, torch.cuda maps to HIP

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-13b-hf",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # ROCm port of FA2
).to(local_rank)
model = ddp.DistributedDataParallel(model, device_ids=[local_rank])

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-13b-hf")
ds = load_dataset("EleutherAI/pile", split="train", streaming=True)
# ... standard PyTorch training loop with gradient accumulation ...

# Launch with:
#   torchrun --nproc_per_node=8 --nnodes=1 train_13b.py

Workload pattern C: Mixtral 8x7B serving with expert parallelism#

MoE serving on 2x MI300X with vLLM's expert-parallel scheduler. Mixtral 8x7B in BF16 (roughly 90 GB total weights) fits comfortably on one MI300X with KV-cache room to spare; the 2-GPU topology gives expert sharding for higher concurrent throughput at the cost of additional xGMI traffic on every routing decision.

# Mixtral 8x7B BF16 on 2x MI300X with vLLM expert parallelism
HIP_VISIBLE_DEVICES=0,1 \
NCCL_P2P_LEVEL=NVL \
vllm serve mistralai/Mixtral-8x7B-Instruct-v0.1 \
  --tensor-parallel-size 2 \
  --expert-parallel-size 2 \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --max-num-batched-tokens 32768 \
  --max-num-seqs 64 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --host 0.0.0.0 --port 8000

Sizing and capacity planning#

Sizing tables we use on InferenceBench for MI300X. All figures assume ROCm 6.3, vLLM 0.6+ with the ROCm backend, BF16 weights (or FP8 where flagged via AMD Quark) and xGMI-local placement. The headline against H100 SXM5: most memory-pressured rows collapse to fewer GPUs per replica thanks to the 192 GB pool; FLOPS-bound rows show MI300X sitting between H100 SXM5 and H100 PCIe Gen5.

• Training rule of thumb: 1 trillion tokens x 70B parameters at BF16 needs roughly 350-450 MI300X-days on 64-GPU clusters — between H100 SXM5 (250-350) and H100 PCIe Gen5 (450-550).
• Memory ceiling for a single MI300X: weights + KV cache + activations + cuBLAS-equivalent scratch < 188 GB usable. Above 188 GB expect OOMs.
• For 500 RPS at 4K-token output, Llama 3.1 70B BF16 needs roughly 4-5 MI300X replicas vs 6-8 H100 SXM5 FP8 replicas — typical fleet compression ratio of 1.5x.
• xGMI overhead at TP=8 inside one OAM baseboard: 10-14 % of step time for 70B BF16 — slightly higher than NVLink 4.0's 6-9 % on H100, reflecting the lower aggregate fabric bandwidth.
• Spot/preemptible MI300X capacity is limited in 2026 — most providers reserve MI300X for committed customers.

Cost & TCO#

MI300X pricing clears materially below H100 SXM5 at every commitment tier in 2026 — typically 15-25 % cheaper per GPU-hour on like-for-like hyperscaler / neocloud SKUs. The combination of 192 GB HBM3 (collapsing TP topologies) and lower hourly cost makes cost-per-token competitive with or better than H100 BF16 on memory-bound serving, and broadly competitive with H100 FP8 on FP8-quantised models via AMD Quark.

• Cost-per-million-output-tokens on Llama 3.1 70B FP8 (Quark) at 32K context, 1x MI300X at $4.00/GPU-hr and 2,000 TPS sustained: roughly $0.55 per million tokens — competitive with H100 SXM5 FP8 ($0.50) and ~25 % cheaper than H200 FP8 ($0.74).
• Commitment savings: 1y reserved ~= 25 % off on-demand, 3y reserved ~= 38 % off — track H100 closely.
• FP8 via AMD Quark yields ~1.5-1.6x throughput vs BF16 at iso-quality on most chat models — same lever as Transformer Engine FP8 on NVIDIA.
• Egress and inter-region data movement frequently exceed 10 % of total MI300X bill at hyperscalers — collocate model artefacts with compute.
• Multi-vendor procurement value: an MI300X commitment alongside an H100 commitment hedges single-vendor supply risk and gives FinOps leverage at renewal time.

Migration and alternatives#

When MI300X is the right choice and when it isn't. The dominant migrations are H100/H200 -> MI300X for memory-pressured serving on ROCm-compatible workloads, and MI300X -> MI325X / MI355X for AMD generation rolls. Two heuristics: choose MI300X when your serving stack is on vLLM, SGLang or Hugging Face Transformers and you want a multi-vendor supply hedge; stay on H-series when your production path is bound to TensorRT-LLM engines, ModelOpt FP4 calibration, or NVIDIA CC-on FedRAMP attestation.

Pitfalls and operational notes#

Three categories account for the majority of MI300X production incidents: ROCm version pinning, RCCL behavioural differences from NCCL, and xGMI topology assumptions carried over from NVLink Switch System expectations. The MI300X has the silicon to deliver — the operational risk lives in the software baseline.

ROCm version pinning is the single highest-leverage operational discipline on MI300X. ROCm releases move faster than the data-centre community is used to from CUDA and break compatibility silently across minor versions. The pattern is: pin ROCm at 6.3.x as the 2026 baseline, pin matching torch wheels from AMD's wheel index, pin matching vLLM wheels from the same index, and treat the combination as one immutable artefact. Mixing ROCm 6.0 drivers with ROCm 6.3 user-space libraries is the single most common production outage on MI300X fleets. The `HIP error: invalid device function` symptom on kernel launch is the canonical fingerprint — it means the user-space libraries are newer than the kernel-space driver, or the custom kernels were compiled against the wrong `gfx` target. Rebuild custom kernels with `--amdgpu-target=gfx942` and tighten the version pins.

RCCL is API-compatible with NCCL but the topology autodetection behaviour differs enough to matter. RCCL AllReduce hangs at job start usually trace to topology autodetection failing on heterogeneous xGMI + Ethernet clusters; setting `RCCL_TOPO_FILE=/etc/rccl/topo.xml` (generate with `rccl-tests`) and verifying with `NCCL_DEBUG=INFO` (RCCL respects the NCCL environment variables) resolves it. xGMI aggregate bandwidth below 600 GB/s usually means a single xGMI link is down or mis-routed, not a baseboard fault — `rocm-smi --showxgmierr` plus a module reseat is the first step; persistent issues usually trace to a topology assumption (xGMI is a 7-link mesh, not the fully-connected NVSwitch fabric some teams expect).

Three software-stack pitfalls round out the high-frequency list. Flash Attention import failures on ROCm 6.0 happen because production-grade FA2 only landed reliably in ROCm 6.3+; upgrade or fall back to `attn_implementation='sdpa'`. `MIOpenStatusUnsupportedOp` during model load means the MIOpen autotuning database lacks entries for the model's specific op shapes — run with `MIOPEN_FIND_MODE=NORMAL`, let autotune run once, and persist `~/.config/miopen/`. FP8 quantisation accuracy regressions after AMD Quark trace to a too-small calibration dataset or AWQ block-size mismatch; increase `--calib-samples` to 2048 or higher and consider excluding attention QKV projections from FP8.

Inference throughput at half the expected rate almost always means vLLM is running in CUDA-emulation mode instead of the native ROCm backend — check `vllm --version` reports `rocm`, uninstall any mainline pip vLLM wheels, and reinstall from AMD's wheel index. Compute-partition mode flips (SPX to CPX) are destructive — drain workloads, mark the node unschedulable, change mode with `amd-smi set --compute-partition CPX`, then redeploy. And MI300X hosts must be vendor-pure: do not co-install NVIDIA drivers on the same kernel — kernel panics on driver upgrade trace to that pattern directly.

On isolation and compliance: MI300X exposes SR-IOV with up to 8 virtual functions per GPU and compute-partition modes (SPX exposes the full 304 CUs as one device; CPX exposes 8 partitions of 38 CUs each with isolated HBM allocations) for multi-tenant scheduling. There is no NVIDIA-CC-on-equivalent attested confidential-compute path in 2026 — for FedRAMP-Moderate sovereign workloads, NVIDIA H-series with CC-on remains the recommended SKU. UK NCSC and EU GDPR postures are achievable on the host environment but the GPU itself does not currently provide attested confidential-compute equivalents.

Where this fits in the Yobitel stack#

MI300X is Yobitel's primary multi-vendor anchor in 2026. Yobibyte — our AI-native managed platform — places memory-pressured serving workloads on MI300X pools where customer compliance permits and where the model is ROCm-compatible (the overwhelming majority of open-weight models are), falling back to NVIDIA H-series for FedRAMP-Moderate sovereign workloads, TensorRT-LLM-bound paths and ModelOpt FP4 calibration. The vLLM and PyTorch commands in this entry are exactly what Yobibyte reconciles under the hood; the customer specifies model, region, replica count and spend cap, and the platform selects the SKU and the FP precision — including which vendor.

Omniscient Compute — our cross-cloud capacity broker — indexes MI300X capacity across Azure ND MI300X v5, Oracle BM.GPU.MI300X.8, and Tier-1/Tier-2 neocloud partners (TensorWave, Crusoe, Hot Aisle, Vultr), normalises pricing onto the FinOps Foundation FOCUS spec, and arbitrages workloads to the cheapest region that meets the workspace's residency posture. Because MI300X frequently clears 15-25 % cheaper than H100 SXM5 at the same commitment tier, multi-vendor procurement via Omniscient Compute is one of the highest-impact FinOps levers in the stack — often the single largest cost-saving recommendation in customer onboarding reviews.

InferenceBench — our public, reproducible benchmarking harness — publishes MI300X throughput, latency and cost-per-token numbers for every major open-weight model across vLLM and SGLang on ROCm, with side-by-side H100/H200/B200 comparisons. The sizing tables above are anchored on InferenceBench runs. If you are sizing a 2026 multi-vendor footprint, start with InferenceBench to see where MI300X wins on cost-per-token and where it loses, lift the platform configuration into the Yobibyte workspace, and let Omniscient Compute pick the region and the vendor.