Volcano Scheduler

Overview#

Volcano is a Kubernetes-native batch scheduler purpose-built for AI / ML / HPC / Big Data workloads. The default kube-scheduler optimises for stateless single-pod workloads — REST services, ingestion daemons, control-plane controllers — and treats every pod as an independent placement decision. That model breaks the moment you submit a distributed training job that needs 64 H100 GPUs across eight nodes to launch atomically: the scheduler admits the first 60 pods, the cluster runs out of fitting GPUs, the remaining four sit Pending, and the 60 admitted ranks burn GPU-hours waiting for the rendezvous that will never complete.

Volcano was created at Huawei in 2019 (initially called `kube-batch`) to fix exactly this class of problem. It introduces a `PodGroup` abstraction with `minMember` and `minResources` so the scheduler can reason about "the whole job is admitted or none of it is"; layers a queue-based fair-share model with DRF (Dominant Resource Fairness) across GPU + CPU + memory; adds topology-aware placement so an eight-rank tensor-parallel job lands on a single NVLink / NVSwitch island; and provides HPC-style preemption with reclaim so a high-priority production training run can evict opportunistic backfill without losing accounting fidelity.

Volcano joined CNCF as a Sandbox project in 2020 and was promoted to Incubating in 2022. By mid-2026 it is on v1.10.x, supports Kubernetes 1.27-1.33, and ships pre-built integrations for PyTorch, TensorFlow, MPI (Horovod / OpenMPI), Ray, Spark, Flink and the Kubeflow Training Operator. It is dual-purpose: it can either fully replace the default scheduler in a Volcano-only namespace, or run side-by-side with kube-scheduler in a mixed cluster where only workloads that opt in via `schedulerName: volcano` use Volcano's placement logic.

This entry helps you decide when Volcano is the right addition to a Kubernetes cluster, how to wire it up against the NVIDIA GPU Operator and your training operators, how to size the resulting queue / cohort plane, and how its job model differs from the lighter-weight Kueue alternative. Yobibyte runs Volcano under the hood across every Yobitel NeoCloud region so that Yobibyte customers never experience a partial gang-admission failure — this entry documents the surface for teams that operate their own clusters or want to understand what Yobibyte provides on their behalf.

Quick start#

The fastest sane path is the upstream Helm install plus a single `Job` (Volcano's own CRD, distinct from `batch/v1 Job`) running a four-rank MPI worker. The five commands below install Volcano, define a queue, submit a gang-scheduled MPI job and observe the admission decision. Run them against a cluster that already has the NVIDIA GPU Operator installed and at least four `nvidia.com/gpu` resources free.

# 1. Install Volcano via the upstream Helm chart
helm repo add volcano-sh https://volcano-sh.github.io/helm-charts
helm repo update
helm install --wait volcano volcano-sh/volcano \
    --version "1.10.0" \
    --namespace volcano-system --create-namespace

# 2. Confirm the controller, scheduler and admission webhook are Ready
kubectl -n volcano-system get pods

# 3. Create a queue with a 16-GPU weight
cat <<'YAML' | kubectl apply -f -
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata: { name: training }
spec:
  weight: 4
  capability:
    nvidia.com/gpu: "16"
YAML

# 4. Submit a gang-scheduled MPI job (4 workers, all-or-nothing)
cat <<'YAML' | kubectl apply -f -
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata: { name: nccl-smoke }
spec:
  schedulerName: volcano
  minAvailable: 4
  queue: training
  plugins:
    ssh: []
    svc: []
    env: []
  tasks:
    - name: worker
      replicas: 4
      template:
        spec:
          containers:
            - name: nccl-test
              image: nvcr.io/nvidia/pytorch:24.10-py3
              command: ["sleep", "infinity"]
              resources:
                limits: { nvidia.com/gpu: 1 }
YAML

# 5. Inspect gang-admission decision
kubectl get podgroup
kubectl describe podgroup nccl-smoke
kubectl get pods -l volcano.sh/job-name=nccl-smoke

How it works#

Internally Volcano is three components running in `volcano-system`: the `vc-controller-manager` (reconciles `Job` and `PodGroup` CRDs into pods), the `vc-scheduler` (the scheduler itself — replaces or supplements kube-scheduler), and the `vc-webhook-manager` (validating + mutating admission for Volcano CRDs). The scheduler is built around a session model: every scheduling tick (default 1 s) opens a session, walks a configurable pipeline of actions (`enqueue`, `allocate`, `backfill`, `preempt`, `reclaim`), and closes the session by committing the resulting bindings to the API server. Plugins implement the policy each action consults: `gang`, `priority`, `drf`, `predicates`, `nodeorder`, `proportion`, `binpack`, `topology-aware`, `numa-aware`, `task-topology`.

The `gang` plugin is the headline. A `PodGroup` (created automatically by Volcano's `Job`, or explicitly for raw pods) declares `minAvailable` and `minResources`. The scheduler will not transition the group to `Inqueue` (allowed to be admitted) until enough total cluster capacity exists; will not start binding individual pods until the entire group can be bound; and will preempt or reclaim only when the gang as a whole can be satisfied after the eviction. This eliminates the partial-admission deadlock that breaks the default scheduler for distributed training.

Queues are first-class. A `Queue` CRD has a `weight` (DRF share), `capability` (hard ceiling per resource), `reclaimable` flag (can higher-priority queues claw back from this queue?) and `priority`. The `proportion` plugin allocates fair shares across queues per resource dimension; the `drf` plugin handles the multi-dimensional case (GPU + CPU + memory) so a CPU-heavy data-prep job and a GPU-heavy training job get fair allocations on the dimension that dominates each. The `reclaim` action then enforces the contract — when a higher-priority queue arrives, opportunistic borrowers are evicted in a deterministic order until the guarantee is restored.

Topology awareness wires Volcano to the underlying fabric. The `topology-aware` plugin reads node labels emitted by NFD and the GPU Operator (`nvidia.com/gpu.nvlink.domain`, `topology.kubernetes.io/zone`, custom `volcano.sh/topology=rack-3-leaf-4`) and groups nodes into hierarchical domains. The `task-topology` plugin then bin-packs an MPI / NCCL job into the smallest fitting domain — an 8-rank tensor-parallel job lands on a single 8-way NVLink island, a 64-rank pipeline-parallel job lands on a single rack, a 512-rank pretraining job picks the smallest spine-leaf cluster that fits.

• Session-based scheduling — actions execute per tick over a snapshot, commits are atomic; failed bindings are reverted without leaking partial state.
• Plugins are config-driven via the `volcano-scheduler-configmap` — enable / disable / reorder without rebuilding the binary.
• Plugin extension model — `gang`, `priority`, `drf`, `binpack`, `topology-aware`, `numa-aware`, `task-topology`, `tdm` (time-division multiplexing), `proportion`, `overcommit`, `usage`, `rescheduling`.
• Per-task templates — a `Job` can declare multiple `tasks` (e.g. `master` + `worker` + `param-server`), each with its own replica count, image and resource shape, all admitted together as a single gang.
• Built-in plugins for framework idioms — `pytorch` (sets `MASTER_ADDR` / `RANK` / `WORLD_SIZE`), `tensorflow` (TF_CONFIG), `mpi` (mpirun rendezvous), `ssh` (SSH key fan-out), `svc` (headless service for collective bootstrap), `env` (rank-aware env vars).
• Pre-emption is policy-driven — `priority` + `victim` selection minimises killed pods; `tdm` preempts on a time-share rather than killing outright.

Reference and specifications#

The fields below are the Volcano CRD surface that matters in production. The reference covers `Job` (batch.volcano.sh/v1alpha1), `PodGroup` (scheduling.volcano.sh/v1beta1), `Queue` (scheduling.volcano.sh/v1beta1) and the scheduler ConfigMap. Defaults are taken from v1.10.0.

Workload patterns#

Three patterns cover the bulk of production Volcano deployments on Yobitel-operated clusters and on the upstream community. Each pattern uses a different combination of plugins and a different queue topology; pick the one closest to your dominant workload.

Pattern A — distributed PyTorch + Horovod gang training. The canonical Volcano use case. A single `Job` declares one or more `worker` tasks at the desired tensor / data-parallel scale, with `minAvailable` set to the full replica count and the `pytorch` (or `mpi`) plugin enabled. Horovod's `mpirun` finds peers through the headless service the `svc` plugin creates; `ssh` plugin handles key fan-out. Volcano holds the gang until the topology-aware plugin can land all ranks on the same NVLink island (or, for >8-way jobs, the same rack with InfiniBand RDMA paths).

Pattern B — multi-queue tenant fair-share on shared GPU capacity. Each tenant gets a `Queue` with a `weight` proportional to their committed share and a `capability` ceiling. Opportunistic backfill is allowed via `reclaimable: true`. Tenants submit `Job`s into their own queue; DRF allocates fair shares; when a higher-priority queue arrives, the `reclaim` action evicts opportunistic borrowers in priority order. This is the substrate Yobitel sovereign tenancies use to publish guaranteed GPU shares per tenant while keeping average utilisation high.

Pattern C — heterogeneous task types in one job. Training Operator and KubeRay drive their own pod lifecycles, but you can use Volcano's `Job` directly for jobs that have asymmetric task types — e.g. one `master` (parameter server, FP32) + many `worker` (FP16, NCCL) + a `metrics-sidecar` (CPU-only, scrapes worker telemetry to Prometheus). Each `task` declares its own replicas, template and policies; the gang admits them together.

# Pattern A: distributed PyTorch + Horovod on 8x H100 with gang scheduling
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata: { name: llama-pretrain }
spec:
  schedulerName: volcano
  minAvailable: 8
  queue: training
  priorityClassName: high-priority
  plugins:
    ssh: []
    svc: []
    env: []
    pytorch: ["--master=master", "--worker=worker", "--port=23456"]
  policies:
    - event: PodEvicted
      action: RestartJob
  tasks:
    - name: master
      replicas: 1
      template:
        spec:
          containers:
            - name: pytorch
              image: nvcr.io/nvidia/pytorch:24.10-py3
              command: ["torchrun", "--standalone", "--nproc_per_node=1", "/app/pretrain.py"]
              resources:
                limits: { nvidia.com/gpu: 1, hugepages-1Gi: 16Gi }
    - name: worker
      replicas: 7
      template:
        spec:
          containers:
            - name: pytorch
              image: nvcr.io/nvidia/pytorch:24.10-py3
              command: ["torchrun", "/app/pretrain.py"]
              resources:
                limits: { nvidia.com/gpu: 1, hugepages-1Gi: 16Gi }

---
# Pattern B: tenant queues with weighted fair-share and hard ceilings
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata: { name: tenant-acme }
spec:
  weight: 8         # 8/16 of cluster GPU share under fair contention
  reclaimable: true # opportunistic borrow allowed
  capability:       # hard ceiling
    nvidia.com/gpu: "64"
  guarantee:        # never preempted below this floor
    resource:
      nvidia.com/gpu: "16"

Sizing and capacity planning#

Volcano's footprint is modest. The control plane runs three deployments in `volcano-system` (controller-manager, scheduler, webhook-manager) that together cost roughly 1-2 vCPU and 2-4 GiB at steady state on a 100-node cluster. The scheduler is the dominant component — it walks the snapshot every tick and the work scales roughly O(pods × nodes × plugins). Past ~1,000 pods or ~500 nodes you should raise the request shape; past ~5,000 pods you should split into multiple scheduler replicas (HA mode, one active at a time).

• Single Volcano scheduler handles ~5,000 pods + 500 nodes comfortably; beyond that, partition by queue or by namespace label.
• Schedule period (default 1 s) trades latency for CPU — drop to 500 ms for high-churn clusters; raise to 2 s if pod creation lags scheduler load.
• Action pipeline matters: enable only what you need. Adding `preempt` and `reclaim` to a small homogeneous cluster wastes session time without measurable benefit.
• Each Volcano `Job` is a single etcd object plus N pods plus 1 `PodGroup` — overhead is dominated by pods, not by the CRDs themselves.
• On Yobitel NeoCloud regions Yobibyte uses, the scheduler runs HA with two replicas and a 750 ms tick — sized for the typical workspace mix on H100 + H200 capacity.

Limits and quotas#

Volcano's quota model is the `Queue.spec.capability` field — a hard ceiling on resources the queue can hold across all admitted gangs. Combined with Kubernetes `ResourceQuota` (namespace-level cap on counts and resources) and NetworkPolicy (tenant isolation), queues form the basis of hard-isolated multi-tenant GPU clusters. The limits below are the practical envelope teams hit in production.

Observability#

Volcano exposes Prometheus metrics on `:8080/metrics` from both the controller-manager and the scheduler. The metric set covers scheduler session timing, allocation success / failure rates, queue share utilisation, plugin error counters and pod-group state transitions. Combined with the standard Kubernetes scheduler audit log, this is enough to alert on starvation, deadlock and reclaim churn — the three failure modes that produce 90% of operator pages on a busy batch cluster.

The metrics worth alerting on are: PodGroup phase distribution (especially `Pending` duration), scheduler session latency, queue allocation vs capability ratios, and the reclaim action's eviction rate. Yobibyte's internal SRE alerts on the equivalent customer-facing signals (workspace pending queue depth, gang admission latency) without exposing the underlying Volcano metric names.

• `volcano_scheduler_session_duration_seconds` — per-tick session time; alert if p95 > schedule period.
• `volcano_scheduler_pod_scheduling_attempts_total` / `_failures_total` — admission throughput and failure rate.
• `volcano_e2e_scheduling_latency_milliseconds` — gang admission end-to-end latency; the SLO that matters to customers.
• `volcano_queue_allocated` / `volcano_queue_deserved` / `volcano_queue_capability` — fair-share vs ceiling per queue; ratio > 1 = overcommit.
• `volcano_podgroup_phase_count{phase=...}` — distribution of `Pending`, `Inqueue`, `Running`, `Completed`, `Failed`.
• `volcano_reclaim_total` — count of reclaim evictions; a sustained non-zero rate means the cluster is over-subscribed.
• `volcano_admission_latency_milliseconds` — webhook latency; slow admission throttles job submission.
• `workqueue_depth{name="volcano-controller"}` — controller-manager backlog; alert if growing.

# Prometheus alerts for Volcano in production
groups:
  - name: volcano-sla
    interval: 30s
    rules:
      - alert: VolcanoGangStarvation
        expr: max by (podgroup, queue) (time() - volcano_podgroup_pending_since_seconds) > 1800
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "PodGroup {{ $labels.podgroup }} in {{ $labels.queue }} pending > 30m"

      - alert: VolcanoSchedulerSlow
        expr: histogram_quantile(0.95, rate(volcano_scheduler_session_duration_seconds_bucket[5m])) > 2
        for: 10m
        labels: { severity: critical }
        annotations:
          summary: "Volcano scheduler p95 session > 2 s — cluster too large for single scheduler"

      - alert: VolcanoQueueOversubscribed
        expr: volcano_queue_allocated / volcano_queue_capability > 1.0
        for: 15m
        labels: { severity: warning }
        annotations:
          summary: "Queue {{ $labels.queue }} allocated > capability — investigate borrowing"

      - alert: VolcanoReclaimThrash
        expr: rate(volcano_reclaim_total[10m]) > 0.5
        for: 30m
        labels: { severity: critical }
        annotations:
          summary: "Reclaim eviction rate > 0.5/s — quota plane misconfigured"

Cost and FinOps#

Volcano itself is free (Apache 2.0). The cost surface is the GPU capacity Volcano allocates. Where Volcano changes FinOps is in the conversion from raw GPU-hours to *useful* GPU-hours: by eliminating partial-admission deadlock, Volcano can lift effective utilisation on a busy training cluster from ~60-70% (default scheduler with no gang) to ~85-90% (Volcano with topology-aware reclaim). On a 64-node H100 cluster at ~$3.00/GPU/hr on Yobitel NeoCloud, that is roughly $90,000-$130,000/month of recovered productivity.

• Effective utilisation lift — measure `volcano_queue_allocated` vs cluster capacity over 30 days; the gap to 100% is what reclaim + backfill can recover.
• Yobitel NeoCloud H100 SXM5 list — roughly $3.00/GPU/hr on-demand, $2.00/GPU/hr reserved, ~$1.50/GPU/hr opportunistic backfill via Volcano's lowest-priority queue.
• Recovered productivity from gang admission alone — typically 10-20% lift on training-heavy clusters that previously saw partial deadlock.
• Reclaim overhead — each reclaim event kills a pod and burns its checkpoint window; budget for one re-do every reclaim cycle on opportunistic backfill jobs.
• Quota-pricing alignment — if you sell capacity to internal teams, set `Queue.spec.weight` to match the dollar committed, and surface `volcano_queue_allocated` as a chargeback feed.
• Yobibyte's workspace billing surface is the customer-facing equivalent — Yobitel runs the Volcano + Kueue plane and bills the customer in USD per GPU-hour without exposing the raw queue metrics.

Security and compliance#

Volcano runs as a privileged controller — it can read every pod and node in the cluster, create / delete pods on behalf of users, and mutate scheduling decisions. The standard mitigations apply: namespace-scoped RBAC for end users (they create `Job` and `PodGroup` in their own namespace, but never in `volcano-system`), restricted PodSecurity for everything except the named controller pods, and audit logging on every `Job` / `Queue` mutation. For UK NCSC OFFICIAL workloads, Volcano sits inside the sovereign perimeter on Yobitel-operated clusters — the scheduler never makes calls to a SaaS control plane.

Multi-tenant isolation comes from the combination of Queue capability ceilings, ResourceQuota, NetworkPolicy and PodSecurity. A tenant can never exceed its queue's `capability`; a tenant's pods can never see another tenant's pods on the network without a NetworkPolicy exception; a tenant's pods cannot mount host paths or run privileged. Volcano enforces the resource ceiling but not the network or security boundary — those are standard Kubernetes primitives layered on top.

Reclaim and preemption are auditable. Every reclaim event is a Kubernetes event with the victim pod, the claimant queue and the reason, fed to the standard audit pipeline. For SOC 2 and ISO 27001 evidence, the audit log plus the Volcano metric stream is sufficient to demonstrate that resource shares were honoured per the contracted SLA.

Migration and alternatives#

Most clusters that need Volcano migrate from one of three starting points: default kube-scheduler (which produces partial-admission deadlock on distributed training), Kueue (which queues jobs but does not gang-schedule individual pods), or a legacy YARN / Slurm cluster (which has gang semantics but lives outside Kubernetes). The migration playbook differs by source.

The dominant 2026 alternative is Kueue — see [[kueue]] for the full comparison. The short version: Kueue queues whole jobs at admission and then delegates pod placement to the default scheduler; Volcano replaces the scheduler entirely with a gang-aware pipeline. Kueue is lighter touch and easier to audit; Volcano is more powerful when you need topology-aware placement and HPC reclaim. Many production clusters run both: Kueue at the platform level for fair-share queueing across teams, Volcano in the training-only namespace for the actual gang admission.

Troubleshooting#

The error patterns below cover roughly 80% of production Volcano incidents observed on Yobitel-operated fleets and on the upstream community tracker. Each row maps a symptom to the underlying mechanism and the minimum-viable fix.

Where this fits in the Yobitel stack#

Volcano is the gang-scheduling substrate under every distributed training and large multi-rank inference job that Yobibyte runs on Yobitel NeoCloud. When a Yobibyte customer launches an 8-rank tensor-parallel inference deployment or a 64-rank pretraining job through their workspace, the underlying job is admitted by Volcano with gang semantics on Yobitel-operated capacity — the customer never sees a partial admission, never burns budget on stranded ranks, and never has to author a `PodGroup` CRD themselves. Yobibyte presents the customer-facing surface as a workspace, a model name and a region; Volcano handles the atomic admission on the back end.

On Yobitel-managed clusters, Volcano is installed via GitOps from the platform's Argo CD root, paired with the NVIDIA GPU Operator (for the hardware layer) and Kueue (for cross-team fair-share at the platform level). Topology labels emitted by NFD and the GPU Operator drive Volcano's `task-topology` plugin so that NCCL collectives stay inside NVLink islands by default and fall back to InfiniBand-connected racks only when island capacity is exhausted. The InferenceBench benchmark engine uses the same plane for reproducible, gang-admitted benchmark runs — every benchmark is admitted atomically or rejected, never partial.

For UK and EU sovereign tenancies, Volcano runs inside the sovereign perimeter on Yobitel-operated clusters under the NCSC Cloud Security Principles and OFFICIAL handling caveat — no SaaS control plane, audit logs feed the regional SIEM, and queue shares are documented in the customer's contracted SLA. Customers who want the gang-scheduling primitive but not the operations burden consume it through Yobibyte; customers who want to run their own cluster with Yobitel Managed Operations get Volcano installed, tuned and on-call covered as part of the engagement.