dev-tools 9 min read

SkyPilot: keeping AI training inside one NVLink rack

SkyPilot's open source scheduler keeps multi-node AI training inside one NVLink rack, and its topology-aware placement claims 27% more usable GPUs.

By
Share: X in
An NVL72 rack next to its non-blocking NVSwitch fabric, which connects every GPU in the rack

TL;DR: SkyPilot is an Apache 2.0 framework that runs the same job YAML across Kubernetes, Slurm, or more than twenty clouds, and it recently published a study on why multi-node training jobs need to stay inside a single Grace Blackwell NVL72 rack. The open source project gives you gang scheduling and a portable task spec, while the rack-aware scheduler that places 16-node slices atomically on one rack ships in the commercial SkyPilot Platform, which is built on Kueue. If you run GPUs at all, the open source half is worth installing.

What it is

SkyPilot is a command line tool and Python library that takes a task description and runs it wherever your GPUs happen to live. You write one YAML file, and the same file runs on a Kubernetes cluster, a Slurm cluster, or AWS, GCP, Azure, RunPod, Lambda, Vast.ai, or anything else in its provider list.

The model is BYOC, bring your own cloud. Resources are created inside your own accounts, VPCs, and clusters, so there is no SkyPilot-hosted sandbox in the middle and no separate egress bill for moving data around. The project calls itself an AI compute platform, and at 10.8k GitHub stars it has turned into the default answer when someone asks how to run the same training job on three different clouds.

The task file is deliberately boring. You name your accelerators, how many nodes you want, where your code lives, what to install, and what to run:

resources:
  accelerators: A100:8

num_nodes: 4

workdir: ~/torch_examples

setup: |
  cd mnist
  pip install -r requirements.txt

run: |
  cd mnist
  python main.py --epochs 1

That is the whole contract. num_nodes includes the head node, and the file is portable because none of it mentions a specific provider.

Why it matters

The reason SkyPilot published a study this week, rather than a feature release, is that the hard part of GPU infrastructure stopped being provisioning and became placement.

An NVL72 rack puts 72 GPUs on one NVLink fabric. Every GPU has an 1.8 TB/s bidirectional link to the rest of the rack, and the 18 NVSwitch chips behind them give the rack 130 TB/s of internal bandwidth with uniform latency. Cross-rack traffic falls back to InfiniBand at 400 Gb/s per direction. That is roughly a ninefold drop, and for a communication-heavy training run it is the difference between the job being compute bound and being network bound.

So the hardware is fast, but only for jobs that stay in one rack. A naive scheduler scatters the workers of a single multi-node job across several racks, and the speedup you paid for never happens.

The other half of the problem is that GPUs fail, and a 72 GPU rack is a much bigger failure target than the 4 GPU nodes that used to make up a rack. Meta has measured a mean time between interruptions of roughly 490 days per 4-GPU node, so dividing by 18 gives the rack a figure of about 27 days. On a 128-rack cluster, that works out to a full rack job becoming unschedulable roughly every five hours.

Getting it running

Installation is a single command. The extras are the providers you want, and you can leave most of them out.

# Install with uv. Trim the provider list to what you actually use.
uv pip install "skypilot[kubernetes,aws,gcp]"

Pip, a nightly build, and installing from source all work too. The docs live at docs.skypilot.co.

Once you are configured against a cluster, launching a task is one command:

sky launch my_task.yaml

SkyPilot picks the cheapest available infrastructure that satisfies your resource request, syncs the working directory, runs setup, runs the job, and tears the cluster down when it finishes unless you asked it to keep it. The four commands you will use most:

CommandWhat it does
sky launch my_task.yamlProvision and run a task
sky statusShow clusters, jobs, and their states
sky logs JOB_IDStream output from a running job
sky cancel JOB_IDStop a job
sky queue myclusterList queued jobs on an existing cluster

For running many jobs against a cluster you already own, sky exec submits work to it and sky queue shows what is pending.

What the topology-aware scheduler actually does

The scheduler described in the study is not in the open source repository. It lives in SkyPilot Platform, the commercial product that came out of a $20M seed round announced in July, and its docs sit behind a login.

What the study describes, in the author’s own summary on Hacker News, is that jobs are decomposed into 16-node slices placed atomically on a single rack, with two spare nodes per rack held back for hot swaps.

The three moving parts:

  • Topology-Aware Scheduler (TAS). Keeps the workers of a job on one rack. For multi-rack workloads it splits the job into 16-node slices, which is configurable, and places each slice on its own rack with two spare nodes.
  • GPU Manager and Frontier Trainer. GPU Manager detects a failed node. Frontier Trainer then hot swaps the faulty node for one of the reserved healthy nodes in the same rack while the rest of the job keeps running. When a rack has no spare left, the slice moves to a healthy rack.
  • Automatic Defragmentation. Large clusters are shared, so jobs finish at different times and leave free nodes scattered. A cluster can have enough free GPUs for a rack-scale job while being unable to schedule it, because those nodes sit on different racks. Auto Defrag finds the least disruptive set of workloads to preempt in order to free a whole rack, then reschedules them elsewhere. It only touches work that restarts safely, meaning managed jobs that resume from a checkpoint, and it leaves interactive development environments alone unless an admin opts them in.

Why 16 nodes and not 18? Because 64 GPUs is a power of two, which matches how parallelism groups are usually sized, and because a smaller slice survives failures better. NVIDIA modeled usable capacity on a 128-rack, 2,304-node cluster as node unavailability grows:

Slice sizeGPUs per sliceUsable nodes on the 128-rack clusterWhat a node failure costs
18 nodes721,602The whole rack goes unschedulable
16 nodes642,037The first two failures are absorbed by spares
9 nodes36not statedHalf a rack, since two slices fill a rack
8 nodes32not statedHalf the GPUs per NVLink domain

At the roughly 2% node unavailability SkyPilot says it measures on production NVL72 clusters, 16-node slices yield 27% more usable capacity than 18-node slices, and they still beat 9-node slices by 4%.

The headline number comes from NVIDIA’s GR00T team, which ran pre-training for GR00T N1.7 on 512 GB300 GPUs across 128 nodes. The first run used a naive scheduler that spread nodes across an arbitrary number of racks. The second used a rack-aware slice policy that landed the job on exactly 8 racks. Throughput rose 14.4%, with each iteration dropping from 1.59 s to 1.39 s. On a 10,000-GPU cluster the same speedup is worth about $16.7K per training hour, or $400K per training day.

What to watch for

The open source and commercial halves are separate products and the split matters if you are deciding what to install. The Apache 2.0 repository at github.com/skypilot-org/skypilot contains the CLI, the task spec, the provider integrations, and what the README calls gang scheduling, multi-node jobs, and queueing on Kubernetes. It also has priority classes and preemption through Kubernetes’ native mechanism. It does not contain TAS, the GPU Manager, Frontier Trainer, or Auto Defrag. SkyPilot Platform is where those live, and it is built on top of Kueue.

That last point is the one a commenter raised, and it is the most interesting thing in the thread. Kueue’s topology-aware scheduling does not support Elastic jobs, which is exactly what makes the hot swap impossible for large NVL72 workloads in the general case. SkyPilot’s author confirmed the dependency and said more detailed posts about the scheduler internals are on the way.

Several other commenters made the same criticism from different angles: that rack-aware and gang scheduling are not new, that they date back to HTCondor and are standard at large GPU deployments, and that Kueue already documents topology-aware scheduling. One asked for the scheduler to be open sourced. Another noted that the study spends most of its length explaining the hardware before describing the scheduler in two paragraphs, which reads as a marketing piece rather than a technical one.

That critique has some force, and it is worth reading the thread alongside the study. What the study does well is quantify a real tradeoff that is usually discussed qualitatively: the choice between slice size, spare node overhead, and the probability that a rack goes dark.

Source and Accuracy Notes