From Checkpoint Roulette to Continuous Training
AI infrastructure downtime in large-scale training refers to the wasted time, GPU resources, and operational effort caused when GPU training failures interrupt long-running jobs, forcing checkpoint restarts, recompute, and manual intervention instead of continuous progress toward a finished model. The uncomfortable truth is that the traditional checkpoint-restart model is now an economic liability. On a large enough GPU cluster, something is always breaking, and at modern scales failures during a run are expected, not exceptional. Rolling back to the last checkpoint—often one or two hours of work—means paying twice for the same gradient steps and accepting unpredictable delivery times. AI teams that keep doing this are not being cautious; they are lighting their GPU budget on fire.
Two emerging strategies are changing the equation. Live migration tools such as Clockwork’s TorchPass move training jobs to healthy GPUs mid-flight, while distributed GPU reliability systems like Databricks’ multi-stage health checks detect and quarantine bad hardware before or during training. Together, they point to a new default: you only compute once, and the infrastructure quietly heals around your job.
Why Checkpoints Alone Are No Longer Enough
The industry has treated checkpoint restart optimization as the primary answer to GPU training failures: save state frequently, tolerate crashes, roll back, and recompute. In practice, this strategy collapses at scale. Distributed training jobs crash for many reasons, from GPUs falling off the bus to RDMA fabric issues and I/O hangs, and each crash forces a rollback to the last checkpoint, which can be an hour or two old. None of that recomputed work is free; teams pay for expensive GPUs to redo work they already did.
This would be tolerable if failures were rare. They are not. A 256‑GPU job running for 30 days has about a 19% chance of seeing a failure, and at 1,024 GPUs, that climbs to 57%. Research cited in Clockwork’s announcement puts the mean time to failure on a 1,024‑GPU cluster at 7.9 hours, dropping to 1.8 hours at 16,384 GPUs. That failure rate turns checkpointing from a safety net into a treadmill. According to one analysis, failure-driven restarts drain more than USD 6 million (approx. RM27.6 million) a year in wasted compute on a typical 2,048‑GPU H200 deployment.
TorchPass and the Rise of Live GPU Failover
Clockwork’s TorchPass is an explicit rejection of the “restart and pray” era. TorchPass, which reached general availability in March, prevents many GPU training failures from stopping the run by moving the job’s in‑memory state—including model weights, gradients, and optimizer state—to a healthy spare GPU or a node running a lower‑priority job, typically recovering in a few minutes. Instead of rolling back to a checkpoint, the job resumes at the next step. The company’s YOCO (“You Only Compute Once”) Guarantee, launched on a Wednesday, promises that 90% of failures on a supported training run will be resolved with no lost progress, no checkpoint rollback, and no recompute.
This is more than a clever feature; it is a bet that failover automation can rewrite AI infrastructure economics. TorchPass can draw replacement GPUs from dedicated standby nodes or from low‑priority jobs that it preempts. It can also act before a crash by migrating jobs when health signals—such as rising temperatures—hint that a GPU will fail, while it still has access to GPU memory. The message is blunt: AI teams need their models done, not their nodes up, and infrastructure should adapt in real time to make that happen.
Databricks and the New Discipline of Distributed GPU Reliability
Live migration alone does not solve the messier side of distributed GPU reliability: silent slowdowns and numerical corruption. At Databricks AI, training workloads run at massive scale every week, and failures appear continuously across hardware, fabric, and software. Most GPU failures fall into three categories: crashed jobs, silent slowdowns, and numerical corruption. A silently degraded GPU can keep training with normal‑looking logs and falling loss, but throughput is bottlenecked on the slowest GPU, wasting compute and money. Numerical corruption, even with ECC, can slip through and show up later as NaN losses or subtle quality drops.
To keep clusters reliable despite this, Databricks built gpu‑monitor, a multi‑stage health check and observability service that runs on every GPU node across its lifecycle. Together, its layers verify hardware before workloads start, watch it while they run, and validate the broader fabric in between. Nodes that fail continuous checks are cordoned, drained, and quarantined, so bad hardware is routed around before it can poison a training run. This is what distributed GPU reliability now demands: not just reacting to crashes, but continuously scoring the health of thousands of GPUs and links so large training clusters can stay online.
Less Ops, More Models: What Comes Next
The direction of travel is clear: manual restarts and blind checkpointing are being replaced by automated failover and health‑aware scheduling. On one side, TorchPass keeps jobs running across hardware failures and is backed by the YOCO Guarantee; if Clockwork falls short of its 90% target in a contract year, customers receive a 25% credit toward their next renewal or expansion. Clockwork is also piloting its capabilities with large cloud operators, signaling that live GPU migration will not remain a niche add‑on.
On the other side, Databricks’ ongoing investigations feed into its health check system, tightening the feedback loop between failure modes and detection logic. AI teams that keep treating GPU reliability as someone else’s problem will be outspent and outpaced. The future belongs to workflows where checkpoint restart optimization is a fallback, not a daily event; where AI infrastructure downtime is handled by automation, not sleep‑deprived engineers; and where expensive GPUs are spent on new gradients, not on replaying the same hours of work twice.
