[HN Gopher] Show HN: Autonomous recovery for distributed trainin...
___________________________________________________________________
Show HN: Autonomous recovery for distributed training jobs
Hi HN! We're TensorPool. We help companies access and optimize
large scale compute for training foundation models. The Problem
It's been almost a year since we've finished YC, and we've just
crossed 100,000 multinode training GPU hours run on our platform.
On those training runs, we've seen countless 3am job crashes
because of issues like an Xid error from a flaky GPU or an S3
timeout that corrupted a checkpoint save. By the time you wake up
and notice, you've lost 8+ hours of compute. You scramble to
diagnose the issue, manually restart from the last checkpoint, and
hope it doesn't happen again. Rinse and repeat. For training runs
that take days to weeks, this constant babysitting is exhausting
and expensive. The research iteration cycles lost can also make or
break a model release (especially for short reservations). What We
Built This agent monitors your training jobs and autonomously
recovers them when things go wrong. It works with Kubernetes,
Slurm, and TensorPool Jobs. We originally built the TensorPool
Agent as an internal tool to help us debug failures with our own
customers. Over time, we realized its performance was so good that
we could automate the entire triage process. We're now releasing a
public beta for people to use. Best case: The TensorPool Agent
detects the failure, diagnoses the root cause, fixes it, and
restarts your job from the last checkpoint - all while you sleep ;)
Worst case: If the TensorPool agent can't fix the issue
automatically, it delivers a preliminary RCA and a list of actions
it attempted, giving you a head start on debugging. How It Works
1) Registration - You provide credentials to your job scheduler via
our dashboard. Perms are granted on a whitelist basis; you
explicitly control what actions the agent can take. 2) Monitoring
- The agent continuously monitors your job for failure conditions.
3) Recovery - On failure, the agent analyzes logs and attempts to
diagnose the issue. If successful, it restarts the job from the
last checkpoint and resumes monitoring. If not, you get an alert
with full context. Target Failure Modes The agent is specifically
designed for runtime errors that occur deep into training, like: -
CUDA OOM: Memory leaks, gradient explosions - Xid errors: GPU
hardware faults (Xid 79, 63, 48, etc.) - Distributed communication
failures: NCCL timeouts, rank failures - Storage I/O errors:
Checkpoint corruption - Network issues: S3 request timeouts on
mounted object storage
Author : tsvoboda
Score : 7 points
Date : 2026-01-29 17:01 UTC (5 hours ago)
(HTM) web link (docs.tensorpool.dev)
(TXT) w3m dump (docs.tensorpool.dev)
| tsvoboda wrote:
| Would love to hear how you're handling recovery for long-running
| training jobs today, as well as what failure modes are most
| common/annoying for you.
| hnotshe wrote:
| We're still figuring out how to detect "silent" failures where
| the job doesn't crash but stops making progress -- like NCCL
| hangs where ranks are waiting indefinitely, or gradient norm
| explosions that don't trigger OOM but tank loss. Right now we
| rely on explicit errors in logs, but curious how others approach
| detecting "the job is technically running but something is very
| wrong" (if at all)?
| jpollock wrote:
| Measurement and alerting is usually done in business metrics,
| not the causes. That way you catch classes of problems.
|
| Not sure about expected loss, that's a decay rate?
|
| But stuck jobs are via tasks being processed and average
| latency.
___________________________________________________________________
(page generated 2026-01-29 23:01 UTC)