rohitg00/ai-engineering-from-scratch · error · RuntimeError

ranks did not finish in time

Error message

ranks did not finish in time

What it means

Error "ranks did not finish in time" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/48-distributed-fsdp-ddp/code/main.py:350

    max_diff = 0.0
    deadline = time.time() + timeout
    collected = 0
    while collected < world_size and time.time() < deadline:
        try:
            rank, payload, diff = result_queue.get(timeout=1.0)
        except Exception:
            continue
        results[rank] = payload
        if diff > max_diff:
            max_diff = diff
        collected += 1
    for p in procs:
        p.join(timeout=max(1.0, deadline - time.time()))
    if any(p.is_alive() for p in procs):
        for p in procs:
            if p.is_alive():
                p.terminate()
        raise RuntimeError("ranks did not finish in time")
    if collected < world_size:
        raise RuntimeError(f"only got {collected}/{world_size} results: {results}")
    for rank, payload in results.items():
        if "error" in payload:
            raise RuntimeError(f"rank {rank} failed: {payload['error']}")
    param_sums = {r: results[r]["post_param_sum"] for r in results}
    spread = max(param_sums.values()) - min(param_sums.values())
    losses = [results[r]["final_loss"] for r in results]
    grad_norm = results[0]["grad_norm_after_all_reduce"]
    return {
        "world_size": world_size,
        "backend": backend,
        "param_sum_per_rank": param_sums,
        "param_sum_spread": spread,
        "losses": losses,
        "grad_norm_after_all_reduce": grad_norm,
        "manual_all_reduce_max_diff_vs_single_process": max_diff,
        "fsdp_round_trip_all_ranks_ok": all(results[r]["fsdp_round_trip_ok"] for r in results),

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/48-distributed-fsdp-ddp/code/main.py:350 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/791fc1a62e0f8d66. Report an issue: GitHub.