{"record":{"id":"2700d8ff5a72e2c3","repo":"xai-org/x-algorithm","slug":"async-emb-label-timed-out-waiting-for-step-s","errorCode":null,"errorMessage":"async_emb: ${label} timed out waiting for step ${step}","messagePattern":"async_emb: (.+?) timed out waiting for step (.+?)","errorType":"exception","errorClass":"std::runtime_error","httpStatus":null,"severity":"critical","filePath":"phoenix/xrex/cuda/async_emb/src/async_emb_api.cc","lineNumber":728,"sourceCode":"  static_assert(\n      std::is_invocable_r_v<XLA_FFI_Error*, T, XLA_FFI_CallFrame*>,\n      \"Encapsulated function must be an XLA FFI handler\"\n  );\n  return nb::capsule(reinterpret_cast<void*>(fn));\n}\n\nuint64_t waitLatest(int64_t context_id, AsyncEmbContext::Operation pipeline, const char* label) {\n  auto ctx = findContext(context_id);\n  if (ctx == nullptr) {\n    return 0;\n  }\n  ctx->ensureHealthy();\n  uint64_t step = ctx->armedStep(pipeline);\n  if (step == 0) {\n    return 0;\n  }\n  if (!ctx->hostWaitDone(pipeline, step)) {\n    throw std::runtime_error(\n        \"async_emb: \" + std::string(label) + \" timed out waiting for step \" + std::to_string(step)\n    );\n  }\n  return step;\n}\n\nnb::bytes testSnapshot(int64_t context_id, const std::string& region) {\n  auto ctx = readyContext(context_id);\n  if (ctx == nullptr) {\n    throw std::invalid_argument(\"async_emb context not initialized\");\n  }\n  const auto& spec = ctx->spec();\n  const auto& layout = ctx->layout();\n  const size_t index_bytes =\n      size_t(ctx->worldSize()) * size_t(spec.tokens_per_rank) * sizeof(int32_t);\n  const size_t block_bytes = size_t(ctx->worldSize()) * size_t(spec.tokens_per_rank) *\n                             size_t(spec.shard_width) * sizeof(__nv_bfloat16);\n  using Operation = AsyncEmbContext::Operation;","sourceCodeStart":710,"sourceCodeEnd":746,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/cuda/async_emb/src/async_emb_api.cc#L710-L746","documentation":"async_emb's waitLatest blocks until the host pipeline reports completion of an armed step. If hostWaitDone returns false within the deadline, a std::runtime_error is thrown with the pipeline label and step number, meaning the embedding pipeline stalled or the CUDA/NCCL copy never signaled completion.","triggerScenarios":"Calling testSnapshot (which internally calls waitLatest) after the pipeline has armed a step (armedStep != 0) but the host wait times out — e.g. stalled NCCL communicator, dead peer rank, GPU hang, or a snapshot requested while training threads are blocked.","commonSituations":"One rank crashed or was preempted so collective copies never complete; NCCL timeout/ watchdog misconfiguration; calling testSnapshot during checkpointing or while the pipeline is deliberately paused; driver-level GPU reset.","solutions":["Check GPU health and dmesg/nvidia-smi for Xid errors or hangs; restart the job if a rank died","Verify all ranks in the world are alive and participating (a single dead peer stalls collectives)","Retry testSnapshot when the pipeline is quiescent rather than mid-step","If reproducible, capture NCCL_DEBUG=INFO logs to identify which collective is stuck"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"# Ensure the pipeline is healthy and a step is armed before snapshotting\nstep = async_emb.armed_step(ctx_id) if hasattr(async_emb, 'armed_step') else None\nif step == 0:\n    pass  # nothing in flight; waitLatest returns 0 without blocking","typeGuard":null,"tryCatchPattern":"try:\n    snap = async_emb._test_snapshot(ctx_id, region)\nexcept RuntimeError as e:\n    if \"timed out waiting for step\" in str(e):\n        log.error(\"pipeline stalled at step, aborting job\")\n        raise SystemExit(3)  # do not retry a hung collective blindly\n    raise","preventionTips":["Monitor peer-rank liveness and NCCL watchdog logs","Snapshot only when the pipeline is quiescent","Abort the job on collective timeouts rather than retrying"],"tags":["async-emb","cpp","cuda","nccl","timeout","distributed-training"],"backgroundTag":"nccl-collective-timeout","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}