{"record":{"id":"5e8009ad1f5bc45e","repo":"xai-org/x-algorithm","slug":"nccl-operation-failed-ncclgeterrorstring-res","errorCode":null,"errorMessage":"NCCL ${operation} failed: ${ncclGetErrorString(result)}","messagePattern":"NCCL (.+?) failed: (.+?)","errorType":"exception","errorClass":"std::runtime_error","httpStatus":null,"severity":"critical","filePath":"phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc","lineNumber":54,"sourceCode":"}\n\nsize_t alignUp(size_t value) {\n  if (value > std::numeric_limits<size_t>::max() - (kAlign - 1)) {\n    throw std::overflow_error(\"async_emb arena alignment overflow\");\n  }\n  return (value + kAlign - 1) / kAlign * kAlign;\n}\n\nstd::runtime_error ncclError(const char* operation, ncclResult_t result) {\n  return std::runtime_error(\n      std::string(\"NCCL \") + operation + \" failed: \" + ncclGetErrorString(result)\n  );\n}\n\nvoid requireEnvironment(const char* name, const char* expected) {\n  const char* value = std::getenv(name);\n  if (value == nullptr || std::strcmp(value, expected) != 0) {\n    throw std::runtime_error(\n        \"async_emb requires \" + std::string(name) + \"=\" + expected + \" before process startup\"\n    );\n  }\n}\n\nstd::chrono::seconds watchdogTimeout() {\n  const char* value = std::getenv(\"XAI_ASYNC_EMB_TIMEOUT_SECONDS\");\n  if (value == nullptr) {\n    return std::chrono::seconds(1800);\n  }\n  char* end = nullptr;\n  long seconds = std::strtol(value, &end, 10);\n  if (end == value || *end != '\\0' || seconds <= 0) {\n    throw std::invalid_argument(\"XAI_ASYNC_EMB_TIMEOUT_SECONDS must be a positive integer\");\n  }\n  return std::chrono::seconds(seconds);\n}\n","sourceCodeStart":36,"sourceCodeEnd":72,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc#L36-L72","documentation":"All NCCL calls in async_emb_comm are wrapped by ncclError(operation, result), which produces std::runtime_error('NCCL <operation> failed: <ncclGetErrorString(result)>') whenever a call returns non-success. The message names the failing operation and NCCL's own error string (e.g. invalid usage, remote peer gone, system error).","triggerScenarios":"Any ncclCommInitRank/ncclGroupEnd/ncclAllReduce/etc. call failing during AsyncEmbContext::handshake or later collectives — e.g. mismatched ranks/ nranks across processes, duplicated MVAPICH/NCCL env, CUDA context errors, or a peer rank dying mid-collective.","commonSituations":"World-size mismatch between ranks (one job started with different nproc); NCCL_SOCKET_IFNAME/NCCL_IB settings broken on the cluster; driver or fabric faults; incompatible NCCL/CUDA versions (often preceded by nccl version checks); OOM-killed peer causing NCCL_ERR_REMOTE_REQUEST or lost connection.","solutions":["Read the ncclGetErrorString portion: NCCL_ERR_INVALID_USAGE points to config (ranks/rings), NCCL_ERR_SYSTEM to driver/fabric/sockets","Verify identical world size, rank assignment, and NCCL env vars across all ranks before launch","Set NCCL_DEBUG=INFO (or WARN) and rerun to get the exact failing call and transport","Confirm CUDA driver compatibility and that nccl_version() matches the version the extension was built against; fix library path if mismatched","If a peer died (OOM/preemption), address that root cause and restart the job"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"# preflight before launching ranks\nassert int(os.environ[\"WORLD_SIZE\"]) == expected_ranks\nassert \"NCCL_SOCKET_IFNAME\" not in os.environ or os.environ[\"NCCL_SOCKET_IFNAME\"]","typeGuard":null,"tryCatchPattern":"try:\n    ctx = async_emb.create_context(spec)  # performs NCCL handshake\nexcept RuntimeError as e:\n    msg = str(e)\n    if \"NCCL\" in msg and \"failed\" in msg:\n        log.error(\"nccl op failed: %s\", msg)\n        raise SystemExit(2)  # restart job after fixing env; blind retry rarely helps\n    raise","preventionTips":["Launch all ranks with identical world size and NCCL env","Set NCCL_DEBUG=INFO during bring-up","Keep NCCL and CUDA driver versions aligned with build","Monitor for OOM-killed peers; a dead rank stalls all collectives"],"tags":["async-emb","cpp","nccl","distributed-training","collective-op"],"backgroundTag":"nccl-collective-failure","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}