xai-org/x-algorithm · critical · std::runtime_error

NCCL ${operation} failed: ${ncclGetErrorString(result)}

Error message

NCCL ${operation} failed: ${ncclGetErrorString(result)}

What it means

All NCCL calls in async_emb_comm are wrapped by ncclError(operation, result), which produces std::runtime_error('NCCL <operation> failed: <ncclGetErrorString(result)>') whenever a call returns non-success. The message names the failing operation and NCCL's own error string (e.g. invalid usage, remote peer gone, system error).

Source

Thrown at phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc:54

}

size_t alignUp(size_t value) {
  if (value > std::numeric_limits<size_t>::max() - (kAlign - 1)) {
    throw std::overflow_error("async_emb arena alignment overflow");
  }
  return (value + kAlign - 1) / kAlign * kAlign;
}

std::runtime_error ncclError(const char* operation, ncclResult_t result) {
  return std::runtime_error(
      std::string("NCCL ") + operation + " failed: " + ncclGetErrorString(result)
  );
}

void requireEnvironment(const char* name, const char* expected) {
  const char* value = std::getenv(name);
  if (value == nullptr || std::strcmp(value, expected) != 0) {
    throw std::runtime_error(
        "async_emb requires " + std::string(name) + "=" + expected + " before process startup"
    );
  }
}

std::chrono::seconds watchdogTimeout() {
  const char* value = std::getenv("XAI_ASYNC_EMB_TIMEOUT_SECONDS");
  if (value == nullptr) {
    return std::chrono::seconds(1800);
  }
  char* end = nullptr;
  long seconds = std::strtol(value, &end, 10);
  if (end == value || *end != '\0' || seconds <= 0) {
    throw std::invalid_argument("XAI_ASYNC_EMB_TIMEOUT_SECONDS must be a positive integer");
  }
  return std::chrono::seconds(seconds);
}

View on GitHub (pinned to 24c60942c5)

Solutions

1. Read the ncclGetErrorString portion: NCCL_ERR_INVALID_USAGE points to config (ranks/rings), NCCL_ERR_SYSTEM to driver/fabric/sockets
2. Verify identical world size, rank assignment, and NCCL env vars across all ranks before launch
3. Set NCCL_DEBUG=INFO (or WARN) and rerun to get the exact failing call and transport
4. Confirm CUDA driver compatibility and that nccl_version() matches the version the extension was built against; fix library path if mismatched
5. If a peer died (OOM/preemption), address that root cause and restart the job

# preflight before launching ranks
assert int(os.environ["WORLD_SIZE"]) == expected_ranks
assert "NCCL_SOCKET_IFNAME" not in os.environ or os.environ["NCCL_SOCKET_IFNAME"]

try:
    ctx = async_emb.create_context(spec)  # performs NCCL handshake
except RuntimeError as e:
    msg = str(e)
    if "NCCL" in msg and "failed" in msg:
        log.error("nccl op failed: %s", msg)
        raise SystemExit(2)  # restart job after fixing env; blind retry rarely helps
    raise

When it happens

Trigger: Any ncclCommInitRank/ncclGroupEnd/ncclAllReduce/etc. call failing during AsyncEmbContext::handshake or later collectives — e.g. mismatched ranks/ nranks across processes, duplicated MVAPICH/NCCL env, CUDA context errors, or a peer rank dying mid-collective.

Common situations: World-size mismatch between ranks (one job started with different nproc); NCCL_SOCKET_IFNAME/NCCL_IB settings broken on the cluster; driver or fabric faults; incompatible NCCL/CUDA versions (often preceded by nccl version checks); OOM-killed peer causing NCCL_ERR_REMOTE_REQUEST or lost connection.

Related errors

• async_emb: ${label} timed out waiting for step ${step}
• NCCL version query failed: ${ncclGetErrorString(result)}
• could not resolve loaded NCCL library path
• async_emb context not initialized
• unknown async_emb snapshot region: ${region}

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/5e8009ad1f5bc45e. Report an issue: GitHub.