xai-org/x-algorithm · error · std::invalid_argument

async_emb received an invalid NCCL bootstrap payload

Error message

async_emb received an invalid NCCL bootstrap payload

What it means

handshake() consumes the bootstrap payload gathered from all ranks and validates it has exactly world_size_ entries whose first entry is sizeof(ncclUniqueId) bytes. A malformed payload means the gather/broadcast of rank 0's unique ID went wrong between reset() and handshake().

Source

Thrown at phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc:264

  if (rank != 0) {
    return bootstrap;
  }

  ncclUniqueId id;
  ncclResult_t result = ncclGetUniqueId(&id);
  if (result != ncclSuccess) {
    throw ncclError("ncclGetUniqueId", result);
  }

  std::vector<uint8_t> bytes(sizeof(id));
  std::memcpy(bytes.data(), &id, sizeof(id));
  std::fill(bootstrap.begin(), bootstrap.end(), bytes);
  return bootstrap;
}

void AsyncEmbContext::handshake(std::vector<std::vector<uint8_t>> data) {
  if (data.size() != size_t(world_size_) || data[0].size() != sizeof(ncclUniqueId)) {
    throw std::invalid_argument("async_emb received an invalid NCCL bootstrap payload");
  }

  requireEnvironment("NCCL_RUNTIME_CONNECT", "0");
  requireEnvironment("NCCL_LAUNCH_ORDER_IMPLICIT", "1");

  ncclUniqueId id;
  std::memcpy(&id, data[0].data(), sizeof(id));
  ncclConfig_t config = NCCL_CONFIG_INITIALIZER;
  config.blocking = 0;
  config.minCTAs = kNcclMinCtas;
  config.maxCTAs = kNcclMaxCtas;
  ncclResult_t result = ncclCommInitRankConfig(&comm_, world_size_, id, rank_, &config);
  if (result != ncclSuccess && result != ncclInProgress) {
    throw ncclError("ncclCommInitRankConfig", result);
  }
  waitNcclReady("communicator initialization");

  ncclResult_t register_result =

View on GitHub (pinned to 24c60942c5)

Solutions

1. Verify the object passed to handshake is exactly the all_gather_object result of rank 0's reset() return, unmodified.
2. Ensure every rank used identical world_size in both reset and the gather.
3. Transport bootstrap bytes as base64 or raw buffers, never as C strings, to avoid truncation at embedded NULs.

Example fix

# before (truncated via string decode)
data = [bytes(v, 'utf-8') for v in store.get('bootstrap')]
ctx.handshake(data)

# after
import base64
data = [base64.b64decode(v) for v in store.get('bootstrap')]
ctx.handshake(data)

if (data.size() != size_t(world_size) || data[0].size() != sizeof(ncclUniqueId))
    throw std::invalid_argument("bad bootstrap payload");

When it happens

Trigger: Calling handshake(data) where data.size() != world_size or data[0].size() != sizeof(ncclUniqueId) — e.g. some ranks returned a truncated vector, ranks disagree on world size, or the caller passed the wrong object (per-rank slices shuffled or deduped).

Common situations: A custom rendezvous (Redis/store) dropping or reordering entries; mixing ranks from two jobs in the same store key; serializing bootstrap bytes through a transport that truncates (JSON strings with embedded NULs).

Related errors

• async_emb received an invalid rank or world size
• async_emb: ${label} timed out waiting for step ${step}
• NCCL version query failed: ${ncclGetErrorString(result)}
• could not resolve loaded NCCL library path
• NCCL ${operation} failed: ${ncclGetErrorString(result)}

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/74b9703eb597c7ff. Report an issue: GitHub.