{"record":{"id":"74b9703eb597c7ff","repo":"xai-org/x-algorithm","slug":"async-emb-received-an-invalid-nccl-bootstrap-paylo","errorCode":null,"errorMessage":"async_emb received an invalid NCCL bootstrap payload","messagePattern":"async_emb received an invalid NCCL bootstrap payload","errorType":"exception","errorClass":"std::invalid_argument","httpStatus":null,"severity":"error","filePath":"phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc","lineNumber":264,"sourceCode":"  if (rank != 0) {\n    return bootstrap;\n  }\n\n  ncclUniqueId id;\n  ncclResult_t result = ncclGetUniqueId(&id);\n  if (result != ncclSuccess) {\n    throw ncclError(\"ncclGetUniqueId\", result);\n  }\n\n  std::vector<uint8_t> bytes(sizeof(id));\n  std::memcpy(bytes.data(), &id, sizeof(id));\n  std::fill(bootstrap.begin(), bootstrap.end(), bytes);\n  return bootstrap;\n}\n\nvoid AsyncEmbContext::handshake(std::vector<std::vector<uint8_t>> data) {\n  if (data.size() != size_t(world_size_) || data[0].size() != sizeof(ncclUniqueId)) {\n    throw std::invalid_argument(\"async_emb received an invalid NCCL bootstrap payload\");\n  }\n\n  requireEnvironment(\"NCCL_RUNTIME_CONNECT\", \"0\");\n  requireEnvironment(\"NCCL_LAUNCH_ORDER_IMPLICIT\", \"1\");\n\n  ncclUniqueId id;\n  std::memcpy(&id, data[0].data(), sizeof(id));\n  ncclConfig_t config = NCCL_CONFIG_INITIALIZER;\n  config.blocking = 0;\n  config.minCTAs = kNcclMinCtas;\n  config.maxCTAs = kNcclMaxCtas;\n  ncclResult_t result = ncclCommInitRankConfig(&comm_, world_size_, id, rank_, &config);\n  if (result != ncclSuccess && result != ncclInProgress) {\n    throw ncclError(\"ncclCommInitRankConfig\", result);\n  }\n  waitNcclReady(\"communicator initialization\");\n\n  ncclResult_t register_result =","sourceCodeStart":246,"sourceCodeEnd":282,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc#L246-L282","documentation":"handshake() consumes the bootstrap payload gathered from all ranks and validates it has exactly world_size_ entries whose first entry is sizeof(ncclUniqueId) bytes. A malformed payload means the gather/broadcast of rank 0's unique ID went wrong between reset() and handshake().","triggerScenarios":"Calling handshake(data) where data.size() != world_size or data[0].size() != sizeof(ncclUniqueId) — e.g. some ranks returned a truncated vector, ranks disagree on world size, or the caller passed the wrong object (per-rank slices shuffled or deduped).","commonSituations":"A custom rendezvous (Redis/store) dropping or reordering entries; mixing ranks from two jobs in the same store key; serializing bootstrap bytes through a transport that truncates (JSON strings with embedded NULs).","solutions":["Verify the object passed to handshake is exactly the all_gather_object result of rank 0's reset() return, unmodified.","Ensure every rank used identical world_size in both reset and the gather.","Transport bootstrap bytes as base64 or raw buffers, never as C strings, to avoid truncation at embedded NULs."],"exampleFix":"# before (truncated via string decode)\ndata = [bytes(v, 'utf-8') for v in store.get('bootstrap')]\nctx.handshake(data)\n\n# after\nimport base64\ndata = [base64.b64decode(v) for v in store.get('bootstrap')]\nctx.handshake(data)","handlingStrategy":"validation","validationCode":"if (data.size() != size_t(world_size) || data[0].size() != sizeof(ncclUniqueId))\n    throw std::invalid_argument(\"bad bootstrap payload\");","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pass bootstrap bytes through the gather unmodified; base64-encode if crossing text transports.","Use unique store keys per job so ranks never mix."],"tags":["nccl","bootstrap","validation","distributed","async-emb"],"backgroundTag":"payload-validation-failed","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}