{"record":{"id":"997351cd51be5654","repo":"xai-org/x-algorithm","slug":"async-emb-requires-name-expected-before-proc","errorCode":null,"errorMessage":"async_emb requires ${name}=${expected} before process startup","messagePattern":"async_emb requires (.+?)=(.+?) before process startup","errorType":"exception","errorClass":"std::runtime_error","httpStatus":null,"severity":"critical","filePath":"phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc","lineNumber":60,"sourceCode":"  return (value + kAlign - 1) / kAlign * kAlign;\n}\n\nstd::runtime_error ncclError(const char* operation, ncclResult_t result) {\n  return std::runtime_error(\n      std::string(\"NCCL \") + operation + \" failed: \" + ncclGetErrorString(result)\n  );\n}\n\nvoid requireEnvironment(const char* name, const char* expected) {\n  const char* value = std::getenv(name);\n  if (value == nullptr || std::strcmp(value, expected) != 0) {\n    throw std::runtime_error(\n        \"async_emb requires \" + std::string(name) + \"=\" + expected + \" before process startup\"\n    );\n  }\n}\n\nstd::chrono::seconds watchdogTimeout() {\n  const char* value = std::getenv(\"XAI_ASYNC_EMB_TIMEOUT_SECONDS\");\n  if (value == nullptr) {\n    return std::chrono::seconds(1800);\n  }\n  char* end = nullptr;\n  long seconds = std::strtol(value, &end, 10);\n  if (end == value || *end != '\\0' || seconds <= 0) {\n    throw std::invalid_argument(\"XAI_ASYNC_EMB_TIMEOUT_SECONDS must be a positive integer\");\n  }\n  return std::chrono::seconds(seconds);\n}\n\nstd::once_flag warmup_once;\n\nvoid warmupKernels(int8_t* scratch, cudaStream_t stream) {\n  auto* bf16 = reinterpret_cast<__nv_bfloat16*>(scratch);\n  auto* f32 = reinterpret_cast<float*>(scratch);\n  auto* i32 = reinterpret_cast<int32_t*>(scratch);","sourceCodeStart":42,"sourceCodeEnd":78,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc#L42-L78","documentation":"Thrown when an environment variable required by the async_emb library has a value different from the one mandated before process startup. The library pins environment settings (e.g. NCCL runtime knobs) that must be fixed before NCCL or the CUDA runtime initializes, so it validates them eagerly in AsyncEmbContext construction and refuses to continue if the operator overrode them.","triggerScenarios":"Constructing AsyncEmbContext (which calls requireEnvironment via watchdogTimeout's call chain) when an env var such as NCCL_RUNTIME_CONNECT or NCCL_LAUNCH_ORDER_IMPLICIT is set to a value other than the required one (e.g. NCCL_RUNTIME_CONNECT=1).","commonSituations":"A site-wide module or container image exports NCCL tuning variables; a launcher script (torchrun, mpirun, srun) injects NCCL_* envs; a newer NCCL version changed the variable name so the required setting was not applied.","solutions":["Check the error text for the exact name=expected pair and export that exact value before launching the process (e.g. export NCCL_RUNTIME_CONNECT=0).","Search job scripts, Dockerfiles, and module files for the offending variable name and remove or correct the override.","Set the variable in the job spec / container env so all ranks see the same value."],"exampleFix":"# before\nexport NCCL_RUNTIME_CONNECT=1\npython train.py\n\n# after\nexport NCCL_RUNTIME_CONNECT=0\npython train.py","handlingStrategy":"validation","validationCode":"const std::pair<const char*, const char*> required[] = {\n    {\"NCCL_RUNTIME_CONNECT\", \"0\"},\n    {\"NCCL_LAUNCH_ORDER_IMPLICIT\", \"1\"}};\nfor (auto& [k, v] : required) {\n    const char* cur = std::getenv(k);\n    if (cur && std::string(cur) != v)\n        throw std::runtime_error(std::string(k) + \" must equal \" + v);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set required env vars in the container image / job spec, not ad-hoc shells.","Add a startup assertion helper that verifies all required env vars before constructing the context."],"tags":["env-var","configuration","startup","nccl","async-emb"],"backgroundTag":"missing-env-var","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}