xai-org/x-algorithm · critical · std::runtime_error

async_emb requires ${name}=${expected} before process startu

Error message

async_emb requires ${name}=${expected} before process startup

What it means

Thrown when an environment variable required by the async_emb library has a value different from the one mandated before process startup. The library pins environment settings (e.g. NCCL runtime knobs) that must be fixed before NCCL or the CUDA runtime initializes, so it validates them eagerly in AsyncEmbContext construction and refuses to continue if the operator overrode them.

Source

Thrown at phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc:60

  return (value + kAlign - 1) / kAlign * kAlign;
}

std::runtime_error ncclError(const char* operation, ncclResult_t result) {
  return std::runtime_error(
      std::string("NCCL ") + operation + " failed: " + ncclGetErrorString(result)
  );
}

void requireEnvironment(const char* name, const char* expected) {
  const char* value = std::getenv(name);
  if (value == nullptr || std::strcmp(value, expected) != 0) {
    throw std::runtime_error(
        "async_emb requires " + std::string(name) + "=" + expected + " before process startup"
    );
  }
}

std::chrono::seconds watchdogTimeout() {
  const char* value = std::getenv("XAI_ASYNC_EMB_TIMEOUT_SECONDS");
  if (value == nullptr) {
    return std::chrono::seconds(1800);
  }
  char* end = nullptr;
  long seconds = std::strtol(value, &end, 10);
  if (end == value || *end != '\0' || seconds <= 0) {
    throw std::invalid_argument("XAI_ASYNC_EMB_TIMEOUT_SECONDS must be a positive integer");
  }
  return std::chrono::seconds(seconds);
}

std::once_flag warmup_once;

void warmupKernels(int8_t* scratch, cudaStream_t stream) {
  auto* bf16 = reinterpret_cast<__nv_bfloat16*>(scratch);
  auto* f32 = reinterpret_cast<float*>(scratch);
  auto* i32 = reinterpret_cast<int32_t*>(scratch);

View on GitHub (pinned to 24c60942c5)

Solutions

1. Check the error text for the exact name=expected pair and export that exact value before launching the process (e.g. export NCCL_RUNTIME_CONNECT=0).
2. Search job scripts, Dockerfiles, and module files for the offending variable name and remove or correct the override.
3. Set the variable in the job spec / container env so all ranks see the same value.

Example fix

# before
export NCCL_RUNTIME_CONNECT=1
python train.py

# after
export NCCL_RUNTIME_CONNECT=0
python train.py

const std::pair<const char*, const char*> required[] = {
    {"NCCL_RUNTIME_CONNECT", "0"},
    {"NCCL_LAUNCH_ORDER_IMPLICIT", "1"}};
for (auto& [k, v] : required) {
    const char* cur = std::getenv(k);
    if (cur && std::string(cur) != v)
        throw std::runtime_error(std::string(k) + " must equal " + v);
}

When it happens

Trigger: Constructing AsyncEmbContext (which calls requireEnvironment via watchdogTimeout's call chain) when an env var such as NCCL_RUNTIME_CONNECT or NCCL_LAUNCH_ORDER_IMPLICIT is set to a value other than the required one (e.g. NCCL_RUNTIME_CONNECT=1).

Common situations: A site-wide module or container image exports NCCL tuning variables; a launcher script (torchrun, mpirun, srun) injects NCCL_* envs; a newer NCCL version changed the variable name so the required setting was not applied.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

• {ENV_CACHE_WARM_SAMPLE_PCT}: {error}
• {misconfiguration}
• API key is required: set EapiModelConfig.api_key or the XAI_
• async_emb: ${label} timed out waiting for step ${step}
• NCCL version query failed: ${ncclGetErrorString(result)}

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/997351cd51be5654. Report an issue: GitHub.