xai-org/x-algorithm · error · std::invalid_argument

async_emb context not initialized

Error message

async_emb context not initialized

What it means

async_emb resolves contexts by numeric id via readyContext. testSnapshot throws std::invalid_argument when the context id does not map to an initialized context (null result), i.e. the context was never created or has already been destroyed.

Source

Thrown at phoenix/xrex/cuda/async_emb/src/async_emb_api.cc:738

    return 0;
  }
  ctx->ensureHealthy();
  uint64_t step = ctx->armedStep(pipeline);
  if (step == 0) {
    return 0;
  }
  if (!ctx->hostWaitDone(pipeline, step)) {
    throw std::runtime_error(
        "async_emb: " + std::string(label) + " timed out waiting for step " + std::to_string(step)
    );
  }
  return step;
}

nb::bytes testSnapshot(int64_t context_id, const std::string& region) {
  auto ctx = readyContext(context_id);
  if (ctx == nullptr) {
    throw std::invalid_argument("async_emb context not initialized");
  }
  const auto& spec = ctx->spec();
  const auto& layout = ctx->layout();
  const size_t index_bytes =
      size_t(ctx->worldSize()) * size_t(spec.tokens_per_rank) * sizeof(int32_t);
  const size_t block_bytes = size_t(ctx->worldSize()) * size_t(spec.tokens_per_rank) *
                             size_t(spec.shard_width) * sizeof(__nv_bfloat16);
  using Operation = AsyncEmbContext::Operation;
  struct Region {
    Operation operation;
    size_t offset;
    size_t bytes;
  };
  const std::unordered_map<std::string, Region> regions = {
      {"lookup_ids", {Operation::Lookup, layout.token_ids_all, index_bytes}},
      {"lookup_embeddings", {Operation::Lookup, layout.lookup_recv, block_bytes}},
      {"lookup_send", {Operation::Lookup, layout.lookup_send, block_bytes}},
      {"update_indices", {Operation::Update, layout.segment_ids_all, index_bytes}},

View on GitHub (pinned to 24c60942c5)

Solutions

1. Create/initialize the async_emb context first and use the id it returns
2. Verify the training job / context that owns the id is still alive when snapshotting
3. Use the module's context-enumeration or lifecycle API (if available) to confirm the id before calling _test_snapshot

Example fix

# before
snap = async_emb._test_snapshot(ctx_id, "grads")  # ctx never created

# after
ctx_id = async_emb.create_context(spec, ...)   # initialize first
snap = async_emb._test_snapshot(ctx_id, "grads")

ctx_id = create_context(spec)         # must succeed first
assert ctx_id is not None and ctx_id > 0
# only then snapshot
snap = async_emb._test_snapshot(ctx_id, "grads")

def contextReady(ctx_id: int) -> bool:
    return ctx_id > 0 and async_emb.has_context(ctx_id) if hasattr(async_emb, 'has_context') else ctx_id > 0

try:
    snap = async_emb._test_snapshot(ctx_id, region)
except ValueError as e:
    if "not initialized" in str(e):
        ctx_id = create_context(spec)
        snap = async_emb._test_snapshot(ctx_id, region)
    else:
        raise

When it happens

Trigger: Calling _test_snapshot(context_id, region) with a context_id that was never created via the module's init/creation API, or one whose context was torn down (training finished, context destroyed, or created in another process).

Common situations: Testing snapshot after training completed and the context was freed; passing 0 or a stale id; calling from a different process than the one that owns the context; ordering bug where the test runs before context creation.

Related errors

• unknown async_emb snapshot region: ${region}
• async_emb: ${label} timed out waiting for step ${step}
• NCCL version query failed: ${ncclGetErrorString(result)}
• could not resolve loaded NCCL library path
• async_emb arena size overflow

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/cff64e0673f73b30. Report an issue: GitHub.