xai-org/x-algorithm · error · std::invalid_argument

unknown async_emb snapshot region: ${region}

Error message

unknown async_emb snapshot region: ${region}

What it means

testSnapshot accepts a region name and looks it up in a fixed map of snapshot regions (e.g. tokens/embeddings/Update grads keyed by Operation). An unknown region string triggers std::invalid_argument listing the rejected region.

Source

Thrown at phoenix/xrex/cuda/async_emb/src/async_emb_api.cc:768

    size_t bytes;
  };
  const std::unordered_map<std::string, Region> regions = {
      {"lookup_ids", {Operation::Lookup, layout.token_ids_all, index_bytes}},
      {"lookup_embeddings", {Operation::Lookup, layout.lookup_recv, block_bytes}},
      {"lookup_send", {Operation::Lookup, layout.lookup_send, block_bytes}},
      {"update_indices", {Operation::Update, layout.segment_ids_all, index_bytes}},
      {"update_gradients", {Operation::Update, layout.update_recv, block_bytes}},
      {"update_send", {Operation::Update, layout.update_send, block_bytes}},
      {"update_stats",
       {Operation::Update, layout.row_sq_sums, size_t(spec.num_unique) * sizeof(float)}},
      {"update_grad_accum",
       {Operation::Update,
        layout.grad_accum,
        size_t(spec.num_unique) * size_t(spec.shard_width) * sizeof(float)}},
  };
  auto found = regions.find(region);
  if (found == regions.end()) {
    throw std::invalid_argument("unknown async_emb snapshot region: " + region);
  }
  waitLatest(context_id, found->second.operation, "test snapshot");
  std::vector<uint8_t> snapshot = ctx->snapshot(found->second.offset, found->second.bytes);
  return nb::bytes(reinterpret_cast<const char*>(snapshot.data()), snapshot.size());
}

}

NB_MODULE(async_emb_api, m) {
  m.def("lookup_start_init", [] { return encapsulate(kLookupStartInit); });
  m.def("lookup_start", [] { return encapsulate(kLookupStart); });
  m.def("lookup_done", [] { return encapsulate(kLookupDone); });
  m.def("rowwise_adagrad_update_start_init", [] {
    return encapsulate(kRowwiseAdagradUpdateStartInit);
  });
  m.def("rowwise_adagrad_update_start", [] { return encapsulate(kRowwiseAdagradUpdateStart); });
  m.def("rowwise_adagrad_lazy_update_start_init", [] {
    return encapsulate(kRowwiseAdagradLazyUpdateStartInit);

View on GitHub (pinned to 24c60942c5)

Solutions

1. Check the region keys defined in the regions map in async_emb_api.cc testSnapshot (e.g. the Operation-keyed entries like tokens/grads/Update) and use one of those exact strings
2. Print/inspect the module's exposed constants or tests for the canonical region names
3. Upgrade/downgrade to matching versions if names changed

Example fix

# before
snap = async_emb._test_snapshot(ctx_id, "optimizer_state")

# after
snap = async_emb._test_snapshot(ctx_id, "grads")  # a key from the regions map

VALID_REGIONS = {"tokens", "grads", "update"}  # mirror keys of the regions map in testSnapshot
if region not in VALID_REGIONS:
    raise ValueError(f"region must be one of {VALID_REGIONS}")
snap = async_emb._test_snapshot(ctx_id, region)

def isSnapshotRegion(name: str) -> bool:
    return name in {"tokens", "grads", "update"}

try:
    snap = async_emb._test_snapshot(ctx_id, region)
except ValueError as e:
    if "unknown async_emb snapshot region" in str(e):
        snap = async_emb._test_snapshot(ctx_id, "grads")
    else:
        raise

When it happens

Trigger: Calling _test_snapshot(context_id, region) with a region string not present in the regions map — e.g. "optimizer_state", "weights", a typo like "update ", or wrong casing.

Common situations: Probing the API by guessing region names; version drift where supported region names changed between async_emb releases; passing an enum's integer or a fully-qualified name instead of the short string key.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• async_emb context not initialized
• async_emb: ${label} timed out waiting for step ${step}
• NCCL version query failed: ${ncclGetErrorString(result)}
• could not resolve loaded NCCL library path
• async_emb arena size overflow

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/7edfec255052317c. Report an issue: GitHub.