xai-org/x-algorithm · critical · std::overflow_error

async_emb arena alignment overflow

Error message

async_emb arena alignment overflow

What it means

alignUp rounds arena offsets up to kAlign; if value is within kAlign-1 of SIZE_MAX, rounding would overflow, so std::overflow_error('async_emb arena alignment overflow') is thrown. Like checkedProduct, this guards the arena layout computation in ArenaLayout::build against size_t wraparound.

Source

Thrown at phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc:40

constexpr size_t kAlign = 128;
constexpr int kNcclMinCtas = 1;
constexpr int kNcclMaxCtas = 4;
constexpr std::chrono::minutes kNcclReadyTimeout{5};

size_t checkedProduct(std::initializer_list<size_t> factors) {
  size_t result = 1;
  for (size_t factor : factors) {
    if (factor != 0 && result > std::numeric_limits<size_t>::max() / factor) {
      throw std::overflow_error("async_emb arena size overflow");
    }
    result *= factor;
  }
  return result;
}

size_t alignUp(size_t value) {
  if (value > std::numeric_limits<size_t>::max() - (kAlign - 1)) {
    throw std::overflow_error("async_emb arena alignment overflow");
  }
  return (value + kAlign - 1) / kAlign * kAlign;
}

std::runtime_error ncclError(const char* operation, ncclResult_t result) {
  return std::runtime_error(
      std::string("NCCL ") + operation + " failed: " + ncclGetErrorString(result)
  );
}

void requireEnvironment(const char* name, const char* expected) {
  const char* value = std::getenv(name);
  if (value == nullptr || std::strcmp(value, expected) != 0) {
    throw std::runtime_error(
        "async_emb requires " + std::string(name) + "=" + expected + " before process startup"
    );
  }
}

View on GitHub (pinned to 24c60942c5)

Solutions

1. Fix the upstream spec/config producing absurd sizes (validate all dimensions positive and bounded)
2. Treat this and error 817 together: whichever fires first indicates the same oversized-arena root cause
3. Add pre-creation assertions on spec magnitudes from the Python side

Example fix

# before
spec.num_unique = 2**63          # absurd
ctx = async_emb.create_context(spec)  # layout build throws

# after
assert spec.num_unique < 10**9
ctx = async_emb.create_context(spec)

total = world_size * spec.tokens_per_rank * spec.shard_width
assert 0 < total < 2**62, f"arena too large: {total}"

try:
    ctx = async_emb.create_context(spec)
except OverflowError as e:
    if "alignment overflow" in str(e) or "arena size overflow" in str(e):
        raise ValueError("reduce spec dimensions / shard the table") from e
    raise

When it happens

Trigger: Building a context whose accumulated arena size is so close to SIZE_MAX that adding the alignment mask overflows — practically only reachable via corrupted/overflowing spec inputs (see also 'async_emb arena size overflow').

Common situations: Same class as the product overflow: negative or uninitialized spec dimensions reinterpreted as huge unsigned values; chained large-allocation requests from a bad config.

Related errors

• async_emb arena size overflow
• async_emb: ${label} timed out waiting for step ${step}
• async_emb context not initialized
• unknown async_emb snapshot region: ${region}
• NCCL version query failed: ${ncclGetErrorString(result)}

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/8092dc5c23a184c4. Report an issue: GitHub.