xai-org/x-algorithm · critical · std::overflow_error

async_emb arena size overflow

Error message

async_emb arena size overflow

What it means

ArenaLayout::build computes total arena buffer sizes by multiplying dimensions (world size, token counts, widths, element sizes) via checkedProduct. If any product would overflow size_t, std::overflow_error('async_emb arena size overflow') is thrown instead of silently wrapping and allocating a wrong-sized buffer.

Source

Thrown at phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc:31

#include "absl/log/log.h"
#include "async_emb_kernel.hpp"
#include "cuda_error_utils.hpp"

namespace xai::kernels::async_emb {

namespace {

constexpr size_t kAlign = 128;
constexpr int kNcclMinCtas = 1;
constexpr int kNcclMaxCtas = 4;
constexpr std::chrono::minutes kNcclReadyTimeout{5};

size_t checkedProduct(std::initializer_list<size_t> factors) {
  size_t result = 1;
  for (size_t factor : factors) {
    if (factor != 0 && result > std::numeric_limits<size_t>::max() / factor) {
      throw std::overflow_error("async_emb arena size overflow");
    }
    result *= factor;
  }
  return result;
}

size_t alignUp(size_t value) {
  if (value > std::numeric_limits<size_t>::max() - (kAlign - 1)) {
    throw std::overflow_error("async_emb arena alignment overflow");
  }
  return (value + kAlign - 1) / kAlign * kAlign;
}

std::runtime_error ncclError(const char* operation, ncclResult_t result) {
  return std::runtime_error(
      std::string("NCCL ") + operation + " failed: " + ncclGetErrorString(result)
  );
}

View on GitHub (pinned to 24c60942c5)

Solutions

1. Sanity-check all spec fields (non-negative, plausible magnitudes) before context creation
2. Watch for signed->unsigned conversion of negative values in the Python->C++ boundary; fix the producer of the bad spec
3. Reduce dimensions/precision or shard the embedding table if the config legitimately approaches size_t limits

Example fix

# before
spec.tokens_per_rank = -1          # reinterprets as huge size_t
ctx = async_emb.create_context(spec)

# after
assert spec.tokens_per_rank > 0
assert spec.shard_width > 0
ctx = async_emb.create_context(spec)

assert spec.tokens_per_rank > 0 and spec.num_unique >= 0 and spec.shard_width > 0
assert spec.tokens_per_rank * spec.shard_width < 2**40, "spec too large"

def specIsSane(spec) -> bool:
    return all(0 < v < 2**40 for v in (spec.tokens_per_rank, spec.shard_width)) and spec.num_unique >= 0

try:
    ctx = async_emb.create_context(spec)
except OverflowError as e:
    if "arena size overflow" in str(e):
        raise ValueError(f"spec dimensions too large: {spec}") from e
    raise

When it happens

Trigger: Creating/initiating an async_emb context whose spec dimensions (tokens_per_rank, num_unique, shard_width, world size) multiplied with element sizes exceed 2^64-1 — typically from garbage spec values (negative ints cast to huge size_t, or zeroed/uninitialized spec fields).

Common situations: Passing negative or uninitialized spec fields from Python that reinterpret as huge size_t; copy-paste spec values with wrong units; a corrupted spec read from checkpoint/config.

Related errors

• async_emb arena alignment overflow
• async_emb: ${label} timed out waiting for step ${step}
• async_emb context not initialized
• unknown async_emb snapshot region: ${region}
• NCCL version query failed: ${ncclGetErrorString(result)}

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/79ce6a72a7888887. Report an issue: GitHub.