xai-org/x-algorithm · critical · std::overflow_error

async_emb arena offset overflow

Error message

async_emb arena offset overflow

What it means

While laying out the communication arena, ArenaLayout::build computes running byte offsets for each pipeline slice; if adding a slice's size would overflow size_t, an overflow_error is thrown instead of silently wrapping. This is a defensive arithmetic check on offset+bytes and alignUp.

Source

Thrown at phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc:95

  auto* f32 = reinterpret_cast<float*>(scratch);
  auto* i32 = reinterpret_cast<int32_t*>(scratch);
  const SliceLayout slices{0, 8, 1};
  launch_lookup_dispatch(i32, bf16, bf16, slices, 1, stream);
  launch_grad_dispatch(bf16, bf16, slices, stream);
  launch_grad_segment_sum(bf16, i32, f32, f32, slices, 0, stream);
  launch_lookup_combine(bf16, bf16, slices, stream);
  XAI_CUDA_CHECK(cudaStreamSynchronize(stream));
}

}

ArenaLayout ArenaLayout::build(const PipelineSpec& spec, int world_size) {
  ArenaLayout layout;
  size_t offset = 0;
  auto take = [&offset](size_t bytes) {
    size_t result = offset;
    if (offset > std::numeric_limits<size_t>::max() - bytes) {
      throw std::overflow_error("async_emb arena offset overflow");
    }
    offset = alignUp(offset + bytes);
    return result;
  };

  const size_t slice_bytes = checkedProduct(
      {size_t(spec.tokens_per_rank), size_t(spec.shard_width), sizeof(__nv_bfloat16)}
  );
  layout.token_ids_all =
      take(checkedProduct({size_t(world_size), size_t(spec.tokens_per_rank), sizeof(int32_t)}));
  layout.lookup_send = take(checkedProduct({size_t(world_size), slice_bytes}));
  layout.lookup_recv = take(checkedProduct({size_t(world_size), slice_bytes}));
  layout.segment_ids_all =
      take(checkedProduct({size_t(world_size), size_t(spec.tokens_per_rank), sizeof(int32_t)}));
  layout.update_send = take(checkedProduct({size_t(world_size), slice_bytes}));
  layout.update_recv = take(checkedProduct({size_t(world_size), slice_bytes}));
  layout.grad_accum =
      take(checkedProduct({size_t(spec.num_unique), size_t(spec.shard_width), sizeof(float)}));

View on GitHub (pinned to 24c60942c5)

Solutions

1. Inspect the PipelineSpec values (hidden, rows, dtype size, world_size) for garbage such as -1 cast to huge unsigned.
2. Add upstream bounds checks / clamps on spec fields before building the layout.
3. Catch std::overflow_error at config-validation time and fail fast with a readable config dump.

Example fix

// before
PipelineSpec spec{.hidden = static_cast<size_t>(hidden_int)};  // hidden_int == -1
auto layout = ArenaLayout::build(spec, world_size);

// after
if (hidden_int <= 0 || hidden_int > kMaxHidden) throw std::invalid_argument("bad hidden");
PipelineSpec spec{.hidden = static_cast<size_t>(hidden_int)};
auto layout = ArenaLayout::build(spec, world_size);

auto sane = [](long long v, long long max) { return v > 0 && v <= max; };
if (!sane(spec.rows, 1LL<<40) || !sane(spec.hidden, 1<<20) || world_size <= 0 || world_size > 4096)
    throw std::invalid_argument("pipeline spec out of sane range");

catch (const std::overflow_error& e) { /* dump spec, fail config validation */ }

When it happens

Trigger: Calling ArenaLayout::build(spec, world_size) with a PipelineSpec whose per-slice sizes (hidden dim, vocab/row counts, dtypes, world_size) multiply to a total arena approaching SIZE_MAX on 64-bit — practically only reachable via checkedProduct returning astronomically large values from garbage spec numbers.

Common situations: Passing an uninitialized or misparsed spec (e.g. hidden size read as a negative int then cast to size_t), or unit tests feeding absurd dimensions; not something realistic model configs hit on 64-bit builds.

Related errors

• Block sparse tensors{context} n-block dimension must be <= {
• The layout of mdV is wrong
• async_emb: ${label} timed out waiting for step ${step}
• async_emb context not initialized
• unknown async_emb snapshot region: ${region}

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/2442aaeff995abdd. Report an issue: GitHub.