{"record":{"id":"79ce6a72a7888887","repo":"xai-org/x-algorithm","slug":"async-emb-arena-size-overflow","errorCode":null,"errorMessage":"async_emb arena size overflow","messagePattern":"async_emb arena size overflow","errorType":"exception","errorClass":"std::overflow_error","httpStatus":null,"severity":"critical","filePath":"phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc","lineNumber":31,"sourceCode":"\n#include \"absl/log/log.h\"\n#include \"async_emb_kernel.hpp\"\n#include \"cuda_error_utils.hpp\"\n\nnamespace xai::kernels::async_emb {\n\nnamespace {\n\nconstexpr size_t kAlign = 128;\nconstexpr int kNcclMinCtas = 1;\nconstexpr int kNcclMaxCtas = 4;\nconstexpr std::chrono::minutes kNcclReadyTimeout{5};\n\nsize_t checkedProduct(std::initializer_list<size_t> factors) {\n  size_t result = 1;\n  for (size_t factor : factors) {\n    if (factor != 0 && result > std::numeric_limits<size_t>::max() / factor) {\n      throw std::overflow_error(\"async_emb arena size overflow\");\n    }\n    result *= factor;\n  }\n  return result;\n}\n\nsize_t alignUp(size_t value) {\n  if (value > std::numeric_limits<size_t>::max() - (kAlign - 1)) {\n    throw std::overflow_error(\"async_emb arena alignment overflow\");\n  }\n  return (value + kAlign - 1) / kAlign * kAlign;\n}\n\nstd::runtime_error ncclError(const char* operation, ncclResult_t result) {\n  return std::runtime_error(\n      std::string(\"NCCL \") + operation + \" failed: \" + ncclGetErrorString(result)\n  );\n}","sourceCodeStart":13,"sourceCodeEnd":49,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/cuda/async_emb/src/async_emb_comm.cc#L13-L49","documentation":"ArenaLayout::build computes total arena buffer sizes by multiplying dimensions (world size, token counts, widths, element sizes) via checkedProduct. If any product would overflow size_t, std::overflow_error('async_emb arena size overflow') is thrown instead of silently wrapping and allocating a wrong-sized buffer.","triggerScenarios":"Creating/initiating an async_emb context whose spec dimensions (tokens_per_rank, num_unique, shard_width, world size) multiplied with element sizes exceed 2^64-1 — typically from garbage spec values (negative ints cast to huge size_t, or zeroed/uninitialized spec fields).","commonSituations":"Passing negative or uninitialized spec fields from Python that reinterpret as huge size_t; copy-paste spec values with wrong units; a corrupted spec read from checkpoint/config.","solutions":["Sanity-check all spec fields (non-negative, plausible magnitudes) before context creation","Watch for signed->unsigned conversion of negative values in the Python->C++ boundary; fix the producer of the bad spec","Reduce dimensions/precision or shard the embedding table if the config legitimately approaches size_t limits"],"exampleFix":"# before\nspec.tokens_per_rank = -1          # reinterprets as huge size_t\nctx = async_emb.create_context(spec)\n\n# after\nassert spec.tokens_per_rank > 0\nassert spec.shard_width > 0\nctx = async_emb.create_context(spec)","handlingStrategy":"validation","validationCode":"assert spec.tokens_per_rank > 0 and spec.num_unique >= 0 and spec.shard_width > 0\nassert spec.tokens_per_rank * spec.shard_width < 2**40, \"spec too large\"","typeGuard":"def specIsSane(spec) -> bool:\n    return all(0 < v < 2**40 for v in (spec.tokens_per_rank, spec.shard_width)) and spec.num_unique >= 0","tryCatchPattern":"try:\n    ctx = async_emb.create_context(spec)\nexcept OverflowError as e:\n    if \"arena size overflow\" in str(e):\n        raise ValueError(f\"spec dimensions too large: {spec}\") from e\n    raise","preventionTips":["Validate all spec magnitudes before context creation","Beware signed->unsigned conversion of negatives at the Python/C++ boundary"],"tags":["async-emb","cpp","integer-overflow","memory-layout","cuda"],"backgroundTag":"integer-overflow-guard","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}