sgl-project/sglang · critical · Error

delta payload size mismatch: expected ${expectedSize}, got $

Error message

delta payload size mismatch: expected ${expectedSize}, got ${restored.length}

What it means

The CustomAllreduceImpl dispatch only instantiates cross_device_reduce kernels for world sizes 2, 4, 6, 8 via REDUCE_CASE macros. Any other world_size_ at allreduce time hits the default branch.

Source

Thrown at python/sglang/multimodal_gen/apps/realtime_webui/decoder_worker.js:27

function reset() {
  lastFrame = null;
}

async function gunzipBytes(payload) {
  if (typeof DecompressionStream === "undefined") {
    throw new Error("This browser does not support gzip stream decoding");
  }
  const stream = new Blob([payload]).stream().pipeThrough(new DecompressionStream("gzip"));
  return new Uint8Array(await new Response(stream).arrayBuffer());
}

async function restoreDeltaGzipFrames(header, payload) {
  const frameBytes = Number(header.bytes_per_frame);
  const count = Number(header.num_frames);
  const expectedSize = frameBytes * count;
  const restored = await gunzipBytes(payload);
  if (restored.length !== expectedSize) {
    throw new Error(`delta payload size mismatch: expected ${expectedSize}, got ${restored.length}`);
  }

  let previous = header.delta_reference === "previous-frame" ? lastFrame : null;
  if (header.delta_reference === "previous-frame") {
    if (!previous) throw new Error("Missing previous frame for delta payload");
    if (previous.byteLength !== frameBytes) {
      throw new Error("Previous frame size does not match current delta payload");
    }
  }

  for (let f = 0; f < count; f++) {
    const offset = f * frameBytes;
    if (previous) {
      for (let i = 0; i < frameBytes; i++) restored[offset + i] ^= previous[i];
    }
    previous = restored.slice(offset, offset + frameBytes);
  }
  lastFrame = previous;

View on GitHub (pinned to 0132848349)

Solutions

1. Use an even TP size in {2,4,6,8}
2. Disable custom allreduce (fall back to RCCL) for unsupported sizes
3. Guard at startup: assert world_size in (2,4,6,8) before enabling custom kernels

assert custom_ar.world_size_ in (2,4,6,8), 'unsupported TP for custom allreduce'

When it happens

Trigger: Calling the allreduce kernel with world_size_ not in {2,4,6,8}, i.e. the object was constructed (or the world changed) to an unsupported size.

Common situations: Odd GPU counts (3,5,7) reaching this path despite init checks elsewhere; world_size 1 erroneously using custom allreduce; version drift where init validation was bypassed.

Related errors

• This browser does not support worker image decoding
• Generate subcommand is not yet supported for model: {model_p
• Error: --model-type requires a non-empty value.
• Out-of-tree serve backends cannot replace reserved or built-
• Missing previous frame for delta payload

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f93a092ddd761113. Report an issue: GitHub.