What causes "Output structure mismatch: {actual_spec} != {expected_spec}" in vllm-project/vllm?

Make the kernel return the same pytree structure as the baseline: same container shape, leaf count, and dtypes

vllm-project/vllm · error · AssertionError

Output structure mismatch: {actual_spec} != {expected_spec}

Error message

Output structure mismatch: {actual_spec} != {expected_spec}

What it means

Assertion in scripts/benchmark_helion_kernels.py `_assert_close` (line 307): the pytree structure spec (shapes, dtypes, nesting) of the kernel output differs from the baseline's spec, so element-wise comparison is not even meaningful. Structure is compared before any tolerance logic; FP8 one-ULP leniency only applies after this check.

Source

Thrown at scripts/benchmark_helion_kernels.py:307

_REDUCERS: dict[str, Callable[[list[float]], float]] = {
    "min": min,
    "max": max,
    "mean": statistics.fmean,
    "median": statistics.median,
}


def _reduce(times: list[float], return_mode: str) -> float:
    return _REDUCERS[return_mode](times)


def _assert_close(actual: object, expected: object, atol: float, rtol: float) -> None:
    """Compare pytrees, allowing the one-ULP FP8 variance used by kernel tests."""
    actual_flat, actual_spec = tree_flatten(actual)
    expected_flat, expected_spec = tree_flatten(expected)
    if actual_spec != expected_spec:
        raise AssertionError(
            f"Output structure mismatch: {actual_spec} != {expected_spec}"
        )

    for actual_leaf, expected_leaf in zip(actual_flat, expected_flat, strict=True):
        is_fp8 = isinstance(actual_leaf, torch.Tensor) and is_fp8_dtype(
            actual_leaf.dtype
        )
        helion_assert_close(
            actual_leaf,
            expected_leaf,
            atol=1 if is_fp8 else atol,
            rtol=0 if is_fp8 else rtol,
        )


def check_correctness(
    kernel: Any,
    baseline_fn: Callable,

View on GitHub (pinned to c794754062)

Solutions

Make the kernel return the same pytree structure as the baseline: same container shape, leaf count, and dtypes
Diff the printed specs — the mismatch names exactly which leaf/shape/dtype differs
Re-run against the unmodified baseline kernel to confirm the harness itself is not stale

Example fix

# before (kernel returns tuple, baseline returns single tensor)
return (out,)

# after
return out

Defensive patterns

Strategy: validation

Validate before calling

from torch.utils._pytree import tree_flatten
ks, kspec = tree_flatten(kernel_output)
bs, bspec = tree_flatten(baseline_output)
assert kspec == bspec, f"structure differs before running: {kspec} vs {bspec}"

Try / catch

try:
    _assert_close(kernel_output, baseline_output, atol, rtol)
except AssertionError as e:
    if "structure mismatch" in str(e):
        # fix the kernel's return container/dtypes, not tolerances
        raise

Prevention

Keep the kernel's return type identical to the baseline when refactoring (same tuple arity, dtypes)
Run the smallest shape case first — structure errors surface immediately and cheaply

When it happens

Trigger: Running the Helion kernel benchmark/correctness harness where the candidate kernel returns e.g. a tuple instead of a single tensor, different number of pytree leaves, different nesting, or different dtypes/shapes than the reference implementation for the same inputs.

Common situations: Editing a Helion kernel and changing what it returns (unbinding a tuple, returning None placeholder); dtype promotions added inside the kernel (fp32 vs bf16 out); baseline refactors that changed the reference output container; shape bugs producing transposed outputs.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/f90557cbcf4ee2f2. Report an issue: GitHub.