{"record":{"id":"f90557cbcf4ee2f2","repo":"vllm-project/vllm","slug":"output-structure-mismatch-actual-spec-expec","errorCode":null,"errorMessage":"Output structure mismatch: {actual_spec} != {expected_spec}","messagePattern":"Output structure mismatch: (.+?) != (.+?)","errorType":"exception","errorClass":"AssertionError","httpStatus":null,"severity":"error","filePath":"scripts/benchmark_helion_kernels.py","lineNumber":307,"sourceCode":"\n_REDUCERS: dict[str, Callable[[list[float]], float]] = {\n    \"min\": min,\n    \"max\": max,\n    \"mean\": statistics.fmean,\n    \"median\": statistics.median,\n}\n\n\ndef _reduce(times: list[float], return_mode: str) -> float:\n    return _REDUCERS[return_mode](times)\n\n\ndef _assert_close(actual: object, expected: object, atol: float, rtol: float) -> None:\n    \"\"\"Compare pytrees, allowing the one-ULP FP8 variance used by kernel tests.\"\"\"\n    actual_flat, actual_spec = tree_flatten(actual)\n    expected_flat, expected_spec = tree_flatten(expected)\n    if actual_spec != expected_spec:\n        raise AssertionError(\n            f\"Output structure mismatch: {actual_spec} != {expected_spec}\"\n        )\n\n    for actual_leaf, expected_leaf in zip(actual_flat, expected_flat, strict=True):\n        is_fp8 = isinstance(actual_leaf, torch.Tensor) and is_fp8_dtype(\n            actual_leaf.dtype\n        )\n        helion_assert_close(\n            actual_leaf,\n            expected_leaf,\n            atol=1 if is_fp8 else atol,\n            rtol=0 if is_fp8 else rtol,\n        )\n\n\ndef check_correctness(\n    kernel: Any,\n    baseline_fn: Callable,","sourceCodeStart":289,"sourceCodeEnd":325,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/scripts/benchmark_helion_kernels.py#L289-L325","documentation":"Assertion in scripts/benchmark_helion_kernels.py `_assert_close` (line 307): the pytree structure spec (shapes, dtypes, nesting) of the kernel output differs from the baseline's spec, so element-wise comparison is not even meaningful. Structure is compared before any tolerance logic; FP8 one-ULP leniency only applies after this check.","triggerScenarios":"Running the Helion kernel benchmark/correctness harness where the candidate kernel returns e.g. a tuple instead of a single tensor, different number of pytree leaves, different nesting, or different dtypes/shapes than the reference implementation for the same inputs.","commonSituations":"Editing a Helion kernel and changing what it returns (unbinding a tuple, returning None placeholder); dtype promotions added inside the kernel (fp32 vs bf16 out); baseline refactors that changed the reference output container; shape bugs producing transposed outputs.","solutions":["Make the kernel return the same pytree structure as the baseline: same container shape, leaf count, and dtypes","Diff the printed specs — the mismatch names exactly which leaf/shape/dtype differs","Re-run against the unmodified baseline kernel to confirm the harness itself is not stale"],"exampleFix":"# before (kernel returns tuple, baseline returns single tensor)\nreturn (out,)\n\n# after\nreturn out","handlingStrategy":"validation","validationCode":"from torch.utils._pytree import tree_flatten\nks, kspec = tree_flatten(kernel_output)\nbs, bspec = tree_flatten(baseline_output)\nassert kspec == bspec, f\"structure differs before running: {kspec} vs {bspec}\"","typeGuard":null,"tryCatchPattern":"try:\n    _assert_close(kernel_output, baseline_output, atol, rtol)\nexcept AssertionError as e:\n    if \"structure mismatch\" in str(e):\n        # fix the kernel's return container/dtypes, not tolerances\n        raise","preventionTips":["Keep the kernel's return type identical to the baseline when refactoring (same tuple arity, dtypes)","Run the smallest shape case first — structure errors surface immediately and cheaply"],"tags":["python","benchmark","helion","kernels","testing"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}