vllm-project/vllm · error · AssertionError

Numerics check failed for case {case}:\n{e}

Error message

Numerics check failed for case {case}:\n{e}

What it means

Wrapper assertion in scripts/benchmark_helion_kernels.py (line 361): re-raises any AssertionError from the numerics comparison (`_assert_close` on outputs and on mutated inputs) with the failing case label prepended, so you know which shape/config case diverged. The inner message carries the real cause: either a structure mismatch (error 114) or values outside atol/rtol tolerance; FP8 leaves get a fixed atol=1, rtol=0.

Source

Thrown at scripts/benchmark_helion_kernels.py:361

        configured_atol = getattr(settings, "autotune_baseline_atol", None)
        configured_rtol = getattr(settings, "autotune_baseline_rtol", None)
        atol = 1e-2 if configured_atol is None else configured_atol
        rtol = 1e-2 if configured_rtol is None else configured_rtol
        _assert_close(
            kernel_output,
            baseline_output,
            atol=atol,
            rtol=rtol,
        )
        _assert_close(
            kernel_inputs,
            baseline_inputs,
            atol=atol,
            rtol=rtol,
        )
    except AssertionError as e:
        raise AssertionError(f"Numerics check failed for case {case}:\n{e}") from e


@dataclass
class CorrectnessResult:
    """Outcome of the numerics check for a single shape case."""

    case: str
    passed: bool
    error: str | None = None


def check_kernel_correctness(
    kernel: Any,
    baseline_fn: Callable,
    inputs_dict: dict[Any, tuple[Any, ...]] | None = None,
) -> list[CorrectnessResult]:
    """Run the per-shape numerics check for a kernel, continuing past failures.

View on GitHub (pinned to c794754062)

Solutions

1. Read the inner error: 'Output structure mismatch' means fix shapes/dtypes first; otherwise values diverge beyond tolerance
2. Compare the kernel's math against the baseline op-by-op (masking, accumulation dtype, reduction order) for the failing case
3. If the divergence is within the noise floor of the dtype, widen atol/rtol explicitly for that case rather than globally

for leaf in tree_flatten(kernel_output)[0]:
    if isinstance(leaf, torch.Tensor):
        assert leaf.isfinite().all(), "kernel produced NaN/inf before tolerance check"

try:
    check_kernel_correctness(case, ...)
except AssertionError as e:
    log_case_failure(case, str(e))  # keep the case label; inspect inner tolerance/structure message
    raise

When it happens

Trigger: Running `check_kernel_correctness` across shape cases: the candidate kernel's outputs (or its in-place modifications to inputs) differ from baseline by more than atol/rtol for the named case.

Common situations: Numerically sloppy kernel rewrites (different reduction order, missing masking, fused ops losing precision); too-tight atol/rtol defaults for bf16/fp16 accumulations; nondeterministic atomics producing run-to-run drift.

Related errors

• Output structure mismatch: {actual_spec} != {expected_spec}
• Cannot find CMake executable
• No compatible wheel found for {arch} at {simple_url}
• Wheel metadata missing path: {wheel}
• CUDA is required for this proof.

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/683f52e309087a3a. Report an issue: GitHub.