sgl-project/sglang · error · NotImplementedError

kv-canary: launch_per_forward not supported on sweep endpoin

Error message

kv-canary: launch_per_forward not supported on sweep endpoint {self.kernel_kind.name}

What it means

launch_per_forward is only implemented for single fixed kernels; canary endpoint objects whose kernel_kind carries a sweep tag run a sweep of kernels instead and cannot be driven per-forward. Calling launch_per_forward on such an endpoint raises NotImplementedError naming the sweep kind.

Source

Thrown at python/sglang/srt/kv_canary/endpoint.py:55

    kernel_run_counter_view: torch.Tensor
    enable_chain_position_assert: torch.Tensor

    def launch_per_forward(
        self,
        *,
        verify_plan: VerifyPlan,
        write_plan: WritePlan,
        input_ids: torch.Tensor,
        positions: torch.Tensor,
        out_cache_loc: torch.Tensor,
        enable_write_input_assert: bool,
        enable_verify_token_assert: bool,
        expected_inputs: ExpectedInputs,
        violation_log: ViolationLog,
        real_kv_hash_mode: RealKvHashMode,
    ) -> None:
        if _is_sweep_tag(self.kernel_kind):
            raise NotImplementedError(
                f"kv-canary: launch_per_forward not supported on sweep endpoint {self.kernel_kind.name}"
            )

        context = self._make_verify_or_write_context(
            violation_log=violation_log,
            real_kv_hash_mode=real_kv_hash_mode,
        )
        launch_canary_verify_kernel(
            context=context,
            plan=verify_plan,
            check_verify_expected_token=enable_verify_token_assert,
        )

        # SWA endpoints translate the per-token slot indices via a device tensor index op before invoking the write kernel.
        if self.full_to_swa_index_mapping is not None:
            out_cache_loc_for_canary = self.full_to_swa_index_mapping[out_cache_loc]
        else:
            out_cache_loc_for_canary = out_cache_loc

View on GitHub (pinned to 0132848349)

Solutions

1. Dispatch sweep-tagged endpoints to the sweep launch path, not launch_per_forward (check _is_sweep_tag(kernel_kind) before calling)
2. Use a single (non-sweep) kernel_kind for endpoints that must support per-forward launches
3. Guard the call site with _is_sweep_tag and skip/log for sweep endpoints

Example fix

# before
for ep in endpoints:
    ep.launch_per_forward(ctx, ...)  # raises on sweep endpoints

# after
for ep in endpoints:
    if not _is_sweep_tag(ep.kernel_kind):
        ep.launch_per_forward(ctx, ...)

from sglang.srt.kv_canary.endpoint import _is_sweep_tag
for ep in endpoints:
    if _is_sweep_tag(ep.kernel_kind):
        continue  # route sweep endpoints to the sweep driver instead
    ep.launch_per_forward(ctx, ...)

def supports_per_forward(ep) -> bool:
    return not _is_sweep_tag(ep.kernel_kind)

try:
    ep.launch_per_forward(...)
except NotImplementedError as e:
    if 'launch_per_forward not supported on sweep endpoint' in str(e):
        run_sweep_instead(ep)
    else:
        raise

When it happens

Trigger: Calling endpoint.launch_per_forward(...) (directly or via launch_endpoints_per_forward) on an endpoint constructed with a sweep kernel_kind (any kind matching the internal sweep tag) instead of a single kernel kind.

Common situations: Extending kv-canary with a new sweep-style kernel kind and routing it through the per-forward path; internal code changes that call per-forward launch on all endpoints indiscriminately instead of dispatching sweep endpoints to the sweep driver.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• /v1/models ${response.status}
• Unsupported msgpack byte ${b}
• This browser does not support gzip stream decoding
• {self._op_label()}: no triton backend
• Backward pass is not implemented yet and we do not have plan

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7df6be3610fdb193. Report an issue: GitHub.