sgl-project/sglang · error · RuntimeError

kv-canary: {type(obj).__name__}.{method_name} already wrappe

Error message

kv-canary: {type(obj).__name__}.{method_name} already wrapped by kv-canary

What it means

wrap_method marks wrapped methods with a marker attribute and refuses to wrap the same method twice. Double-wrapping would stack two transforms and corrupt return values, so the idempotency guard raises RuntimeError.

Source

Thrown at python/sglang/srt/kv_canary/pool_patcher/utils.py:33

    """Replace ``obj.method_name`` with a closure that delegates to ``wrapper``.

    ``wrapper(original, *args, **kwargs)`` receives the original bound method as its first arg and the
    call-site args/kwargs as the rest. It decides when (and whether) to call ``original`` and what to
    return. The patched callable is installed as a plain function; :func:`functools.wraps` preserves
    ``__name__`` / ``__doc__`` but the bound-method nature of the original is not retained.

    Raises:
        AttributeError: ``obj`` has no attribute ``method_name``.
        RuntimeError: ``obj.method_name`` has already been wrapped by ``wrap_method`` (idempotency
            guard — re-wrapping silently would stack two transforms and corrupt return values).
    """
    if not hasattr(obj, method_name):
        raise AttributeError(
            f"kv-canary: {type(obj).__name__} missing required method {method_name!r}"
        )
    original = getattr(obj, method_name)
    if getattr(original, _WRAPPED_MARKER_ATTR, None) is not None:
        raise RuntimeError(
            f"kv-canary: {type(obj).__name__}.{method_name} already wrapped by kv-canary"
        )

    @functools.wraps(original)
    def patched(*args: Any, **kwargs: Any) -> Any:
        return wrapper(original, *args, **kwargs)

    setattr(patched, _WRAPPED_MARKER_ATTR, method_name)
    setattr(obj, method_name, patched)

View on GitHub (pinned to 0132848349)

Solutions

1. Check the wrapped marker (getattr(method, _WRAPPED_MARKER_ATTR, None)) or expose/track patch state before calling wrap_method again
2. Add an uninstall/unpatch step that restores the original before re-patching
3. Make the install path idempotent: skip if already wrapped instead of calling wrap_method

Example fix

// before
wrap_method(model, 'forward', wrapper)  # called again -> RuntimeError
# after
if getattr(getattr(model, 'forward', None), '__kv_canary_wrapped__', None) is None:
    wrap_method(model, 'forward', wrapper)

orig = getattr(obj, method_name, None)
if getattr(orig, '__kv_canary_wrapped__', None) is not None:
    return  # already patched — idempotent skip
wrap_method(obj, method_name, wrapper)

def is_wrapped_by_canary(fn) -> bool:
    return getattr(fn, _WRAPPED_MARKER_ATTR, None) is not None

try:
    wrap_method(obj, method_name, wrapper)
except RuntimeError as e:
    if 'already wrapped' not in str(e):
        raise

When it happens

Trigger: Calling wrap_method twice on the same obj/method without unwrapping — e.g. patching model forward in _patch_model_forward, then calling patch again on re-init, or a test reusing a patched object across cases.

Common situations: Calling the kv-canary install/patch routine twice in one process (server restart-in-place, retry logic in tests); patching both at module level and instance level so the same bound method is seen twice.

Related errors

• kv-canary: {type(obj).__name__} missing required method {met
• kv-canary: launch_canary_plan_kernels requires full_to_swa_i
• kv-canary: launch_canary_plan_kernels verify_capacity={verif
• kv-canary: launch_plan_entries_kernel requires req_to_verify
• kv-canary: offsets kernel bs must be in [0, {_PLAN_BS_BLOCK_

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5824775cb2a8f0be. Report an issue: GitHub.