sgl-project/sglang · error · RuntimeError

[Grafter] tags={tags} matched BOTH grafter_b2t_filter and gr

Error message

[Grafter] tags={tags} matched BOTH grafter_b2t_filter and grafter_t2b_filter

What it means

The Grafter can interpose data flowing bottom-to-top (B2T) or top-to-bottom (T2B), chosen by tag filters. If a tag dict matches BOTH grafter_b2t_filter and grafter_t2b_filter, direction is ambiguous and the grafter aborts with a RuntimeError to avoid grafting in the wrong direction.

Source

Thrown at python/sglang/srt/debug_utils/dumper.py:964

                f"before_overridden={info_before_overridden} "
                f"to_override={get_tensor_info(value_to_override)} "
                f"diff_pre_vs_new={diff}"
            )
            value.copy_(value_to_override)
        except Exception as e:
            _log(
                f"[Grafter] recv role={role.value} dir={direction.value} "
                f"tags={tags} transform/copy_ raised {type(e).__name__}: {e}; "
                f"skipping graft for this call (target tensor unchanged)\n"
                f"{traceback.format_exc()}"
            )

    def _classify_direction(self, tags: dict) -> Optional["_GraftDirection"]:
        cfg = self._config
        match_b2t = self._match(cfg.grafter_b2t_filter, tags)
        match_t2b = self._match(cfg.grafter_t2b_filter, tags)
        if match_b2t and match_t2b:
            raise RuntimeError(
                f"[Grafter] tags={tags} matched BOTH grafter_b2t_filter and grafter_t2b_filter"
            )
        if match_b2t:
            return _GraftDirection.B2T
        if match_t2b:
            return _GraftDirection.T2B
        return None

    @staticmethod
    def _is_sender(*, role: "_GraftRole", direction: "_GraftDirection") -> bool:
        # baseline is the sender for B2T names; target is the sender for T2B.
        return (role == _GraftRole.BASELINE) == (direction == _GraftDirection.B2T)

    def _sender_slice(self, *, direction: "_GraftDirection", gathered: list) -> list:
        cfg = self._config
        if direction == _GraftDirection.B2T:
            return gathered[: cfg.grafter_baseline_world_size]
        return gathered[cfg.grafter_baseline_world_size :]

View on GitHub (pinned to 0132848349)

Solutions

1. Make the filters mutually exclusive, e.g. disjoint tag values per direction
2. Avoid empty/wildcard filters that match every tag dict
3. Test with a single tagged tensor and assert exactly one direction matches before running the full model

Example fix

# before
cfg.grafter_b2t_filter = {"phase": "decode"}
cfg.grafter_t2b_filter = {"phase": "decode"}
# after
cfg.grafter_b2t_filter = {"phase": "decode", "dir": "up"}
cfg.grafter_t2b_filter = {"phase": "decode", "dir": "down"}

def _matches(filt, tags):
    return all(tags.get(k) == v for k, v in (filt or {}).items())
probe_tags = {"phase": "decode", "dir": "up"}
assert not (_matches(cfg.grafter_b2t_filter, probe_tags) and _matches(cfg.grafter_t2b_filter, probe_tags))

try:
    grafter.maybe_intercept(tags)
except RuntimeError as e:
    if "matched BOTH" in str(e):
        # fix filters to be disjoint, then retry
        ...

When it happens

Trigger: Configuring grafter_b2t_filter={'phase': 'decode'} and grafter_t2b_filter={'phase': 'decode'} (or overlapping patterns) so maybe_intercept's _classify_direction matches both for some tensor's tags.

Common situations: Copy-pasting one filter as the start of the other and editing incompletely; overly broad filters like {} (match-all) on both channels.

Related errors

• requires #senders == #recvs but got #senders={len(received_l
• requires matching shapes but received_list[{my_recv_rank}].s
• world_size must be positive and divide global_heads
• Group {group_name} is destroyed.
• world_size ({world_size}) is less than tensor_parallel_degre

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3b3743dc4ec05b83. Report an issue: GitHub.