What causes "unsupported profile stage: {forward_mode=}" in sgl-project/sglang?

Update sglang to a version where the profiler covers all forward modes

sgl-project/sglang · error · RuntimeError

unsupported profile stage: {forward_mode=}

Error message

unsupported profile stage: {forward_mode=}

What it means

While converting a batch's forward_mode into a profile stage, the mode was neither prefill, decode, nor idle, so the stage-triggered profiler cannot classify it and raises RuntimeError. This is an internal invariant violation, typically after a new ForwardMode was added without updating the profiler.

Source

Thrown at python/sglang/srt/utils/profile_utils.py:185

        logger.info(
            f"Profiling done. Traces are saved to: {self.profiler_kwargs['output_dir']}"
        )
        self.profiler = None
        # Clear the detailed step-span toggle here too: the v2 trigger auto-stop
        # goes through _do_stop (not SchedulerProfilerManager._stop_profile), so
        # this guarantees the flag resets on every stop path.
        set_detailed_annotations_enabled(False)


def _get_stage_from_forward_mode(forward_mode: ForwardMode):
    if forward_mode.is_prefill():
        return "prefill"
    elif forward_mode.is_decode():
        return "decode"
    elif forward_mode.is_idle():
        return None
    else:
        raise RuntimeError(f"unsupported profile stage: {forward_mode=}")


# ======================================== Stage related ==========================================


class _StageBasedTrigger:
    @dataclass
    class _StageConfig:
        target_count: int

    @dataclass
    class _RunningState:
        curr_stage: str
        curr_count: int

    def __init__(self, on_start: Callable, on_stop: Callable):
        self.on_start = on_start
        self.on_stop = on_stop

View on GitHub (pinned to 0132848349)

Solutions

Update sglang to a version where the profiler covers all forward modes
Disable stage-based profiling (--profile-stages unset) and use whole-run profiling
If developing a new forward mode, add a branch in _get_stage_from_forward_mode

Example fix

# before
stage = _get_stage_from_forward_mode(forward_mode)  # MIXED -> RuntimeError
# after
if forward_mode.is_mixed():
    stage = 'prefill'
else:
    stage = _get_stage_from_forward_mode(forward_mode)

Defensive patterns

Strategy: fallback

Validate before calling

null

Type guard

def is_supported_profile_mode(fm) -> bool:
    return fm is not None and (fm.is_prefill() or fm.is_decode() or fm.is_idle())

Try / catch

try:
    stage = _get_stage_from_forward_mode(forward_mode)
except RuntimeError:
    stage = None  # skip classifying this step

Prevention

Update sglang alongside scheduler/forward-mode changes
Run stage-based profiling in CI smoke tests to catch unclassified modes
Disable --profile-stages if an exotic mode appears

When it happens

Trigger: Scheduler step() handling a forward_mode like MIXED, EXTEND, SPLIT_PREFILL (whatever is not covered by is_prefill/is_decode/is_idle) reaching _get_stage_from_forward_mode.

Common situations: Version mismatch: running a scheduler producing new forward modes with older profiler code, or custom attention/pipeline paths emitting unusual modes during profiling.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/643de53eda3f16d2. Report an issue: GitHub.