sgl-project/sglang · error · ValueError

L3 cleaner low_watermark must be lower than high_watermark (

Error message

L3 cleaner low_watermark must be lower than high_watermark (low_watermark={}, high_watermark={})

What it means

Error "L3 cleaner low_watermark must be lower than high_watermark (low_watermark={}, high_watermark={})" thrown in sgl-project/sglang.

Source

Thrown at python/sglang/srt/mem_cache/storage/nixl/nixl_cleaner.py:112

        low_watermark: float = _DEFAULT_LOW_WATERMARK,
        interval_sec: float = _DEFAULT_INTERVAL_SEC,
        recheck_groups: int = _DEFAULT_RECHECK_GROUPS,
        unlink_workers: Optional[int] = None,
    ) -> None:
        if isinstance(storage_dirs, str):
            storage_dirs = [storage_dirs] if storage_dirs else []
        self.storage_dirs = [path for path in storage_dirs if path]
        self.tp_rank = tp_rank
        self.high_watermark = high_watermark
        self.low_watermark = low_watermark
        self.interval_sec = interval_sec
        self.recheck_groups = max(1, recheck_groups)
        self.unlink_workers = unlink_workers or max(
            8, 8 * max(len(self.storage_dirs), 1)
        )

        if self.low_watermark >= self.high_watermark:
            raise ValueError(
                "L3 cleaner low_watermark must be lower than high_watermark "
                f"(low_watermark={self.low_watermark}, "
                f"high_watermark={self.high_watermark})"
            )

        self._stop = threading.Event()
        self._thread: Optional[threading.Thread] = None

    def start(self) -> None:
        """Start the cleaner thread on TP rank 0."""
        if self.tp_rank != 0 or not self.storage_dirs:
            return
        if self._thread is not None and self._thread.is_alive():
            return
        self._thread = threading.Thread(
            target=self._loop, name="hicache-l3-cleaner", daemon=True
        )
        self._thread.start()

View on GitHub (pinned to 0132848349)

When it happens

Trigger: Thrown at python/sglang/srt/mem_cache/storage/nixl/nixl_cleaner.py:112 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5daf1825a368c3cd. Report an issue: GitHub.