sgl-project/sglang · error · ValueError

prefetch_timeout_per_ki_token must be number, got {type(pref

Error message

prefetch_timeout_per_ki_token must be number, got {type(prefetch_timeout_per_ki_token).__name__}

What it means

Validates that prefetch_timeout_per_ki_token (the per-1000-token slope of the linear prefetch timeout, in seconds) in the parsed extra config is an int or float. String, bool, null, or list values fail the isinstance check and raise with the offending type name.

Source

Thrown at python/sglang/srt/mem_cache/hiradix_cache.py:773

            "prefetch_timeout_per_ki_token", defaults.per_ki_token
        )  # seconds per 1024 tokens
        prefetch_timeout_max = extra_config.pop(
            "prefetch_timeout_max", defaults.max
        )  # seconds, upper bound for the linear timeout
        hicache_storage_pass_prefix_keys = extra_config.pop(
            "hicache_storage_pass_prefix_keys", False
        )

        if not isinstance(prefetch_threshold, int):
            raise ValueError(
                f"prefetch_threshold must be int, got {type(prefetch_threshold).__name__}"
            )
        if not isinstance(prefetch_timeout_base, (int, float)):
            raise ValueError(
                f"prefetch_timeout_base must be number, got {type(prefetch_timeout_base).__name__}"
            )
        if not isinstance(prefetch_timeout_per_ki_token, (int, float)):
            raise ValueError(
                f"prefetch_timeout_per_ki_token must be number, got {type(prefetch_timeout_per_ki_token).__name__}"
            )
        if not isinstance(prefetch_timeout_max, (int, float)):
            raise ValueError(
                f"prefetch_timeout_max must be number, got {type(prefetch_timeout_max).__name__}"
            )
        if not isinstance(hicache_storage_pass_prefix_keys, bool):
            raise ValueError(
                "hicache_storage_pass_prefix_keys must be bool, got "
                f"{type(hicache_storage_pass_prefix_keys).__name__}"
            )

        prefetch_timeout_config = PrefetchTimeoutConfig(
            base=float(prefetch_timeout_base),
            per_ki_token=float(prefetch_timeout_per_ki_token),
            max=float(prefetch_timeout_max),
        )

View on GitHub (pinned to 0132848349)

Solutions

Write the value as a bare number: prefetch_timeout_per_ki_token: 0.5
Fix YAML indentation so the key sits at the top level of the extra-config mapping (not nested elsewhere, yielding null)
Validate the config with a quick yaml.safe_load + isinstance check before launching

Example fix

# before
prefetch_timeout_per_ki_token: "0.5"

# after
prefetch_timeout_per_ki_token: 0.5

Defensive patterns

Strategy: type-guard

Validate before calling

v = cfg.get("prefetch_timeout_per_ki_token", 0.1)
assert isinstance(v, (int, float)) and not isinstance(v, bool), "prefetch_timeout_per_ki_token must be numeric"

Type guard

def is_number_not_bool(v) -> bool:
    return isinstance(v, (int, float)) and not isinstance(v, bool)

Prevention

Check YAML indentation so keys resolve to values, not null
Smoke-test the extra config with yaml.safe_load + isinstance checks in CI

When it happens

Trigger: Extra config containing prefetch_timeout_per_ki_token: "0.5", null, true, or a list — anything YAML/TOML/JSON parses to a non-number.

Common situations: Quoted numeric strings in YAML configs; copy-paste from docs that render numbers as strings; misnested YAML putting the key under the wrong mapping so it resolves to null.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a59b98182920e2da. Report an issue: GitHub.