sgl-project/sglang · error · ValueError

Unknown dtype: {dtype}

Error message

Unknown dtype: {dtype}

What it means

_get_and_verify_dtype converts the dtype argument to a torch.dtype; if dtype is a string not present in _STR_DTYPE_TO_TORCH_DTYPE (e.g. 'bfloat', 'float', 'bf16 ' with whitespace, 'fp16'), it raises 'Unknown dtype'.

Source

Thrown at python/sglang/srt/configs/model_config.py:1819

                    if model_type == "gemma":
                        gemma_version = ""
                    else:
                        gemma_version = model_type[5]
                    logger.info(
                        f"For Gemma {gemma_version}, we downcast float32 to bfloat16 instead "
                        "of float16 by default. Please specify `dtype` if you "
                        "want to use float16."
                    )
                    torch_dtype = torch.bfloat16
                else:
                    # Following the common practice, we use float16 for float32
                    # models.
                    torch_dtype = torch.float16
            else:
                torch_dtype = config_dtype
        else:
            if dtype not in _STR_DTYPE_TO_TORCH_DTYPE:
                raise ValueError(f"Unknown dtype: {dtype}")
            torch_dtype = _STR_DTYPE_TO_TORCH_DTYPE[dtype]
    elif isinstance(dtype, torch.dtype):
        torch_dtype = dtype
    else:
        raise ValueError(f"Unknown dtype: {dtype}")

    # Verify the dtype.
    if torch_dtype != config_dtype:
        if torch_dtype == torch.float32:
            # Upcasting to float32 is allowed.
            logger.debug("Upcasting %s to %s.", config_dtype, torch_dtype)
            pass
        elif config_dtype == torch.float32:
            # Downcasting from float32 to float16 or bfloat16 is allowed.
            logger.debug("Downcasting %s to %s.", config_dtype, torch_dtype)
            pass
        else:
            # Casting between float16 and bfloat16 is allowed with a warning.

View on GitHub (pinned to 0132848349)

Solutions

1. Use one of the exact keys of _STR_DTYPE_TO_TORCH_DTYPE: 'float16', 'bfloat16', 'float32' (or 'auto')
2. Strip/normalize user-supplied dtype strings before passing them to ModelConfig
3. Pass a torch.dtype instance instead of a string when constructing ModelConfig programmatically

Example fix

# before
--dtype fp16
# after
--dtype float16

VALID = {'auto','float16','bfloat16','float32','half','float'}  # per _STR_DTYPE_TO_TORCH_DTYPE
if isinstance(dtype, str) and dtype.strip().lower() not in VALID:
    raise SystemExit(f"bad dtype {dtype!r}")

def is_valid_dtype_str(s: str) -> bool:
    from sglang.srt.configs.model_config import _STR_DTYPE_TO_TORCH_DTYPE
    return s in _STR_DTYPE_TO_TORCH_DTYPE

When it happens

Trigger: ModelConfig(..., dtype='fp16') or server --dtype fp16 / bfloat / float — any string not exactly matching keys like 'float16','bfloat16','float32'. Falls through the str branch at model_config.py:1819.

Common situations: Passing shorthand dtype names from other frameworks, trailing whitespace or wrong casing in scripts, typos like 'flaot16'.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• q must be torch.float8_e4m3fn, got {q.dtype}
• Unknown dtype: {sampling_params.dtype}
• Invalid dtype: {sampling_params.dtype}
• DeepSeekV4 only supports interleave CP strategy, got {cfg.cp
• HiSparse requires one of {HISPARSE_KV_CACHE_DTYPES} KV cache

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/48ba5b5aaf0438d8. Report an issue: GitHub.