sgl-project/sglang · error · ValueError

GGUF BF16 payload does not have a byte-pair layout

Error message

GGUF BF16 payload does not have a byte-pair layout

What it means

_bf16_tensor reinterprets a GGUF tensor's raw bytes as BF16 pairs; it requires the numpy array to be uint8 with an even last dimension (bytes). If the payload dtype is not uint8 or its last axis has odd length, bytes cannot form 2-byte BF16 values and ValueError is raised — the buffer isn't a valid BF16 byte-pair layout.

Source

Thrown at python/sglang/srt/model_loader/expert_pack_loader.py:52

    KIMI_K3_MODEL_TYPE,
    validate_expert_pack_model_config,
)
from sglang.srt.model_loader.kimi_k3_gguf import kimi_k3_nonexpert_weights_iterator
from sglang.srt.model_loader.loader import (
    BaseModelLoader,
    _initialize_model,
    device_loading_context,
)
from sglang.srt.model_loader.utils import set_default_torch_dtype
from sglang.srt.runtime_context import get_exec, get_parallel

logger = logging.getLogger(__name__)


def _bf16_tensor(data: np.ndarray) -> torch.Tensor:
    raw = np.asarray(data)
    if raw.dtype != np.uint8 or raw.shape[-1] % 2:
        raise ValueError("GGUF BF16 payload does not have a byte-pair layout")
    values = raw.view(np.uint16).reshape(*raw.shape[:-1], raw.shape[-1] // 2)
    return torch.from_numpy(values.copy()).view(torch.bfloat16)


def _compressor_component(source_name: str) -> str | None:
    if "_compressor_kv.weight" in source_name:
        return "kv"
    if "_compressor_gate.weight" in source_name:
        return "gate"
    return None


def _fused_compressor_name(checkpoint_name: str) -> str:
    result = checkpoint_name.replace(".wkv.weight", ".wkv_gate.weight")
    result = result.replace(".wgate.weight", ".wkv_gate.weight")
    if result == checkpoint_name:
        raise ValueError(f"invalid compressor checkpoint name: {checkpoint_name}")
    return result

View on GitHub (pinned to 0132848349)

Solutions

1. Verify the GGUF file integrity (llama.cpp gguf-dump / re-download the file)
2. Ensure tensor data is read as raw bytes (uint8) — update gguf/sglang to compatible versions
3. Re-convert the model to GGUF with an official, up-to-date convert script
4. Check that tensor byte sizes are even and match expected element counts for BF16

raw = np.asarray(data)
assert raw.dtype == np.uint8 and raw.shape[-1] % 2 == 0, 'BF16 payload must be even-length uint8 bytes'

def is_bf16_byte_payload(raw: np.ndarray) -> bool:
    return raw.dtype == np.uint8 and raw.shape[-1] % 2 == 0

try:
    t = _bf16_tensor(raw)
except ValueError:
    raise RuntimeError('GGUF file appears corrupted or incompatible; re-download/re-convert') from None

When it happens

Trigger: deepseek4_nonexpert_weights_iterator hitting a BF16 GGUF tensor whose numpy payload arrives as a non-uint8 dtype (e.g. pre-decoded float32/uint16) or with an odd trailing byte count (truncated/misaligned data).

Common situations: Corrupted or truncated GGUF file, a gguf reader version that returns typed arrays instead of raw bytes, or an endianness/packing mismatch in a custom GGUF conversion.

Related errors

• GGUFConfig must be constructed from a GGUF checkpoint
• A GGUF encoder checkpoint cannot be combined with a second q
• Cannot parse checkpoint quantization for {component_name!r}:
• GGUF tensor {tensor.name} declares original shape {logical_s
• GGUF tensor {tensor.name} is quantized, but diffusion GGUF c

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6a159203fce906a9. Report an issue: GitHub.