sgl-project/sglang · error · IOError

Short read for {suffixed}

Error message

Short read for {suffixed}

What it means

Raised by HiCache file storage backend (hicache_storage.py) when reading a tensor page from a .bin file on disk returns fewer bytes than the tensor's expected size (numel * element_size). It means the backing file is truncated, corrupted, or was written with a different layout/dtype than the reader expects. The read is done via raw f.readinto into the target tensor's byte buffer, so any size mismatch surfaces as a short read.

Source

Thrown at python/sglang/srt/mem_cache/hicache_storage.py:478

            stem = fn[:-4]
            # Only files belonging to this rank/model.
            if stem.endswith(self.config_suffix):
                self.metadata_cache.add(stem)

    def get(
        self,
        key: str,
        target_location: torch.Tensor,
        target_sizes: Optional[Any] = None,
    ) -> torch.Tensor | None:
        suffixed = self._get_suffixed_key(key)
        tensor_path = os.path.join(self.file_path, f"{suffixed}.bin")
        try:
            expected = target_location.numel() * target_location.element_size()
            with open(tensor_path, "rb", buffering=0) as f:
                buf = memoryview(target_location.view(torch.uint8).contiguous().numpy())
                if f.readinto(buf) != expected:
                    raise IOError(f"Short read for {suffixed}")
            self._evictor.touch(suffixed, tensor_path)
            if self.metadata_cache is not None:
                self.metadata_cache.add(suffixed)
            return target_location
        except FileNotFoundError:
            if self.metadata_cache is not None:
                self.metadata_cache.remove(suffixed)
            logger.warning(f"Failed to fetch {key} from HiCacheFile storage.")
            return None

    def batch_get(
        self,
        keys: List[str],
        target_locations: List[torch.Tensor],
        target_sizes: Optional[Any] = None,
    ) -> List[torch.Tensor | None]:
        return [
            self.get(key, target_location)

View on GitHub (pinned to 0132848349)

Solutions

1. Delete or regenerate the affected cache files/directory so pages are rewritten with the current layout (page contents are reconstructable, they are just a cache)
2. Verify hicache configuration (page size, dtype, layer count, mem layout) matches what was used when the dump was written
3. Check filesystem integrity/free space and ensure the writer fully flushes/closes files (fsync) before readers access them
4. If sharing a cache directory across models/runs, give each config its own --hicache-dir to avoid layout mismatches

Example fix

# before: reusing stale dump after changing page size / dtype
server_args = ServerArgs(..., hicache_dir="/cache/shared", page_size=64)

# after: separate directory per layout, or wipe stale dump
import shutil; shutil.rmtree("/cache/shared", ignore_errors=True)
server_args = ServerArgs(..., hicache_dir="/cache/mha-bf16-p64", page_size=64)

import os
expected = target.numel() * target.element_size()
path = os.path.join(hicache_dir, f"{suffixed}.bin")
if os.path.exists(path) and os.path.getsize(path) != expected:
    os.remove(path)  # stale/corrupt page; it will be recomputed

try:
    tensor = backend.get(suffixed, target)
except IOError as e:
    if "Short read" in str(e):
        # treat as cache miss: delete corrupt file and recompute page
        os.remove(os.path.join(backend.file_path, f"{suffixed}.bin"))
        tensor = None
    else:
        raise

When it happens

Trigger: Calling get()/batch_get() on FileSystemBackend (or _read_page) for a suffixed page whose {suffixed}.bin file is smaller than target_location.numel()*element_size(); e.g. file truncated mid-write by a crash, page saved under a different dtype/page size and reused after config change, or disk/NFS returning partial data.

Common situations: Reusing a persisted HiCache dump after changing --page-size, kv dtype (fp8 vs bf16), or tensor layout; a server crash while writing pages leaving truncated .bin files; copying cache directories incompletely; multiple writers with different model configs sharing one hicache directory.

Related errors

• /v1/models ${response.status}
• Unsupported msgpack byte ${b}
• This browser does not support gzip stream decoding
• Image file not found: {image_path}
• Mask file not found: {mask_path}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e49dc4e12102c3e9. Report an issue: GitHub.