sgl-project/sglang · error · ValueError

Hf3fsClient.check: {offsets=}, {sizes=}

Error message

Hf3fsClient.check: {offsets=}, {sizes=}

What it means

check() validates batch_read/batch_write inputs: offsets must be page-aligned/monotonic (implied by the sorted offsets condition) and every size must exceed bytes_per_page. On violation it closes the client and raises ValueError echoing the offending offsets and sizes, because the usrbio batch API cannot express such requests.

Source

Thrown at python/sglang/srt/mem_cache/storage/hf3fs/hf3fs_usrbio_client.py:204

        return results

    def check(self, offsets: List[int], tensors: List[torch.Tensor]) -> None:
        sizes = [t.numel() * t.itemsize for t in tensors]
        if any(
            [
                len(offsets) > self.entries,
                len(offsets) != len(sizes),
                all(
                    [
                        offset < 0 or offset + size > self.size
                        for offset, size in zip(offsets, sizes)
                    ]
                ),
                all([size > self.bytes_per_page for size in sizes]),
            ]
        ):
            self.close()
            raise ValueError(f"Hf3fsClient.check: {offsets=}, {sizes=}")

    def get_size(self) -> int:
        return self.size

    def close(self) -> None:
        deregister_fd(self.file)
        os.close(self.file)
        del self.ior_r
        del self.ior_w
        del self.iov_r
        del self.iov_w
        self.shm_r.close()
        self.shm_w.close()

    def flush(self) -> None:
        os.fsync(self.file)

View on GitHub (pinned to 0132848349)

Solutions

1. Check the echoed {offsets=}/{sizes=} — non-multiples of bytes_per_page mean your index math dropped the page-size multiply
2. Coalesce reads so each entry is a full page (size > bytes_per_page) or route small reads through a non-batched path
3. Note the client is closed by check(): recreate the client after fixing inputs instead of reusing it

Example fix

# before
offs = [slot for slot in slots]  # slot indices, not byte offsets
client.batch_read(offs, sizes)

# after
offs = [slot * bytes_per_page for slot in slots]
sizes = [bytes_per_page * n_pages for n_pages in page_counts]
client.batch_read(offs, sizes)

def valid_batch(offsets: list[int], sizes: list[int], bpp: int) -> bool:
    return (
        all(o % bpp == 0 for o in offsets)
        and offsets == sorted(offsets)
        and all(s > bpp for s in sizes)
    )

assert valid_batch(offsets, sizes, client.bytes_per_page)
client.batch_read(offsets, sizes)

def is_page_aligned_batch(offsets, sizes, bytes_per_page):
    return isinstance(offsets, list) and isinstance(sizes, list) and valid_batch(offsets, sizes, bytes_per_page)

try:
    client.batch_read(offsets, sizes)
except ValueError as e:
    if 'Hf3fsClient.check' in str(e):
        client = recreate_client()  # check() closed it
        offsets, sizes = realign_to_pages(offsets, sizes, client.bytes_per_page)
        client.batch_read(offsets, sizes)
    else:
        raise

When it happens

Trigger: Calling batch_read/batch_write with unaligned or non-monotonic offsets, or with a size <= bytes_per_page (e.g. a single-token partial page read). The client is closed, so subsequent calls fail too.

Common situations: Caller computing offsets from token slot indices without multiplying by page size; mixing page-sized and sub-page reads in one batch after a config change to bytes_per_page; sorting bug producing non-monotonic offsets.

Related errors

• kv-canary: RealKvSource.num_bytes_per_token must be a positi
• kv-canary: RealKvSource.read_bytes must be a positive multip
• kv-canary: RealKvSource.tensor dim-1 byte width must be a mu
• seq_len {seq_len} < used rows {used}
• kv-canary: read_bytes must be a multiple of {_REAL_KV_READ_A

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/59ce02265921f617. Report an issue: GitHub.