vllm-project/vllm · error · ValueError

Hf3fsClient.check Failed

Error message

Hf3fsClient.check Failed

What it means

Hf3fsClient.check validates a batch of (offset, size) I/O operations against the client's capacity: number of ops must not exceed 'entries', offsets and sizes arrays must be equal length, every [offset, offset+size) must fall inside the storage file of 'size' bytes, and each size must not exceed 'bytes_per_page'. Any violation closes the client and raises this ValueError.

Source

Thrown at vllm/distributed/kv_transfer/kv_connector/v1/hf3fs/hf3fs_client.py:278

        results = [res.result for res in resv]

        return results

    def check(self, offsets: list[int], tensors: list[torch.Tensor]) -> None:
        sizes = [t.numel() * t.itemsize for t in tensors]
        if any(
            [
                len(offsets) > self.entries,
                len(offsets) != len(sizes),
                any(
                    offset < 0 or offset + size > self.size
                    for offset, size in zip(offsets, sizes)
                ),
                any(size > self.bytes_per_page for size in sizes),
            ]
        ):
            self.close()
            raise ValueError("Hf3fsClient.check Failed")

    def get_size(self) -> int:
        """Get the total size of the storage file.

        Returns:
            Size of the file in bytes
        """
        return self.size

    def close(self) -> None:
        """Close the client and clean up resources."""
        if self._closed:
            return
        self._closed = True
        self._release_resources()

    def flush(self) -> None:
        """Flush any pending writes to disk."""

View on GitHub (pinned to c794754062)

Solutions

1. Ensure per-op size <= bytes_per_page and align the connector's page size with the client configuration
2. Validate len(offsets)==len(sizes) and len(offsets)<=entries before issuing the batch
3. Check that all offsets satisfy 0 <= offset and offset+size <= total size — stale/overflowing page indices are the usual culprit
4. After this error the client is closed; recreate the Hf3fsClient before retrying

Example fix

# before
client.write(offsets=[p * page for p in pages], sizes=[page * 2])  # size > bytes_per_page

# after
assert all(0 <= o and o + s <= client.size for o, s in zip(offsets, sizes))
assert all(s <= client.bytes_per_page for s in sizes)
assert len(offsets) <= client.entries and len(offsets) == len(sizes)
client.write(offsets=offsets, sizes=sizes)

def valid_batch(client, offsets, sizes):
    return (
        len(offsets) == len(sizes)
        and len(offsets) <= client.entries
        and all(0 <= o and o + s <= client.size for o, s in zip(offsets, sizes))
        and all(s <= client.bytes_per_page for s in sizes)
    )

try:
    client.write(offsets=offsets, sizes=sizes)
except ValueError as e:
    if 'Hf3fsClient.check Failed' in str(e):
        client = Hf3fsClient(path, size, bytes_per_page, entries)  # client closed on failure
        raise  # re-report after recreating
    raise

When it happens

Trigger: Calling the client's batch read/write with mismatched offsets/sizes lengths, more entries than the configured concurrency, an out-of-bounds offset/size, or a transfer larger than bytes_per_page.

Common situations: Page-size mismatch between the connector and the Hf3fsClient configuration (size > bytes_per_page); off-by-one or stale page indices producing out-of-range offsets; batching more requests than 'entries' allows.

Related errors

• Invalid initialization parameters
• Invalid request format: need 'rank' and 'keys'
• HTTP request failed: {0}
• kv_connector_module_path cannot be an empty string.
• hf3fs_fuse.io is not available. Please install the hf3fs_fus

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/50067e8e9866a1d9. Report an issue: GitHub.