vllm-project/vllm · error · ValueError

Hidden-states block-size mismatch: derived {self._block_size

Error message

Hidden-states block-size mismatch: derived {self._block_size} but buffer block size is {self._kv_cache.shape[1]}; read slots would be wrong (likely a hybrid block-size resolution bug).

What it means

During worker-side binding, the connector derived a block size for the hidden-states buffer but the actual allocated torch buffer's second dimension differs. Because read slots are computed from the derived block size, a mismatch would silently corrupt reads, so the code raises (deliberately not assert, so it survives python -O). The message points at a hybrid block-size resolution bug in the KV cache config.

Source

Thrown at vllm/distributed/kv_transfer/kv_connector/v1/example_hidden_states_connector.py:314

        from vllm.model_executor.models.extract_hidden_states import (
            CacheOnlyAttentionLayer,
        )

        # Filter layers to only include CacheOnlyAttentionLayers
        layers = get_layers_from_vllm_config(
            self._vllm_config, CacheOnlyAttentionLayer, list(kv_caches.keys())
        )
        self.cache_layers = list(layers.keys())
        assert len(self.cache_layers) == 1, (
            f"Expected 1 CacheOnlyAttentionLayer, got {len(self.cache_layers)}"
        )
        self._kv_cache = kv_caches[self.cache_layers[0]]

        # Block size must match the indexed buffer, else reads hit the wrong
        # slots. Raise (not assert) so the check survives `python -O`.
        if self._block_size != self._kv_cache.shape[1]:
            raise ValueError(
                f"Hidden-states block-size mismatch: derived {self._block_size} "
                f"but buffer block size is {self._kv_cache.shape[1]}; read slots "
                "would be wrong (likely a hybrid block-size resolution bug)."
            )

    @staticmethod
    def _write_tensors(
        tensors: dict[str, torch.Tensor],
        event: torch.cuda.Event,
        filename: str,
        lock_fd: int | None,
    ) -> None:
        """Thread worker: wait for async DtoH copy, write to disk, release lock.

        ``lock_fd`` is an open file descriptor on the companion ``.lock``
        file with ``LOCK_EX`` already held.  Closing it releases the lock,
        which unblocks any client sleeping on ``LOCK_SH``.
        """

View on GitHub (pinned to c794754062)

Solutions

Report/investigate as a vLLM hybrid KV-cache block-size resolution bug: compare kv_cache_config group specs' block_size with the allocated buffer shape
As a workaround, force a single uniform block size for all groups (e.g. adjust --block-size or max block sizes of constituent specs so they unify)
Check you are on a version where the hybrid allocator and this connector agree; upgrade vLLM

Defensive patterns

Strategy: validation

Validate before calling

assert derived_block_size == kv_caches[layer].shape[1], (
    f"block size {derived_block_size} != buffer {kv_caches[layer].shape[1]}"
)

Try / catch

try:
    connector.bind_connector_metadata(md)
except ValueError as e:
    if 'block-size mismatch' in str(e):
        raise RuntimeError('Hybrid block-size resolution bug — file a vLLM issue') from e
    raise

Prevention

Keep block sizes uniform across groups in hybrid configs
Re-run this check after any vLLM upgrade that touches the KV cache allocator

When it happens

Trigger: bind_connector_metadata / kv_caches binding: kv_caches[cache_layer].shape[1] != block size derived from the kv_cache_config group spec. Happens when hybrid (multi-spec) block-size resolution computes a different block size than the allocator used for the CacheOnlyAttentionLayer buffer.

Common situations: Hybrid KV cache models (e.g. sliding+full attention plus a hidden-states layer) where block-size unification picked a different size for the group; bugs or version mismatches between spec derivation and buffer allocation.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/33b8b7e55798135a. Report an issue: GitHub.