What causes "The batch size is not consistent across layers: {values}" in huggingface/transformers?

Ensure every tensor written into the cache (and every preloaded layer) shares one batch dimension

huggingface/transformers · error · ValueError

The batch size is not consistent across layers: {values}

Error message

The batch size is not consistent across layers: {values}

What it means

Cache.batch_size property raises ValueError when initialized layers report different batch sizes. It collects layer.batch_size across layers that have the attribute; more than one distinct value means the cache holds mutually inconsistent states (usually from mismatched inputs or wrongly concatenated caches).

Source

Thrown at src/transformers/cache_utils.py:1641

        """
        Calling this function will activate past state recording, meaning that cache with fixed size such as a linear cache will
        wait for a call to `crop` before restricting the size of its cached states, in order to be able to retrieve previous full states.
        """
        for layer_idx in range(len(self.layers)):
            if hasattr(self.layers[layer_idx], "activate_past_recording"):
                self.layers[layer_idx].activate_past_recording()

    @property
    def batch_size(self) -> int:
        """Return the batch size of the cache, or ``-1`` if no layer has been initialized yet
        (e.g. an all-linear-attention cache queried before the first forward)."""
        # ``LinearAttentionLayer`` sets ``batch_size`` lazily — skip layers that haven't been
        # initialized yet (``generate`` queries this on a fresh cache during cache-reuse checks).
        values = [layer.batch_size for layer in self.layers if hasattr(layer, "batch_size")]
        if not values:
            return -1
        if len(set(values)) > 1:
            raise ValueError(f"The batch size is not consistent across layers: {values}")
        return values[0]

    @property
    def is_compileable(self) -> bool:
        """Return whether the cache is compilable"""
        # For DynamicCache dispatching the layers lazily (otherwise, all([]) is True)
        if len(self.layers) == 0:
            return False
        return all(layer.is_compileable for layer in self.layers)

    @property
    def is_initialized(self) -> bool:
        """Return whether the cache data is initialized"""
        layers = [layer for layer in self.layers if layer.supports_early_init]
        return len(layers) > 0 and all(layer.is_initialized for layer in layers)

    @property
    def is_sliding(self) -> list[bool]:

View on GitHub (pinned to a597f97485)

Solutions

Ensure every tensor written into the cache (and every preloaded layer) shares one batch dimension
When concatenating caches, verify batch_size equality first and re-pad/copy as needed
After batch-changing ops (e.g. expand/transpose for beams), update all layers consistently via the provided cache ops rather than manual tensor surgery

Example fix

# before
big_cache = DynamicCache(layers=layer_a.layers + layer_b.layers)  # bs 2 + bs 3
print(big_cache.batch_size)  # ValueError

# after
assert layer_a.batch_size == layer_b.batch_size, "batch mismatch before concat"
big_cache = DynamicCache(layers=layer_a.layers + layer_b.layers)

Defensive patterns

Strategy: validation

Validate before calling

sizes = {l.batch_size for l in cache.layers if hasattr(l, "batch_size") and l.is_initialized}
assert len(sizes) <= 1, f"inconsistent batch sizes before use: {sizes}"
bs = cache.batch_size

Try / catch

try:
    bs = cache.batch_size
except ValueError as e:
    raise RuntimeError(f"refusing to continue with mixed-batch cache: {e}") from e

Prevention

Validate one shared batch dimension before writing any tensors into a cache
After beam/cache surgery, recompute all layers from the same source tensors instead of mixing
Treat -1 from batch_size as 'uninitialized' — do not concatenate caches until sizes are known

When it happens

Trigger: Concatenating or updating caches fed with tensors of different batch dimensions; batching ops (dp/ddp gather, batch concatenation) that merge per-device caches with different local batch sizes; updating one layer with a new batch size while other layers keep old states.

Common situations: Beam search or cache-reuse flows that manipulate batch dims; vLLM/sglang-style detach and concat of caches; mismatched batch size between prompt cache and continuation in speculative decoding.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/f119d260a4313160. Report an issue: GitHub.