{"record":{"id":"f119d260a4313160","repo":"huggingface/transformers","slug":"the-batch-size-is-not-consistent-across-layers-v","errorCode":null,"errorMessage":"The batch size is not consistent across layers: {values}","messagePattern":"The batch size is not consistent across layers: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/cache_utils.py","lineNumber":1641,"sourceCode":"        \"\"\"\n        Calling this function will activate past state recording, meaning that cache with fixed size such as a linear cache will\n        wait for a call to `crop` before restricting the size of its cached states, in order to be able to retrieve previous full states.\n        \"\"\"\n        for layer_idx in range(len(self.layers)):\n            if hasattr(self.layers[layer_idx], \"activate_past_recording\"):\n                self.layers[layer_idx].activate_past_recording()\n\n    @property\n    def batch_size(self) -> int:\n        \"\"\"Return the batch size of the cache, or ``-1`` if no layer has been initialized yet\n        (e.g. an all-linear-attention cache queried before the first forward).\"\"\"\n        # ``LinearAttentionLayer`` sets ``batch_size`` lazily — skip layers that haven't been\n        # initialized yet (``generate`` queries this on a fresh cache during cache-reuse checks).\n        values = [layer.batch_size for layer in self.layers if hasattr(layer, \"batch_size\")]\n        if not values:\n            return -1\n        if len(set(values)) > 1:\n            raise ValueError(f\"The batch size is not consistent across layers: {values}\")\n        return values[0]\n\n    @property\n    def is_compileable(self) -> bool:\n        \"\"\"Return whether the cache is compilable\"\"\"\n        # For DynamicCache dispatching the layers lazily (otherwise, all([]) is True)\n        if len(self.layers) == 0:\n            return False\n        return all(layer.is_compileable for layer in self.layers)\n\n    @property\n    def is_initialized(self) -> bool:\n        \"\"\"Return whether the cache data is initialized\"\"\"\n        layers = [layer for layer in self.layers if layer.supports_early_init]\n        return len(layers) > 0 and all(layer.is_initialized for layer in layers)\n\n    @property\n    def is_sliding(self) -> list[bool]:","sourceCodeStart":1623,"sourceCodeEnd":1659,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/cache_utils.py#L1623-L1659","documentation":"Cache.batch_size property raises ValueError when initialized layers report different batch sizes. It collects layer.batch_size across layers that have the attribute; more than one distinct value means the cache holds mutually inconsistent states (usually from mismatched inputs or wrongly concatenated caches).","triggerScenarios":"Concatenating or updating caches fed with tensors of different batch dimensions; batching ops (dp/ddp gather, batch concatenation) that merge per-device caches with different local batch sizes; updating one layer with a new batch size while other layers keep old states.","commonSituations":"Beam search or cache-reuse flows that manipulate batch dims; vLLM/sglang-style detach and concat of caches; mismatched batch size between prompt cache and continuation in speculative decoding.","solutions":["Ensure every tensor written into the cache (and every preloaded layer) shares one batch dimension","When concatenating caches, verify batch_size equality first and re-pad/copy as needed","After batch-changing ops (e.g. expand/transpose for beams), update all layers consistently via the provided cache ops rather than manual tensor surgery"],"exampleFix":"# before\nbig_cache = DynamicCache(layers=layer_a.layers + layer_b.layers)  # bs 2 + bs 3\nprint(big_cache.batch_size)  # ValueError\n\n# after\nassert layer_a.batch_size == layer_b.batch_size, \"batch mismatch before concat\"\nbig_cache = DynamicCache(layers=layer_a.layers + layer_b.layers)","handlingStrategy":"validation","validationCode":"sizes = {l.batch_size for l in cache.layers if hasattr(l, \"batch_size\") and l.is_initialized}\nassert len(sizes) <= 1, f\"inconsistent batch sizes before use: {sizes}\"\nbs = cache.batch_size","typeGuard":null,"tryCatchPattern":"try:\n    bs = cache.batch_size\nexcept ValueError as e:\n    raise RuntimeError(f\"refusing to continue with mixed-batch cache: {e}\") from e","preventionTips":["Validate one shared batch dimension before writing any tensors into a cache","After beam/cache surgery, recompute all layers from the same source tensors instead of mixing","Treat -1 from batch_size as 'uninitialized' — do not concatenate caches until sizes are known"],"tags":["cache","batch-size","consistency","concat","valueerror"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}