{"record":{"id":"746af476e091bd40","repo":"huggingface/transformers","slug":"head-dim-was-provided-as-a-list-of-length-len-n","errorCode":null,"errorMessage":"`head_dim` was provided as a list of length {len(num_heads)}, but the Cache currently has {len(self.layers)} layers","messagePattern":"`head_dim` was provided as a list of length (.+?), but the Cache currently has (.+?) layers","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/cache_utils.py","lineNumber":1471,"sourceCode":"        dtype: torch.dtype,\n        device: torch.device,\n    ):\n        \"\"\"\n        Initialize all the layers in advance (it's otherwise lazily initialized on the first `update` call).\n        This is useful for our `export` recipes, as `export` needs everything in advance.\n        \"\"\"\n        # To allow different num_heads and head_dim depending on layers, we accept lists\n        if isinstance(num_heads, int):\n            num_heads = [num_heads] * len(self)\n        if isinstance(head_dim, int):\n            head_dim = [head_dim] * len(self)\n\n        if len(num_heads) != len(self.layers):\n            raise ValueError(\n                f\"`num_head` was provided as a list of length {len(num_heads)}, but the Cache currently has {len(self.layers)} layers\"\n            )\n        if len(head_dim) != len(self.layers):\n            raise ValueError(\n                f\"`head_dim` was provided as a list of length {len(num_heads)}, but the Cache currently has {len(self.layers)} layers\"\n            )\n\n        for layer, layer_num_heads, layer_head_dim in zip(self.layers, num_heads, head_dim):\n            if not layer.supports_early_init or layer.is_initialized:\n                continue\n            # Note that the initialization needs all dimensions (except -2), as well as device and dtype, so we use\n            # this fake tensor approach. It has size 0 on the -2 dimension, so it does not allocate any data (it only\n            # creates an empty tensor with correct shape, dtype and device), which is very efficient and practical\n            fake_kv_tensor = torch.zeros((batch_size, layer_num_heads, 0, layer_head_dim), dtype=dtype, device=device)\n            # Init the layer\n            layer.lazy_initialization(fake_kv_tensor, fake_kv_tensor)\n\n    def get_seq_length(self, layer_idx: int = 0) -> int:\n        \"\"\"Returns the sequence length of the cache for the given layer.\"\"\"\n        if layer_idx >= len(self.layers):\n            return 0\n","sourceCodeStart":1453,"sourceCodeEnd":1489,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/cache_utils.py#L1453-L1489","documentation":"Cache.early_initialization() raises ValueError when head_dim is a list whose length differs from len(self.layers). Like num_heads, per-layer head dims must match the layer count; ints broadcast. Note the f-string reports len(num_heads) rather than len(head_dim) — a formatting slip in the message, the actual check is on head_dim.","triggerScenarios":"Calling cache.early_initialization(...) with head_dim=[128, 128] on a cache with e.g. 28 layers. The message will misleadingly show the num_heads length; trust the condition, not the printed variable.","commonSituations":"Mixed-dimension architectures (varying head_dim per layer) where the head_dim list was built from a stale config; hitting the error after the num_heads check passed and misreading the message as a num_heads problem.","solutions":["Make len(head_dim) equal the number of cache layers, or pass a single int","If the message shows a num_heads length you believe is correct, the real mismatch is head_dim — the message interpolates the wrong variable","Derive both lists from the same config used to construct the cache"],"exampleFix":"# before\ncache.early_initialization(bs, num_heads=8, head_dim=[128, 128], ...)  # cache has 12 layers\n\n# after\ncache.early_initialization(bs, num_heads=8, head_dim=128, dtype=torch.bfloat16, device=device)","handlingStrategy":"validation","validationCode":"n_layers = len(cache.layers)\nif isinstance(head_dim, list):\n    assert len(head_dim) == n_layers, f\"head_dim list must have {n_layers} entries\"\ncache.early_initialization(batch_size, num_heads=num_heads, head_dim=head_dim, dtype=dtype, device=device)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["The error text interpolates len(num_heads) even though head_dim is what mismatched — verify both lists when you see it","Pass a single int head_dim unless the architecture truly varies head_dim per layer"],"tags":["cache","early-initialization","export","shape-mismatch","valueerror","misleading-message"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}