huggingface/transformers · error · ValueError

Block size must be at least {}, but got {}

Error message

Block size must be at least {}, but got {}

What it means

ValueError raised while building the paged KV cache: continuous_batching_config.block_size must be at least Cache._min_block_size. The default was raised from 32 to 256 tokens to stay compatible with flash_attn_with_kvcache; smaller blocks break flash-attention's paged KV kernel constraints.

Source

Thrown at src/transformers/generation/continuous_batching/cache.py:179

            config: Model configuration
            continuous_batching_config: Continuous batching configuration containing cache parameters
            device: Device for the cache tensors
            distributed_helper: TP-aware helper. Used to dispatch attention heads and ensure coherent cache size
            tp_plan: Tensor parallelism plan
            dtype: Data type of the activation and the cache (for now, these are the same)
        """
        self.config = config
        self.dtype = dtype
        self.device = device

        # Extract model dimensions
        self.num_key_value_heads: int = find_num_kv_heads(config)
        self.head_dim: int = find_head_dim(config)

        # Extract cache dimensions. Default used to be 32, now it's 256 to be compatible with flash_with_kvcache.
        self.block_size = continuous_batching_config.block_size
        if self.block_size < self._min_block_size:
            raise ValueError(f"Block size must be at least {self._min_block_size}, but got {self.block_size}")

        # Group layers depending on the attention mix
        layer_groups, group_types = group_layers_by_attn_type(config)
        group_size = len(layer_groups[0])
        self.num_groups = len(layer_groups)

        self.sliding_windows = {}
        self.layer_index_to_group_indices = {}
        for i, group in enumerate(layer_groups):
            sliding_window = config.sliding_window if group_types[i] == "sliding_attention" else 1
            for j, layer in enumerate(group):
                self.layer_index_to_group_indices[layer] = (i, j)
                self.sliding_windows[layer] = sliding_window

        # Check if the KV heads are part of the TP plan. If they are not, the cache does not need plan for TP.
        # TODO: this is fragile. If your model fails to TP properly because of this, please open an issue.
        kv_is_tp = True
        for key in ["layers.*.self_attn.k_proj", "layers.*.self_attn.v_proj"]:

View on GitHub (pinned to a597f97485)

Solutions

1. Use block_size >= the class's minimum (check Cache._min_block_size; historically 256 for flash-attn compatibility)
2. Or omit block_size to accept the library default
3. If you truly need small blocks, you cannot with this path — use a different cache implementation

Example fix

# before
cb_cfg = ContinuousBatchingConfig(block_size=16)
# after
cb_cfg = ContinuousBatchingConfig(block_size=256)

from transformers.generation.continuous_batching.cache import Cache
cb_cfg.block_size = max(cb_cfg.block_size or 0, Cache._min_block_size)

def valid_block_size(bs: int) -> bool:
    from transformers.generation.continuous_batching.cache import Cache
    return bs >= Cache._min_block_size

When it happens

Trigger: Constructing the continuous-batching cache with ContinuousBatchingConfig(block_size=16 or 32); copying vLLM-style block sizes (16) into transformers' implementation; stale configs from before the 256 default.

Common situations: Porting vLLM tuning guides (which use block_size 16) to transformers continuous batching; trying to reduce memory fragmentation with small blocks.

Related errors

• flash_attn_with_kvcache_fn does not have a block_table or pa
• num_key_value_heads or num_attention_heads could not be foun
• head_dim or (hidden_size and num_attention_heads) could not
• Number of key value heads {} must be divisible by tensor par
• Invalid group type: {}

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/c44308ad33b260bd. Report an issue: GitHub.