huggingface/transformers · error · ValueError

Tensor query has shape with a zero dimension. FlashAttentio

Error message

Tensor query has shape  with a zero dimension.
FlashAttention does not support inputs with dim=0.
Please check your input shapes or use SDPA instead.

What it means

The FlashAttention-2 integration (_flash_attention_forward) checks every dimension of the query tensor before transposing and calling into flash_attn. flash-attn kernels cannot operate on tensors with a 0-sized dimension (e.g. batch=0 or seq_len=0), so transformers raises ValueError up front with a suggestion to use SDPA, which tolerates empty batches.

Source

Thrown at src/transformers/integrations/flash_attention.py:50

    dropout: float = 0.0,
    scaling: float | None = None,
    sliding_window: int | None = None,
    softcap: float | None = None,
    is_causal: bool | None = None,
    s_aux: torch.Tensor | None = None,  # alias: learnable attention sink
    **kwargs,
) -> tuple[torch.Tensor, None]:
    if kwargs.get("output_attentions", False):
        logger.warning_once(
            "Flash Attention does not support `output_attentions=True`."
            " Please set your attention to `eager` if you want any of these features."
        )

    # This is before the transpose
    seq_len = query.shape[2]

    if any(dim == 0 for dim in query.shape):
        raise ValueError(
            "Tensor query has shape  with a zero dimension.\n"
            "FlashAttention does not support inputs with dim=0.\n"
            "Please check your input shapes or use SDPA instead."
        )
    # FA2 uses non-transposed inputs
    query = query.transpose(1, 2)
    key = key.transpose(1, 2)
    value = value.transpose(1, 2)

    # FlashAttention requires the query and value to share a head dim; pad `value` up to the
    # query head dim (e.g. MLA, where `v_head_dim < qk_head_dim`) and crop the output below.
    head_dim, v_head_dim = query.shape[-1], value.shape[-1]
    if v_head_dim != head_dim:
        value = torch.nn.functional.pad(value, [0, head_dim - v_head_dim])

    # In PEFT, usually we cast the layer norms in float32 for training stability reasons
    # therefore the input hidden states gets silently casted in float32. Hence, we need
    # cast them back in the correct dtype just to be sure everything works as expected.

View on GitHub (pinned to a597f97485)

Solutions

  1. Guard the forward call: skip or short-circuit when query/input batch or sequence length is 0
  2. Filter empty sequences out of the batch before the model call
  3. Switch attention to "sdpa" (model.config._attn_implementation or attn_implementation kwarg) if empty batches must flow through

Example fix

# before
out = model(input_ids)  # input_ids.shape == (0, 128) with flash_attention_2
# ValueError: Tensor query has shape with a zero dimension

# after
if input_ids.shape[0] == 0 or input_ids.shape[-1] == 0:
    out = None  # or skip batch
else:
    out = model(input_ids)
Defensive patterns

Strategy: type-guard

Validate before calling

def has_zero_dim(t):
    return any(d == 0 for d in t.shape)

if not has_zero_dim(query) and not has_zero_dim(key):
    out = model(input_ids)  # flash_attention_2 safe

Type guard

def is_fa2_safe(batch) -> bool:
    """True when no tensor dimension is 0 (FlashAttention requirement)."""
    return all(all(d != 0 for d in t.shape) for t in batch.values() if hasattr(t, "shape"))

Prevention

When it happens

Trigger: Calling a model with attn_implementation="flash_attention_2" where the input produces a zero-dim query: batch_size=0, an empty padded batch, seq_length=0, or num_key_value_heads=0 due to config errors.

Common situations: DataLoader tail batches that are empty; speculative/lookahead decoding paths that occasionally build 0-length sequences; tests that pass dummy empty tensors; feeding preprocessed input_ids of shape [0, N].

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/9ed4793bb0ecbd5f. Report an issue: GitHub.