keras-team/keras · error · ValueError

The last dimension of `query_shape` and `value_shape` must b

Error message

The last dimension of `query_shape` and `value_shape` must be equal, but are {query_shape[-1]}, {value_shape[-1]}. Received: query_shape={query_shape}, value_shape={value_shape}

What it means

In GroupedQueryAttention.compute_output_shape, the model dimension of the query must equal that of the value (query_shape[-1] == value_shape[-1]); otherwise output projection shapes cannot be computed and a ValueError with both shapes is raised. The check fires during shape inference (called via compute_output_spec), typically at build time, first call, or when the layer is used inside a Functional model.

Source

Thrown at keras/src/layers/attention/grouped_query_attention.py:577

            # key_attention_dims>)
            mask_expansion_axis = -1 * 2 - 1
            for _ in range(len(scores.shape) - len(attention_mask.shape)):
                attention_mask = ops.expand_dims(
                    attention_mask, axis=mask_expansion_axis
                )
        return self._softmax(scores, mask=attention_mask)

    def compute_output_shape(
        self,
        query_shape,
        value_shape,
        key_shape=None,
    ):
        if key_shape is None:
            key_shape = value_shape

        if query_shape[-1] != value_shape[-1]:
            raise ValueError(
                "The last dimension of `query_shape` and `value_shape` "
                f"must be equal, but are {query_shape[-1]}, {value_shape[-1]}. "
                f"Received: query_shape={query_shape}, "
                f"value_shape={value_shape}"
            )

        if value_shape[1:-1] != key_shape[1:-1]:
            raise ValueError(
                "All dimensions of `value` and `key`, except the last one, "
                f"must be equal. Received: value_shape={value_shape} and "
                f"key_shape={key_shape}"
            )

        return query_shape

    def compute_output_spec(
        self,
        query,

View on GitHub (pinned to 7a34a03db6)

Solutions

Project the query (or value) to a common width first, e.g. Dense(model_dim) on the smaller side before the attention layer.
Fix the inputs so query_shape[-1] equals value_shape[-1]; both encode the same model dimension.
Check for swapped or misspelled shape tuples when constructing the layer programmatically.

Example fix

# before
q = keras.Input((10, 128)); v = keras.Input((10, 256))
out = GroupedQueryAttention(head_dim=64)(q, v, v)  # -> ValueError: 128 vs 256

# after
proj = keras.layers.Dense(256)
out = GroupedQueryAttention(head_dim=64)(proj(q), v, v)

Defensive patterns

Strategy: validation

Validate before calling

q, v = list(query_shape), list(value_shape)
assert q[-1] == v[-1], f'query/value model dims differ: {q[-1]} vs {v[-1]}'

Type guard

def dims_compatible(q_shape, v_shape) -> bool:
    q, v = list(q_shape), list(v_shape)
    return len(q) == len(v) and q[-1] == v[-1]

Prevention

Keep one model_dim constant and use it for all Q/K/V producers.
Insert Dense(model_dim) projections before cross-attention between different-width models.
Assert shape contracts during config validation, before model build.

When it happens

Trigger: Passing query of width 128 and value of width 256, e.g. q=Input((10,128)) and v=Input((10,256)) into GroupedQueryAttention; cross-attention where key/value come from an encoder with a different model dim and no projection to match it.

Common situations: Cross-attention between models of different widths (e.g. querying a 768-dim encoder with a 512-dim decoder) without a projection; config mistakes where query_shape or value_shape tuples are swapped or a dim is wrong.

Related errors

AI-assisted analysis of keras-team/keras@7a34a03db6 (2026-08-25). Data as JSON: /api/errors/f7c8109c49485778. Report an issue: GitHub.