jax-ml/jax · error · ValueError

Expected 'key' {k.shape} and 'value' {v.shape} to have the s

Error message

Expected 'key' {k.shape} and 'value' {v.shape} to have the same leading dimensions.

What it means

The 'key' and 'value' tensors must share identical leading dimensions (everything except the final head-dim axis), because the kernel iterates KV blocks over the same grid for both. If k.shape[:-1] != v.shape[:-1] the kernel raises this ValueError.

Source

Thrown at jax/experimental/pallas/ops/tpu/splash_attention/splash_attention_kernel.py:962

    raise ValueError(
        f"Expected {expected_kv_rank}-dim 'key' tensor for MQA. Instead got a"
        f" {len(k.shape)}-dim one."
    )

  if k.shape[kv_head_dimension] != head_dim_qk:
    raise ValueError(
        f"Expected 'key' head dimension to be: {head_dim_qk}. Instead got:"
        f" {k.shape[kv_head_dimension]}."
    )

  if not is_mqa and num_q_heads % num_kv_heads != 0:
    raise ValueError(
        f"In MHA, expected number of 'key' heads ({num_kv_heads}) to be a"
        f" multiple of the number of 'query' heads ({num_q_heads})"
    )

  if k.shape[:-1] != v.shape[:-1]:
    raise ValueError(
        f"Expected 'key' {k.shape} and 'value' {v.shape} to have the same "
        "leading dimensions."
    )

  assert bkv_compute is not None
  if bkv % bkv_compute:
    raise ValueError(f"{bkv=} must be a multiple of {bkv_compute=}.")
  if bkv_compute % NUM_LANES:
    raise ValueError(f"{bkv_compute=} must be a multiple of {NUM_LANES}.")

  kv_seq_len = k.shape[kv_seq_len_dimension]

  q_heads_per_kv_head = num_q_heads // num_kv_heads

  if segment_ids is not None:
    if segment_ids.q.shape != (q_seq_len,):
      raise ValueError(
          "Invalid shape for q segment_ids: "

View on GitHub (pinned to 1e1c6a8fc0)

Solutions

Ensure num_kv_heads and kv_seq_len match between k and v (only the last dim may differ for head_dim_vo)
Re-slice the fused cache: k = kv[..., :kv_seq_len, :hd], v = kv[..., :kv_seq_len, hd:] carefully
Print k.shape[:-1] and v.shape[:-1] right before the call to confirm

Example fix

// before
k = cache[: , :, 0]  # [8, 1024, 128]
v = cache[:, :, 1]    # [8, 512, 128]
// after
k = cache[:, :seq, 0]
v = cache[:, :seq, 1]

Defensive patterns

Strategy: validation

Validate before calling

assert k.shape[:-1] == v.shape[:-1], f'{k.shape} vs {v.shape}'

Type guard

def kv_leading_dims_match(k, v) -> bool:
    return k.shape[:-1] == v.shape[:-1]

Prevention

Slice KV caches with a single shared index for k and v
Assert shapes before every attention call in debug builds

When it happens

Trigger: Passing k shaped [8, 1024, 128] and v shaped [8, 1024, 256] (different value head dim is allowed only via last axis but same leading dims required — here leading dims differ if head counts or seq lens differ), or k [8, 1024, 128] with v [8, 512, 128].

Common situations: Using models with separate KV sequence lengths (e.g. sliding-window caches); incorrectly slicing a fused KV cache; value head dim differences are fine only in the last axis, so mis-slicing produces leading-dim mismatches.

Related errors

AI-assisted analysis of jax-ml/jax@1e1c6a8fc0 (2026-08-27). Data as JSON: /api/errors/67a4f7b8ceee959f. Report an issue: GitHub.