{"record":{"id":"fde18cb8d5a93dc3","repo":"huggingface/candle","slug":"flash-attn-v3-varlen-expects-input-tensors-of-rank","errorCode":null,"errorMessage":"flash-attn-v3-varlen expects input tensors of rank 3 (q: {q_rank}, k: {k_rank}, v: {v_rank}","messagePattern":"flash-attn-v3-varlen expects input tensors of rank 3 \\(q: (.+?), k: (.+?), v: (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-flash-attn-v3/src/lib.rs","lineNumber":499,"sourceCode":"        let q = q.as_cuda_slice::<T>()?;\n        let k = k.as_cuda_slice::<T>()?;\n        let v = v.as_cuda_slice::<T>()?;\n        let q = q.slice(q_l.start_offset()..);\n        let k = k.slice(k_l.start_offset()..);\n        let v = v.slice(v_l.start_offset()..);\n\n        let q_stride = q_l.stride();\n        let k_stride = k_l.stride();\n        let v_stride = v_l.stride();\n        let o_stride = out_l.stride();\n\n        let q_rank = q_stride.len();\n        let k_rank = k_stride.len();\n        let v_rank = v_stride.len();\n        let o_rank = o_stride.len();\n\n        if q_rank != 3 || k_rank != 3 || v_rank != 3 {\n            candle::bail!(\n                \"flash-attn-v3-varlen expects input tensors of rank 3 (q: {q_rank}, k: {k_rank}, v: {v_rank}\"\n            )\n        }\n        if q_stride[q_rank - 1] != 1 {\n            candle::bail!(\"the last dim of q must be contiguous {q_stride:?}\")\n        }\n        if k_stride[k_rank - 1] != 1 {\n            candle::bail!(\"the last dim of k must be contiguous {k_stride:?}\")\n        }\n        if v_stride[v_rank - 1] != 1 {\n            candle::bail!(\"the last dim of v must be contiguous {v_stride:?}\")\n        }\n\n        let (total_q, num_heads, head_size_og) = q_l.shape().dims3()?;\n        let (total_k, num_heads_k, _head_size_og) = k_l.shape().dims3()?;\n        let expected_kv = (total_k, num_heads_k, head_size_og);\n        if expected_kv != k_l.shape().dims3()? {\n            candle::bail!(\"shape mismatch q {:?} and k {:?}\", q_l.shape(), k_l.shape())","sourceCodeStart":481,"sourceCodeEnd":517,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-flash-attn-v3/src/lib.rs#L481-L517","documentation":"The varlen kernel operates on packed (total_tokens, num_heads, head_dim) tensors. The wrapper reads q/k/v stride lengths to get each rank and bails unless all three are exactly rank 3. Passing rank-2 (missing heads axis) or rank-4 (batched, non-varlen) tensors triggers this.","triggerScenarios":"Calling flash-attn-v3 varlen forward with q/k/v shaped (B, S, H, D) or (S, D) — e.g. reusing the batched (non-varlen) flash-attn input shapes or feeding projections without the heads axis.","commonSituations":"Migrating from the regular batched flash-attn API to the varlen API without reshaping; forgetting to merge batch and seq into a total-token axis; (tokens, heads*dim) tensors not split into two axes.","solutions":["Reshape inputs to (total_q, num_heads, head_dim) / (total_k, num_heads_k, head_dim): for (B,S,H,D) use q.reshape((b*s, h, d)).","If the tensor is (tokens, num_heads*head_dim), reshape the last dim into (heads, head_dim).","Check q.rank() == 3 (etc.) at the call site and reshape otherwise."],"exampleFix":"// before\nlet q = q.reshape((b * s, h * d))?; // rank 2\n// after\nlet q = q.reshape((b * s, h, d))?; // (total, heads, head_dim)","handlingStrategy":"validation","validationCode":"if q.rank() != 3 || k.rank() != 3 || v.rank() != 3 {\n    candle_core::bail!(\"varlen inputs must be (total, heads, head_dim); got q rank {}\", q.rank());\n}\n// (B,S,H,D) -> (B*S,H,D)\nlet q = q.reshape((b * s, h, d))?;","typeGuard":"fn is_rank3(t: &candle_core::Tensor) -> bool { t.rank() == 3 }","tryCatchPattern":"match forward_varlen(&q, &k, &v, ...) {\n    Err(e) if e.to_string().contains(\"rank 3\") => {\n        let q = flatten_batch_seq(&q)?;\n        let k = flatten_batch_seq(&k)?;\n        let v = flatten_batch_seq(&v)?;\n        forward_varlen(&q, &k, &v, ...)\n    }\n    other => other,\n}","preventionTips":["Remember varlen inputs are packed: merge batch and seq axes into a total-token axis, keeping (tokens, heads, head_dim).","Check rank == 3 in a shared wrapper around every flash-attn call.","Document the expected packed input shape at the attention module's public API."],"tags":["shape","rank-mismatch","flash-attention","varlen"],"backgroundTag":"tensor-rank-mismatch","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}