{"record":{"id":"d6f0fc9030938e4a","repo":"huggingface/candle","slug":"causal-run-causal-attn-cpu-is-b-1-only-got-b-b","errorCode":null,"errorMessage":"causal::run_causal_attn_cpu is B=1 only (got B={b}). Multi-batch should be routed through the varlen path.","messagePattern":"causal::run_causal_attn_cpu is B=1 only \\(got B=(.+?)\\)\\. Multi-batch should be routed through the varlen path\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-nn/src/attention/cpu_flash/causal.rs","lineNumber":68,"sourceCode":"///\n/// Squeezes batch dim, extracts contiguous slices, dispatches to\n/// f32 or generic kernel. The inner kernels operate on raw slices only.\n#[allow(clippy::too_many_arguments)]\npub fn run_causal_attn_cpu<T>(\n    q: &Tensor,\n    k: &Tensor,\n    v: &Tensor,\n    softmax_scale: f32,\n    kv_offset: usize,\n    max_bias: Option<f32>,\n    softcap: Option<f32>,\n) -> Result<Tensor>\nwhere\n    T: WithDType + num_traits::Float,\n{\n    let b = q.dims()[0];\n    if b != 1 {\n        candle::bail!(\n            \"causal::run_causal_attn_cpu is B=1 only (got B={b}). \\\n             Multi-batch should be routed through the varlen path.\"\n        );\n    }\n\n    let q = q.squeeze(0)?.contiguous()?;\n    let k = k.squeeze(0)?.contiguous()?;\n    let v = v.squeeze(0)?.contiguous()?;\n\n    let (s_q, h_q, d) = q.dims3()?;\n    let (s_kv, h_kv, _) = k.dims3()?;\n    let (_, h_v, _) = v.dims3()?;\n\n    let max_bias = max_bias.unwrap_or(0.0);\n    let softcap = softcap.unwrap_or(0.0);\n\n    if q.dtype() == DType::F32 {\n        let (q_g, q_l) = q.storage_and_layout();","sourceCodeStart":50,"sourceCodeEnd":86,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-nn/src/attention/cpu_flash/causal.rs#L50-L86","documentation":"The CPU causal attention kernel run_causal_attn_cpu only supports a batch size of 1; it squeezes the batch dim immediately after the check. Multi-batch inputs must go through the variable-length (varlen) attention path instead.","triggerScenarios":"Calling run_causal_attn_cpu (or the CPU causal flash-attention wrapper) with a q tensor whose dims()[0] > 1.","commonSituations":"Batched inference on CPU with batch>1 and the code path selecting the B=1 causal kernel; refactoring code that previously ran single sequences; tests with stacked examples.","solutions":["Route batched inputs through the varlen attention path instead","Split the batch and call run_causal_attn_cpu per element with b=1","Squeeze to a single sequence if the input truly has one example (fix upstream shape)"],"exampleFix":"// before\nlet y = causal::run_causal_attn_cpu(&q, &k, &v, &mask)?; // q dims (4, s, h, d)\n// after\nlet y = varlen::run_varlen_attn_cpu(&q, &k, &v, &cu_seqlens)?; // multi-batch path","handlingStrategy":"validation","validationCode":"if q.dim(0)? != 1 {\n    return Err(anyhow!(\"use the varlen path for batch>1\"));\n}","typeGuard":"fn is_single_batch(q: &Tensor) -> bool { q.dim(0).map(|b| b == 1).unwrap_or(false) }","tryCatchPattern":"match result {\n    Err(e) if e.to_string().contains(\"B=1 only\") => {\n        // route to varlen::run_varlen_attn_cpu instead\n    }\n    other => other?,\n}","preventionTips":["Dispatch to the varlen path whenever batch > 1 at the call site","Keep batch handling in one dispatcher function","Add a batch-size assert in tests covering the CPU attention path"],"tags":["candle-nn","cpu","attention","batch-size","varlen"],"backgroundTag":"unsupported-batch-size","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}