screenpipe/screenpipe · error · Error::Inference

boxes cat

Error message

boxes cat

What it means

Raised when concatenating the final centers `final_cxcy` and sizes `final_wh` along the last axis to form the (B, 300, 4) output boxes. mlx `concatenate_axis` requires all inputs to have identical shapes on every axis except the concatenation axis. The library throws it when the two halves differ in batch, query count, or (if a prior split misbehaved) when the halves aren't each 2 wide.

Source

Thrown at crates/screenpipe-rfdetr-mlx/src/decoder/mod.rs:689

        // Final LN.
        output = ln(&output, &self.final_norm_w, &self.final_norm_b)?;

        // Final heads.
        let logits = linear(&output, &self.class_head_w, &self.class_head_b)?;
        let bbox_delta = self.bbox_head.forward(&output)?;
        // Final bbox refinement vs. refpoint_embed (cxcywh).
        let rb_parts = ops::split_sections(&refpoint_embed, &[2], -1).map_err(err("rb split"))?;
        let bd_parts = ops::split_sections(&bbox_delta, &[2], -1).map_err(err("bd split"))?;
        let final_cxcy = bd_parts[0]
            .multiply(&rb_parts[1])
            .map_err(err("final cxcy mul"))?
            .add(&rb_parts[0])
            .map_err(err("final cxcy add"))?;
        let final_wh = exp_(&bd_parts[1])?
            .multiply(&rb_parts[1])
            .map_err(err("final wh mul"))?;
        let boxes =
            ops::concatenate_axis(&[&final_cxcy, &final_wh], -1).map_err(err("boxes cat"))?;

        Ok((boxes, logits))
    }

    /// Gather slices of `x` along axis 1 by integer indices `idx (B, K)`.
    /// Result: `(B, K, last_dim)`.
    fn gather(&self, x: &Array, idx: &Array, last_dim: i32) -> Result<Array> {
        let s = x.shape();
        let (b, _len, _c) = (s[0], s[1], s[2]);
        let k = idx.shape()[1];
        // Broadcast idx (B, K) → (B, K, last_dim) so take_along_axis can gather.
        let idx_3d = idx.reshape(&[b, k, 1]).map_err(err("idx 3d"))?;
        let idx_bc = ops::broadcast_to(&idx_3d, &[b, k, last_dim]).map_err(err("idx bc"))?;
        x.take_along_axis(&idx_bc, 1).map_err(err("gather"))
    }
}

#[allow(dead_code)]

View on GitHub (pinned to 4ebf712990)

Solutions

1. Check `final_cxcy.shape()` and `final_wh.shape()` are identical except allowing last-axis concat; assert both are (B, 300, 2).
2. Fix the producing op (split/mul/add) rather than patching here — the mismatch originates earlier.
3. If one operand is (B, 300, 1, 2), reshape/squeeze to (B, 300, 2) before concatenating.
4. Compare against the ONNX exporter's expected (dets) output shape (B, 300, 4) as a sanity check.

Example fix

// before
let boxes = ops::concatenate_axis(&[&final_cxcy, &final_wh], -1).map_err(err("boxes cat"))?;
// after
assert_eq!(final_cxcy.shape(), final_wh.shape(), "cxcy/wh halves must match");
let final_wh = final_wh.reshape(final_cxcy.shape())?; // drop stray axis if any
let boxes = ops::concatenate_axis(&[&final_cxcy, &final_wh], -1).map_err(err("boxes cat"))?;

// Before concatenation
let c = final_cxcy.shape(); let w = final_wh.shape();
if c != w || c.last() != Some(&2) {
    return Err(anyhow!("cannot concat cxcy {c:?} with wh {w:?}"));
}

fn concat_halves_ok(cxcy: &mlx_rs::Array, wh: &mlx_rs::Array) -> bool {
    cxcy.shape() == wh.shape() && cxcy.shape().last() == Some(&2)
}

match decoder.forward(&enc, &refpoints) {
    Err(Error::Inference(m)) if m.contains("boxes cat") => {
        eprintln!("box halves not concatenable: {m}");
        // fall back to a previous known-good checkpoint or skip the frame
        fallback_decode(&enc)
    }
    other => other,
}

When it happens

Trigger: Downstream mismatch after the mul/add steps — e.g. final_cxcy ended up (B, 300, 2) but final_wh is (B, 1, 2) or (B, 299, 2) due to an upstream broadcast/slice bug, or an extra axis survived a reshape.

Common situations: Custom head modifications changing channel widths; partially failing earlier ops retried with different shapes; porting changes from the ONNX exporter where halves were computed with different slicing.

Related errors

• coord cat
• rb split
• final wh mul
• at least 2 chunks
• max class

AI-assisted analysis of screenpipe/screenpipe@4ebf712990 (2026-09-01). Data as JSON: /api/errors/b046c8ae67cf1e7e. Report an issue: GitHub.