pola-rs/polars · error

Offset to fit in `usize`

Error message

Offset to fit in `usize`

What it means

`buffer_offset` in crates/polars-arrow/src/ffi/array.rs computes the byte offset into buffer index 1 for FixedSizeBinary arrays: `array.offset.try_into().expect("Offset to fit in `usize`") * size`. Since `array.offset` is i64, a negative value (or one too large for usize on 32-bit) panics here before the multiplication with the fixed element width.

Source

Thrown at crates/polars-arrow/src/ffi/array.rs:358

    let storage = SharedStorage::from_slice_with_owner(slice, owner);

    let null_count = if is_validity {
        Some(array.null_count())
    } else {
        None
    };
    Ok(Bitmap::from_inner_unchecked(
        storage, offset, len, null_count,
    ))
}

fn buffer_offset(array: &ArrowArray, dtype: &ArrowDataType, i: usize) -> usize {
    use PhysicalType::*;
    match (dtype.to_physical_type(), i) {
        (LargeUtf8, 2) | (LargeBinary, 2) | (Utf8, 2) | (Binary, 2) => 0,
        (FixedSizeBinary, 1) => {
            if let ArrowDataType::FixedSizeBinary(size) = dtype.to_storage() {
                let offset: usize = array.offset.try_into().expect("Offset to fit in `usize`");
                offset * *size
            } else {
                unreachable!()
            }
        },
        _ => array.offset.try_into().expect("Offset to fit in `usize`"),
    }
}

/// Returns the length, in slots, of the buffer `i` (indexed according to the C data interface)
unsafe fn buffer_len(array: &ArrowArray, dtype: &ArrowDataType, i: usize) -> PolarsResult<usize> {
    Ok(match (dtype.to_physical_type(), i) {
        (PhysicalType::FixedSizeBinary, 1) => {
            if let ArrowDataType::FixedSizeBinary(size) = dtype.to_storage() {
                *size * (array.offset as usize + array.length as usize)
            } else {
                unreachable!()
            }

View on GitHub (pinned to 9b5d73fd00)

Solutions

1. Fix the producer's offset computation for sliced FixedSizeBinary exports (offset must be the element-slot count, non-negative).
2. Validate/zero-init the whole ArrowArray struct on export and set fields explicitly.
3. Wrap `from_ffi` in catch_unwind at the boundary to convert the panic into an error you can attribute to the producer.
4. Reproduce with a known-good producer (pyarrow) to confirm the bug is on the foreign side.

Example fix

// before
let array = unsafe { from_ffi(fsb_imported, owner) }?; // panics in buffer_offset(FixedSizeBinary, 1)

// after
let array = std::panic::catch_unwind(|| unsafe { from_ffi(fsb_imported, owner) })
    .map_err(|_| polars_err!(ComputeError: "FixedSizeBinary export has invalid offset"))??;

// Producer-side: for FixedSizeBinary(size) exports, offset must be a non-negative
// element-slot count: assert(offset >= 0 && offset * size + length * size <= buffer_len).

let array = std::panic::catch_unwind(|| unsafe { polars_arrow::ffi::from_ffi(fsb, owner) })
    .map_err(|_| polars_err!(ComputeError: "FixedSizeBinary FFI offset invalid"))?;

When it happens

Trigger: Importing a FixedSizeBinary array over the C Data Interface whose ArrowArray.offset is negative or corrupt. The FixedSizeBinary arm is hit because its values buffer starts at byte `offset * size`, unlike types where the offset only affects bitmap indexing.

Common situations: FFI producers exporting sliced fixed-size-binary columns (e.g. 16-byte UUID/hash arrays) with a buggy slice calculation; uninitialized struct fields; struct-layout mismatch between the producer's Arrow version and polars-arrow's generated bindings (ffi/generated.rs).

Related errors

• offset to fit in `usize`
• length to fit in `usize`
• The external API has a non-utf8 as format
• not implemented
• The schema declaration does not match the deserialization

AI-assisted analysis of pola-rs/polars@9b5d73fd00 (2026-08-19). Data as JSON: /api/errors/06bfc4fea58d1575. Report an issue: GitHub.