pola-rs/polars · error

length to fit in `usize`

Error message

length to fit in `usize`

What it means

When importing an array over the Arrow C Data Interface (crates/polars-arrow/src/ffi/array.rs), `ArrowArray.length` is an `i64` per the spec. `create_bitmap` does `array.length.try_into().expect("length to fit in `usize`")`. On 64-bit targets this only fails for negative lengths; on 32-bit targets also for lengths above u32::MAX. A negative/garbage length from the foreign producer is treated as an unrecoverable invariant violation and panics.

Source

Thrown at crates/polars-arrow/src/ffi/array.rs:328

        Ok(Buffer::from(v))
    }
}

/// returns the buffer `i` of `array` interpreted as a [`Bitmap`].
/// # Safety
/// This function is safe iff:
/// * the buffer at position `index` is valid for the declared length
/// * the buffers' pointer is not mutable for the lifetime of `owner`
unsafe fn create_bitmap(
    array: &ArrowArray,
    dtype: &ArrowDataType,
    owner: InternalArrowArray,
    index: usize,
    // if this is the validity bitmap
    // we can use the null count directly
    is_validity: bool,
) -> PolarsResult<Bitmap> {
    let len: usize = array.length.try_into().expect("length to fit in `usize`");
    if len == 0 {
        // Zero-length arrays might have invalid pointers for zero-length slices in Rust,
        // so this is more than just an optimization.
        return Ok(Bitmap::new());
    }
    let ptr = get_buffer_ptr(array, dtype, index)?;

    // Pointer of u8 has alignment 1, so we don't have to check alignment.
    let offset: usize = array.offset.try_into().expect("offset to fit in `usize`");
    let bytes_len = bytes_for(offset + len);
    let slice = core::slice::from_raw_parts(ptr, bytes_len);
    let storage = SharedStorage::from_slice_with_owner(slice, owner);

    let null_count = if is_validity {
        Some(array.null_count())
    } else {
        None
    };

View on GitHub (pinned to 9b5d73fd00)

Solutions

  1. Fix the producer: initialize every ArrowArray field per the C data interface spec (length >= 0, sane offset, release callback set).
  2. Validate the struct in the producer/exporter before handing it to Rust.
  3. Wrap `from_ffi` in `std::panic::catch_unwind` at the interop boundary and report which batch failed instead of crashing the process.
  4. If on a 32-bit target with legitimately huge arrays, move to 64-bit — lengths that don't fit usize can't be handled at all.

Example fix

// before (Rust consumer)
let array = unsafe { from_ffi(imported, owner) }?; // panics if producer sent negative length

// after
let result = std::panic::catch_unwind(|| unsafe { from_ffi(imported, owner) });
let array = result.map_err(|_| polars_err!(ComputeError: "foreign producer exported an ArrowArray with invalid length"))??;
Defensive patterns

Strategy: try-catch

Validate before calling

// ArrowArray fields are pub(super); validate on the producer side before export:
// C/other-language producer: assert(array->length >= 0 && array->offset >= 0);
// and (on 32-bit) assert((uint64_t)array->length <= UINT32_MAX);

Try / catch

let array = std::panic::catch_unwind(|| unsafe { polars_arrow::ffi::from_ffi(imported, owner) })
    .map_err(|_| polars_err!(ComputeError: "foreign ArrowArray has invalid length; check producer"))?;

Prevention

When it happens

Trigger: `from_ffi`/`try_from_ffi` on an `ArrowArray` whose `length` field is negative (uninitialized or corrupted C struct) or, on 32-bit builds, larger than usize. The panic fires while building the validity/offset bitmap for any array with non-zero length, including validity-only paths.

Common situations: FFI interop with C/C++/Golang/Java producers: passing an ArrowArray struct that wasn't fully initialized (e.g. zeroing only some fields, or exporting via a stale/mismatched ABI version), or memory corruption/missing owner lifetime on the producer side. Rare in correct pyarrow-based flows, more common in hand-written C bridges.

Related errors

AI-assisted analysis of pola-rs/polars@9b5d73fd00 (2026-08-19). Data as JSON: /api/errors/63795a777278ec44. Report an issue: GitHub.