What causes "List offset is too large :/" in pola-rs/polars?

If input is untrusted, cap accepted payload size (Content-Length / read limits) before parsing

List offset is too large :/

Error message

List offset is too large :/

What it means

When JSON arrays are deserialized into an Arrow ListArray<i64>, offsets accumulate the running element count via Offsets::try_push. It can only fail when the total would exceed i64::MAX (~9.2e18) elements; the .expect() then aborts with this message. This is the Array branch (multi-element rows) of deserialize_list. In practice the panic indicates a polars offset-accounting bug or deliberately hostile input rather than a realistic payload.

Source

Thrown at crates/polars-json/src/json/deserialize.rs:189

    rows: &[A],
    dtype: ArrowDataType,
    allow_extra_fields_in_struct: bool,
) -> PolarsResult<ListArray<i64>> {
    let mut err_idx = rows.len();
    let child = ListArray::<i64>::get_child_type(&dtype);

    let mut validity = BitmapBuilder::with_capacity(rows.len());
    let mut offsets = Offsets::<i64>::with_capacity(rows.len());
    let mut inner = vec![];
    rows.iter()
        .enumerate()
        .for_each(|(i, row)| match row.borrow() {
            BorrowedValue::Array(value) => {
                inner.extend(value.iter());
                validity.push(true);
                offsets
                    .try_push(value.len())
                    .expect("List offset is too large :/");
            },
            BorrowedValue::Static(StaticNode::Null) => {
                validity.push(false);
                offsets.extend_constant(1)
            },
            value @ (BorrowedValue::Static(_) | BorrowedValue::String(_)) => {
                inner.push(value);
                validity.push(true);
                offsets.try_push(1).expect("List offset is too large :/");
            },
            _ => {
                err_idx = if err_idx == rows.len() { i } else { err_idx };
            },
        });

    check_err_idx(rows, err_idx, "list")?;

    let values = _deserialize(&inner, child.clone(), allow_extra_fields_in_struct)?;

View on GitHub (pinned to 68506541d2)

Solutions

If input is untrusted, cap accepted payload size (Content-Length / read limits) before parsing
Split absurdly large inputs into multiple reads so each stays far below 2^63 elements
If hit with plausible data, file a polars bug with the reproducing JSON - offset accounting should not overflow

Defensive patterns

Strategy: validation

Validate before calling

fn json_input_within_budget(path: &std::path::Path, max_bytes: u64) -> std::io::Result<()> {
    let len = std::fs::metadata(path)?.len();
    if len > max_bytes {
        return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, format!("json input {len} bytes exceeds budget {max_bytes}")));
    }
    Ok(())
}
// a budget of e.g. 2^40 bytes keeps cumulative list elements far below i64::MAX

Try / catch

catch_unwind(AssertUnwindSafe(|| read_json(path))) to reject the input as malformed/hostile - but treat any hit as a bug report candidate since realistic data cannot overflow i64 offsets.

Prevention

Impose input size limits at every API boundary that accepts JSON
Stream large NDJSON in bounded batches instead of one giant read
Fuzz with input budgets so overflow paths surface in CI, not production

When it happens

Trigger: A JSON/NDJSON input whose list column's cumulative inner element count across all rows exceeds i64::MAX - e.g. crafted or fuzzed JSON streams; real datasets cannot reach it.

Common situations: Fuzzing campaigns against read_json/scan_ndjson; adversarial-input test suites; essentially never in production data.

Related errors

AI-assisted analysis of pola-rs/polars@68506541d2 (2026-08-19). Data as JSON: /api/errors/a3da97cc3b379120. Report an issue: GitHub.