{"record":{"id":"a3da97cc3b379120","repo":"pola-rs/polars","slug":"list-offset-is-too-large","errorCode":null,"errorMessage":"List offset is too large :/","messagePattern":"List offset is too large :/","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/polars-json/src/json/deserialize.rs","lineNumber":189,"sourceCode":"    rows: &[A],\n    dtype: ArrowDataType,\n    allow_extra_fields_in_struct: bool,\n) -> PolarsResult<ListArray<i64>> {\n    let mut err_idx = rows.len();\n    let child = ListArray::<i64>::get_child_type(&dtype);\n\n    let mut validity = BitmapBuilder::with_capacity(rows.len());\n    let mut offsets = Offsets::<i64>::with_capacity(rows.len());\n    let mut inner = vec![];\n    rows.iter()\n        .enumerate()\n        .for_each(|(i, row)| match row.borrow() {\n            BorrowedValue::Array(value) => {\n                inner.extend(value.iter());\n                validity.push(true);\n                offsets\n                    .try_push(value.len())\n                    .expect(\"List offset is too large :/\");\n            },\n            BorrowedValue::Static(StaticNode::Null) => {\n                validity.push(false);\n                offsets.extend_constant(1)\n            },\n            value @ (BorrowedValue::Static(_) | BorrowedValue::String(_)) => {\n                inner.push(value);\n                validity.push(true);\n                offsets.try_push(1).expect(\"List offset is too large :/\");\n            },\n            _ => {\n                err_idx = if err_idx == rows.len() { i } else { err_idx };\n            },\n        });\n\n    check_err_idx(rows, err_idx, \"list\")?;\n\n    let values = _deserialize(&inner, child.clone(), allow_extra_fields_in_struct)?;","sourceCodeStart":171,"sourceCodeEnd":207,"githubUrl":"https://github.com/pola-rs/polars/blob/68506541d2de983056c9eb244e1ea05fab377dfc/crates/polars-json/src/json/deserialize.rs#L171-L207","documentation":"When JSON arrays are deserialized into an Arrow ListArray<i64>, offsets accumulate the running element count via Offsets::try_push. It can only fail when the total would exceed i64::MAX (~9.2e18) elements; the .expect() then aborts with this message. This is the Array branch (multi-element rows) of deserialize_list. In practice the panic indicates a polars offset-accounting bug or deliberately hostile input rather than a realistic payload.","triggerScenarios":"A JSON/NDJSON input whose list column's cumulative inner element count across all rows exceeds i64::MAX - e.g. crafted or fuzzed JSON streams; real datasets cannot reach it.","commonSituations":"Fuzzing campaigns against read_json/scan_ndjson; adversarial-input test suites; essentially never in production data.","solutions":["If input is untrusted, cap accepted payload size (Content-Length / read limits) before parsing","Split absurdly large inputs into multiple reads so each stays far below 2^63 elements","If hit with plausible data, file a polars bug with the reproducing JSON - offset accounting should not overflow"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"fn json_input_within_budget(path: &std::path::Path, max_bytes: u64) -> std::io::Result<()> {\n    let len = std::fs::metadata(path)?.len();\n    if len > max_bytes {\n        return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, format!(\"json input {len} bytes exceeds budget {max_bytes}\")));\n    }\n    Ok(())\n}\n// a budget of e.g. 2^40 bytes keeps cumulative list elements far below i64::MAX","typeGuard":null,"tryCatchPattern":"catch_unwind(AssertUnwindSafe(|| read_json(path))) to reject the input as malformed/hostile - but treat any hit as a bug report candidate since realistic data cannot overflow i64 offsets.","preventionTips":["Impose input size limits at every API boundary that accepts JSON","Stream large NDJSON in bounded batches instead of one giant read","Fuzz with input budgets so overflow paths surface in CI, not production"],"tags":["polars","json","deserialization","list","offset-overflow","panic","fuzzing"],"backgroundTag":"integer-overflow","analyzedSha":"68506541d2de983056c9eb244e1ea05fab377dfc","analyzedAt":"2026-08-19T12:15:06.350Z","contentChangedAt":"2026-08-19T12:15:06.350Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}