{"record":{"id":"c5236c812b581146","repo":"pola-rs/polars","slug":"unexpected-dtype-when-deserializing-ndjson","errorCode":null,"errorMessage":"unexpected dtype when deserializing ndjson","messagePattern":"unexpected dtype when deserializing ndjson","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/polars-io/src/ndjson/buffer.rs","lineNumber":138,"sourceCode":"                    TimeUnit::Microseconds, // ignored\n                )?;\n                buf.append_option(v);\n                Ok(())\n            },\n            All(dtype, buf) => {\n                let av = deserialize_all(value, dtype, self.ignore_errors)?;\n                buf.push(av);\n                Ok(())\n            },\n            Null(builder) => {\n                if !(matches!(value, Value::Static(StaticNode::Null)) || self.ignore_errors) {\n                    polars_bail!(ComputeError: \"got non-null value for NULL-typed column: {}\", value)\n                };\n\n                builder.append_null();\n                Ok(())\n            },\n            _ => panic!(\"unexpected dtype when deserializing ndjson\"),\n        }\n    }\n\n    pub fn add_null(&mut self) {\n        self.buf.add(AnyValue::Null).expect(\"should not fail\");\n    }\n}\npub(crate) fn init_buffers(\n    schema: &Schema,\n    capacity: usize,\n    ignore_errors: bool,\n) -> PolarsResult<PlIndexMap<BufferKey<'_>, Buffer<'_>>> {\n    schema\n        .iter()\n        .map(|(name, dtype)| {\n            let av_buf = (dtype, capacity).into();\n            let key = KnownKey::from(name.as_str());\n            Ok((","sourceCodeStart":120,"sourceCodeEnd":156,"githubUrl":"https://github.com/pola-rs/polars/blob/9b5d73fd00236295624374b075d16b1fe6ec6df9/crates/polars-io/src/ndjson/buffer.rs#L120-L156","documentation":"In NDJSON reading, each column's values are appended into an AnyValueBuffer, and the match in Buffer::add only handles Boolean, Int32/Int64, UInt32/UInt64, Float32/64, String, Datetime, Date, the recursive All bucket and Null. The AnyValueBuffer construction, however, also produces Int8, Int16, UInt8, UInt16, Duration and Time variants when the corresponding dtype features are on - those fall into the catch-all panic on the first non-null value.","triggerScenarios":"scan_ndjson/read_ndjson with an explicit schema (or inferred dtype) containing Int8, Int16, UInt8, UInt16, Duration or Time columns - the buffer variants exist only when the dtype-i8/-i16/-u8/-u16/-duration/-time features are enabled - and a row where that field is non-null.","commonSituations":"Reusing a parquet/Arrow-derived schema for NDJSON ingestion; pushing narrow integer types through a pipeline; hand-authored schemas with Time or Duration columns.","solutions":["Cast those columns to a supported dtype in the schema you pass: Int8/Int16 -> Int32 or Int64, UInt8/UInt16 -> UInt32/UInt64, Duration -> Int64, Time -> Int64 (or String)","Or omit the problem fields from the ndjson schema and cast after the read","Track upstream: extend the match in polars-io/src/ndjson/buffer.rs to cover these buffer variants"],"exampleFix":"// before\nlet schema = Schema::from_iter(vec![\n    (PlSmallStr::from(\"delta\"), DataType::Duration(TimeUnit::Milliseconds)), // panics on first row\n]);\n\n// after\nlet schema = Schema::from_iter(vec![\n    (PlSmallStr::from(\"delta\"), DataType::Int64),  // read raw, .cast(Duration) afterwards\n]);","handlingStrategy":"validation","validationCode":"use polars_core::prelude::*;\nfn ndjson_schema_supported(schema: &Schema) -> Result<(), String> {\n    const BAD: &[DataType] = &[\n        DataType::Int8, DataType::Int16, DataType::UInt8, DataType::UInt16,\n        DataType::Duration(TimeUnit::Nanoseconds), DataType::Time,\n    ];\n    for (name, dt) in schema.iter() {\n        let hit = BAD.iter().any(|b| b == dt)\n            || matches!(dt, DataType::Duration(_) | DataType::Time);\n        if hit { return Err(format!(\"ndjson column {name} has unsupported dtype {dt}\")); }\n    }\n    Ok(())\n}","typeGuard":"fn is_ndjson_safe_dtype(dt: &DataType) -> bool {\n    !matches!(\n        dt,\n        DataType::Int8 | DataType::Int16 | DataType::UInt8 | DataType::UInt16\n            | DataType::Duration(_) | DataType::Time\n    )\n}","tryCatchPattern":"catch_unwind around scan_ndjson(...).collect() only re-labels the panic; the value arrives per-row so pre-validating the schema is the only reliable guard.","preventionTips":["Validate externally-supplied schemas against the NDJSON dtype whitelist before the first read","Contract-test a one-line NDJSON sample for every dtype in your schema","Prefer Int32/Int64 at ingestion boundaries; narrow types at the end of the pipeline"],"tags":["polars","ndjson","json","schema","dtype","deserialization","panic"],"backgroundTag":"unsupported-data-type","analyzedSha":"9b5d73fd00236295624374b075d16b1fe6ec6df9","analyzedAt":"2026-08-19T12:15:06.350Z","contentChangedAt":"2026-08-19T12:15:06.350Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}