pola-rs/polars · error

unexpected dtype when deserializing ndjson

Error message

unexpected dtype when deserializing ndjson

What it means

In NDJSON reading, each column's values are appended into an AnyValueBuffer, and the match in Buffer::add only handles Boolean, Int32/Int64, UInt32/UInt64, Float32/64, String, Datetime, Date, the recursive All bucket and Null. The AnyValueBuffer construction, however, also produces Int8, Int16, UInt8, UInt16, Duration and Time variants when the corresponding dtype features are on - those fall into the catch-all panic on the first non-null value.

Source

Thrown at crates/polars-io/src/ndjson/buffer.rs:138

                    TimeUnit::Microseconds, // ignored
                )?;
                buf.append_option(v);
                Ok(())
            },
            All(dtype, buf) => {
                let av = deserialize_all(value, dtype, self.ignore_errors)?;
                buf.push(av);
                Ok(())
            },
            Null(builder) => {
                if !(matches!(value, Value::Static(StaticNode::Null)) || self.ignore_errors) {
                    polars_bail!(ComputeError: "got non-null value for NULL-typed column: {}", value)
                };

                builder.append_null();
                Ok(())
            },
            _ => panic!("unexpected dtype when deserializing ndjson"),
        }
    }

    pub fn add_null(&mut self) {
        self.buf.add(AnyValue::Null).expect("should not fail");
    }
}
pub(crate) fn init_buffers(
    schema: &Schema,
    capacity: usize,
    ignore_errors: bool,
) -> PolarsResult<PlIndexMap<BufferKey<'_>, Buffer<'_>>> {
    schema
        .iter()
        .map(|(name, dtype)| {
            let av_buf = (dtype, capacity).into();
            let key = KnownKey::from(name.as_str());
            Ok((

View on GitHub (pinned to 9b5d73fd00)

Solutions

  1. Cast those columns to a supported dtype in the schema you pass: Int8/Int16 -> Int32 or Int64, UInt8/UInt16 -> UInt32/UInt64, Duration -> Int64, Time -> Int64 (or String)
  2. Or omit the problem fields from the ndjson schema and cast after the read
  3. Track upstream: extend the match in polars-io/src/ndjson/buffer.rs to cover these buffer variants

Example fix

// before
let schema = Schema::from_iter(vec![
    (PlSmallStr::from("delta"), DataType::Duration(TimeUnit::Milliseconds)), // panics on first row
]);

// after
let schema = Schema::from_iter(vec![
    (PlSmallStr::from("delta"), DataType::Int64),  // read raw, .cast(Duration) afterwards
]);
Defensive patterns

Strategy: validation

Validate before calling

use polars_core::prelude::*;
fn ndjson_schema_supported(schema: &Schema) -> Result<(), String> {
    const BAD: &[DataType] = &[
        DataType::Int8, DataType::Int16, DataType::UInt8, DataType::UInt16,
        DataType::Duration(TimeUnit::Nanoseconds), DataType::Time,
    ];
    for (name, dt) in schema.iter() {
        let hit = BAD.iter().any(|b| b == dt)
            || matches!(dt, DataType::Duration(_) | DataType::Time);
        if hit { return Err(format!("ndjson column {name} has unsupported dtype {dt}")); }
    }
    Ok(())
}

Type guard

fn is_ndjson_safe_dtype(dt: &DataType) -> bool {
    !matches!(
        dt,
        DataType::Int8 | DataType::Int16 | DataType::UInt8 | DataType::UInt16
            | DataType::Duration(_) | DataType::Time
    )
}

Try / catch

catch_unwind around scan_ndjson(...).collect() only re-labels the panic; the value arrives per-row so pre-validating the schema is the only reliable guard.

Prevention

When it happens

Trigger: scan_ndjson/read_ndjson with an explicit schema (or inferred dtype) containing Int8, Int16, UInt8, UInt16, Duration or Time columns - the buffer variants exist only when the dtype-i8/-i16/-u8/-u16/-duration/-time features are enabled - and a row where that field is non-null.

Common situations: Reusing a parquet/Arrow-derived schema for NDJSON ingestion; pushing narrow integer types through a pipeline; hand-authored schemas with Time or Duration columns.

Related errors

AI-assisted analysis of pola-rs/polars@9b5d73fd00 (2026-08-19). Data as JSON: /api/errors/c5236c812b581146. Report an issue: GitHub.