lancedb/lancedb · error

failed to convert Polars DataFrame schema to Arrow schema

Error message

failed to convert Polars DataFrame schema to Arrow schema

What it means

When a Polars `DataFrame` is used as a `Scannable` data source (with the `polars` feature), its schema is converted from Polars' Arrow representation to the standard Arrow `SchemaRef` used by LanceDB. If that conversion returns an error, `.expect()` panics with this message, since a Polars frame schema should always be convertible.

Solutions

1. Cast problematic columns to supported dtypes (e.g. standard numeric, string, list, struct) before passing the DataFrame.
2. Align the installed polars version with the one LanceDB was built against (upgrade both packages together).
3. Convert the DataFrame to Arrow explicitly (`df.to_arrow()` / record batches) and pass Arrow data instead.
4. Report the failing dtype so the converter can be extended, if it is a legitimate supported dtype.

Example fix

# before
db.create_table("t", df)  # df has an exotic dtype

# after
df = df.with_columns(pl.col("weird_col").cast(pl.String))
db.create_table("t", df)

SUPPORTED = {pl.Int8, pl.Int16, pl.Int32, pl.Int64, pl.UInt8, pl.UInt16, pl.UInt32, pl.UInt64,
              pl.Float32, pl.Float64, pl.Boolean, pl.String, pl.Date, pl.Datetime, pl.List, pl.Struct}
bad = [c for c, dt in df.schema.items() if dt.base_type() not in SUPPORTED]
if bad:
    df = df.with_columns([pl.col(c).cast(pl.String) for c in bad])

When it happens

Trigger: Passing a `polars.DataFrame` as source data to `add`/`create` table APIs where `convert_polars_df_schema_to_arrow_rb_schema` fails on the frame's schema (typically an unsupported/exotic Polars dtype or a version mismatch between polars and arrow crates).

Common situations: Using very new or very old Polars versions whose dtype mapping is not covered by the converter; columns with unusual nested or extension dtypes; Decimal or custom dtypes in the frame.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• field of index must exist in schema
• array is non-nullable
• At least one record or a schema needs to be provided
• Cannot create table from empty list without a schema
• Expected a Date type to have a `unit` property

AI-assisted analysis of lancedb/lancedb@c7b051aff7 (2026-09-08). Data as JSON: /api/errors/ce971a3845913625. Report an issue: GitHub.

Appendix: source

Thrown at rust/lancedb/src/data/scannable.rs:194

        let error_stream = Box::pin(SimpleRecordBatchStream {
            schema: schema.clone(),
            stream: once(async {
                Err(Error::InvalidInput {
                    message: "Stream has already been consumed".to_string(),
                })
            }),
        });
        std::mem::replace(self, error_stream)
    }
}

#[cfg(feature = "polars")]
impl Scannable for polars::frame::DataFrame {
    fn schema(&self) -> SchemaRef {
        crate::polars_arrow_convertors::convert_polars_df_schema_to_arrow_rb_schema(
            self.schema().clone(),
        )
        .expect("failed to convert Polars DataFrame schema to Arrow schema")
    }

    fn scan_as_stream(&mut self) -> SendableRecordBatchStream {
        let schema = Scannable::schema(self);
        let batches: crate::Result<Vec<RecordBatch>> =
            match crate::arrow::PolarsDataFrameRecordBatchReader::new(self.clone()) {
                Err(e) => Err(e),
                Ok(reader) => reader.map(|b| b.map_err(Into::into)).collect(),
            };
        match batches {
            Err(e) => Box::pin(SimpleRecordBatchStream {
                schema,
                stream: once(async move { Err(e) }),
            }),
            Ok(batches) => {
                let stream = futures::stream::iter(batches.into_iter().map(Ok));
                Box::pin(SimpleRecordBatchStream { schema, stream })
            }

View on GitHub (pinned to c7b051aff7)