{"record":{"id":"ce971a3845913625","repo":"lancedb/lancedb","slug":"failed-to-convert-polars-dataframe-schema-to-arrow","errorCode":null,"errorMessage":"failed to convert Polars DataFrame schema to Arrow schema","messagePattern":"failed to convert Polars DataFrame schema to Arrow schema","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"rust/lancedb/src/data/scannable.rs","lineNumber":194,"sourceCode":"        let error_stream = Box::pin(SimpleRecordBatchStream {\n            schema: schema.clone(),\n            stream: once(async {\n                Err(Error::InvalidInput {\n                    message: \"Stream has already been consumed\".to_string(),\n                })\n            }),\n        });\n        std::mem::replace(self, error_stream)\n    }\n}\n\n#[cfg(feature = \"polars\")]\nimpl Scannable for polars::frame::DataFrame {\n    fn schema(&self) -> SchemaRef {\n        crate::polars_arrow_convertors::convert_polars_df_schema_to_arrow_rb_schema(\n            self.schema().clone(),\n        )\n        .expect(\"failed to convert Polars DataFrame schema to Arrow schema\")\n    }\n\n    fn scan_as_stream(&mut self) -> SendableRecordBatchStream {\n        let schema = Scannable::schema(self);\n        let batches: crate::Result<Vec<RecordBatch>> =\n            match crate::arrow::PolarsDataFrameRecordBatchReader::new(self.clone()) {\n                Err(e) => Err(e),\n                Ok(reader) => reader.map(|b| b.map_err(Into::into)).collect(),\n            };\n        match batches {\n            Err(e) => Box::pin(SimpleRecordBatchStream {\n                schema,\n                stream: once(async move { Err(e) }),\n            }),\n            Ok(batches) => {\n                let stream = futures::stream::iter(batches.into_iter().map(Ok));\n                Box::pin(SimpleRecordBatchStream { schema, stream })\n            }","sourceCodeStart":176,"sourceCodeEnd":212,"githubUrl":"https://github.com/lancedb/lancedb/blob/c7b051aff7039333a3f61b79217246c27676806a/rust/lancedb/src/data/scannable.rs#L176-L212","documentation":"When a Polars `DataFrame` is used as a `Scannable` data source (with the `polars` feature), its schema is converted from Polars' Arrow representation to the standard Arrow `SchemaRef` used by LanceDB. If that conversion returns an error, `.expect()` panics with this message, since a Polars frame schema should always be convertible.","triggerScenarios":"Passing a `polars.DataFrame` as source data to `add`/`create` table APIs where `convert_polars_df_schema_to_arrow_rb_schema` fails on the frame's schema (typically an unsupported/exotic Polars dtype or a version mismatch between polars and arrow crates).","commonSituations":"Using very new or very old Polars versions whose dtype mapping is not covered by the converter; columns with unusual nested or extension dtypes; Decimal or custom dtypes in the frame.","solutions":["Cast problematic columns to supported dtypes (e.g. standard numeric, string, list, struct) before passing the DataFrame.","Align the installed polars version with the one LanceDB was built against (upgrade both packages together).","Convert the DataFrame to Arrow explicitly (`df.to_arrow()` / record batches) and pass Arrow data instead.","Report the failing dtype so the converter can be extended, if it is a legitimate supported dtype."],"exampleFix":"# before\ndb.create_table(\"t\", df)  # df has an exotic dtype\n\n# after\ndf = df.with_columns(pl.col(\"weird_col\").cast(pl.String))\ndb.create_table(\"t\", df)","handlingStrategy":"validation","validationCode":"SUPPORTED = {pl.Int8, pl.Int16, pl.Int32, pl.Int64, pl.UInt8, pl.UInt16, pl.UInt32, pl.UInt64,\n              pl.Float32, pl.Float64, pl.Boolean, pl.String, pl.Date, pl.Datetime, pl.List, pl.Struct}\nbad = [c for c, dt in df.schema.items() if dt.base_type() not in SUPPORTED]\nif bad:\n    df = df.with_columns([pl.col(c).cast(pl.String) for c in bad])","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Cast exotic dtypes (decimal, categorical edge cases, custom extensions) before passing Polars frames.","Keep polars and lancedb versions in sync; upgrade together.","Prefer passing Arrow data when schemas contain unusual types."],"tags":["rust","polars","arrow","schema","panic"],"backgroundTag":"schema-validation-failed","analyzedSha":"c7b051aff7039333a3f61b79217246c27676806a","analyzedAt":"2026-09-08T23:42:37.579Z","contentChangedAt":"2026-09-08T23:42:37.579Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}