What causes "unimplemented: AnonymousScan" in pola-rs/polars?

Run the query with the in-memory engine: lf.collect(engine='in-memory')

unimplemented: AnonymousScan

Error message

unimplemented: AnonymousScan

What it means

The new streaming engine's IR lowering has no physical source node for anonymous scans (FileScanIR::Anonymous), so it panics with todo!(). Anonymous scans inject arbitrary user callbacks that the streaming graph cannot schedule; only concrete file/IO sources are lowered.

Source

Thrown at crates/polars-stream/src/physical_plan/lower_ir.rs:832

                            .python_scan()
                            .expect("should be python scan");

                        python_dataset_scan_to_reader_builder(expanded_scan)
                    },

                    #[cfg(feature = "scan_lines")]
                    FileScanIR::Lines { name: _ } => {
                        Arc::new(crate::nodes::io_sources::lines::LineReaderBuilder {
                            prefetch_limit: RelaxedCell::new_usize(0),
                            prefetch_semaphore: std::sync::OnceLock::new(),
                            shared_prefetch_wait_group_slot: Default::default(),
                            io_metrics: std::sync::OnceLock::new(),
                        }) as _
                    },

                    FileScanIR::ExpandedPaths { name: _ } => unreachable!(),

                    FileScanIR::Anonymous { .. } => todo!("unimplemented: AnonymousScan"),
                };

                {
                    let cloud_options = unified_scan_args.cloud_options.clone().map(Arc::new);
                    let file_schema = file_info.schema;

                    let (projected_schema, file_schema) =
                        multi_scan::functions::resolve_projections::resolve_projections(
                            &output_schema,
                            &file_schema,
                            &mut hive_parts,
                            unified_scan_args
                                .row_index
                                .as_ref()
                                .map(|ri| ri.name.as_str()),
                            unified_scan_args
                                .include_file_paths
                                .as_ref()

View on GitHub (pinned to df599052da)

Solutions

Run the query with the in-memory engine: lf.collect(engine='in-memory')
Materialize the source first (pl.read_* into a DataFrame, then pl.LazyFrame(df)) so streaming only handles downstream stages
Replace the anonymous scan with a native scan of concrete files (scan_parquet/scan_csv/scan_ipc) that the streaming engine supports

Example fix

# before
lf = pl.scan_pyarrow_dataset(ds)  # anonymous scan
lf.collect(engine="streaming")  # panics during IR lowering

# after
lf.collect(engine="in-memory")  # in-memory engine lowers anonymous scans

Defensive patterns

Strategy: fallback

Validate before calling

def is_anonymous_scan(lf: pl.LazyFrame) -> bool:
    return "scan_pyarrow_dataset" in (lf.explain() or "") or " Anonymous" in lf.explain()

Type guard

def engine_for(lf: pl.LazyFrame, preferred: str = "streaming") -> str:
    try:
        lf.collect(engine=preferred)
        return preferred
    except Exception:
        return "in-memory"

Try / catch

try:
    out = lf.collect(engine="streaming")
except pl.exceptions.PanicException:
    out = lf.collect(engine="in-memory")  # anonymous scans need the in-memory engine

Prevention

Avoid scan_pyarrow_dataset and callback-based scans when targeting the streaming engine
Prefer native scan_parquet/scan_csv/scan_ipc sources for streaming pipelines
Materialize custom sources to DataFrames before handing data to streaming queries

When it happens

Trigger: Executing a lazy query with engine='streaming' (or a context where the streaming engine is default) whose source is an anonymous scan - in Python typically pl.scan_pyarrow_dataset(...) or another scan function backed by a user callback - at collect()/explain() time.

Common situations: Users switch queries to the streaming engine (large data, low memory) while keeping a pyarrow-dataset or custom-callback scan; or polars automatically falls back to an anonymous scan for a late-materialized DataFrame.

Related errors

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/ff70f2ba994b961e. Report an issue: GitHub.