pola-rs/polars · error · NotImplementedError

`collect_batches` is not supported by {type(self).__name__}

Error message

`collect_batches` is not supported by {type(self).__name__}

What it means

`Engine.collect_batches` (engine.py:204) is an optional method on the `Engine` ABC whose default stub raises `NotImplementedError` with the engine's class name. It backs `LazyFrame.collect_batches`, which streams a query result in `DataFrame` chunks. Only the `_LocalEngine` family (`in-memory`, `streaming`, `gpu`, `auto`) implements it; `RemoteEngine` and minimal custom engines do not.

Source

Thrown at py-polars/src/polars/lazyframe/engine.py:204

        optimizations: QueryOptFlags,
        gevent: bool = False,
    ) -> AsyncResult[DataFrame]:
        """Execute `lf` asynchronously."""
        msg = f"`collect_async` is not supported by {type(self).__name__}"
        raise NotImplementedError(msg)

    def collect_batches(
        self,
        lf: LazyFrame,
        *,
        optimizations: QueryOptFlags,
        maintain_order: bool = True,
        chunk_size: int | None = None,
        lazy: bool = False,
    ) -> Iterator[DataFrame]:
        """Execute `lf`, yielding its result in batches."""
        msg = f"`collect_batches` is not supported by {type(self).__name__}"
        raise NotImplementedError(msg)

    def collect_all(
        self, lfs: Iterable[LazyFrame], *, optimizations: QueryOptFlags
    ) -> list[DataFrame]:
        """Execute several queries, potentially in parallel."""
        msg = f"`collect_all` is not supported by {type(self).__name__}"
        raise NotImplementedError(msg)

    def collect_all_async(
        self,
        lfs: Iterable[LazyFrame],
        *,
        optimizations: QueryOptFlags,
        gevent: bool = False,
    ) -> AsyncResult[list[DataFrame]]:
        """Execute several queries asynchronously."""
        msg = f"`collect_all_async` is not supported by {type(self).__name__}"
        raise NotImplementedError(msg)

View on GitHub (pinned to df599052da)

Solutions

1. Use a local engine: `lf.collect_batches(engine='streaming')` or `engine='in-memory'`
2. For remote execution, run the query to a cloud sink (`sink_parquet` is supported by RemoteEngine) and read it back in batches locally
3. Implement `collect_batches` on your custom `Engine` subclass
4. Guard the call with a capability check on `type(engine)` before invoking it

Example fix

# before
for batch in lf.collect_batches(engine=pl.RemoteEngine()):  # NotImplementedError
    ...

# after
for batch in lf.collect_batches(engine='streaming'):
    ...

from polars.lazyframe.engine import Engine

def supports_batches(engine: pl.Engine) -> bool:
    return type(engine).collect_batches is not Engine.collect_batches

if not supports_batches(engine):
    raise ValueError(f'{type(engine).__name__} cannot stream batches; use a local engine')

try:
    for batch in lf.collect_batches(engine=engine):
        handle_batch(batch)
except NotImplementedError:
    df = lf.collect(engine=engine)  # fall back to full materialization

When it happens

Trigger: `lf.collect_batches(chunk_size=..., engine=pl.RemoteEngine())`, or the same call with a custom `pl.Engine` subclass that implements only the abstract `collect`/`execute`. Also triggered indirectly by code that iterates `lf.collect_batches(...)` while a non-local engine affinity is configured via `pl.Config.set_engine_affinity`.

Common situations: Memory-conscious batch-processing pipelines moved to Polars Cloud; custom engines written as thin wrappers around `execute`; configurable engine arguments resolved at runtime where one branch picks an engine without batching support.

Related errors

• `collect_async` is not supported by {type(self).__name__}
• `collect_all` is not supported by {type(self).__name__}
• `sink_parquet` is not supported by {type(self).__name__}
• `sink_ipc` is not supported by {type(self).__name__}
• `sink_csv` is not supported by {type(self).__name__}

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/4080584180179406. Report an issue: GitHub.