pola-rs/polars · error · NotImplementedError

`collect_batches` is not supported by

Error message

`collect_batches` is not supported by {type(self).__name__}

What it means

`collect_batches` executes a LazyFrame and yields results in batches; only some engines implement it. The base `Engine` class raises NotImplementedError with this message when the selected engine does not provide batched collection.

Solutions

  1. Use `lf.collect_batches()` with the default/local engine that implements batching.
  2. If batching is not essential, call `lf.collect()` and slice the DataFrame into chunks yourself.
  3. Implement `collect_batches` in your custom Engine subclass, returning a `_CollectBatches` iterator.
  4. Verify the engine parameter value against polars docs for batched-execution support.

Example fix

// before
for df in lf.collect_batches(engine="custom-engine"): ...
// after
for df in lf.collect_batches(): ...  # default engine supports batching
Defensive patterns

Strategy: try-catch

Validate before calling

if type(engine_obj).collect_batches is Engine.collect_batches:
    use_default_engine = True

Type guard

def supports_batches(engine: object) -> bool:
    return type(engine).collect_batches is not Engine.collect_batches

Try / catch

try:
    it = lf.collect_batches(engine=eng)
except NotImplementedError:
    df = lf.collect()

Prevention

When it happens

Trigger: Calling `LazyFrame.collect_batches(engine=<engine>)` where the engine class does not override `collect_batches` — e.g. a custom engine or an engine whose base-class stub is reached.

Common situations: Custom Engine implementations missing the batched-collect override; passing an engine alias that maps to a non-batching engine; code copied from streaming examples executed against another engine.

Related errors

AI-assisted analysis of pola-rs/polars@fe841f959e (2026-09-18). Data as JSON: /api/errors/4080584180179406. Report an issue: GitHub.

Appendix: source

Thrown at py-polars/src/polars/lazyframe/engine.py:164

        optimizations: QueryOptFlags,
        gevent: bool = False,
    ) -> AsyncResult[DataFrame]:
        """Execute `lf` asynchronously."""
        msg = f"`collect_async` is not supported by {type(self).__name__}"
        raise NotImplementedError(msg)

    def collect_batches(
        self,
        lf: LazyFrame,
        *,
        optimizations: QueryOptFlags,
        maintain_order: bool = True,
        chunk_size: int | None = None,
        lazy: bool = False,
    ) -> _CollectBatches:
        """Execute `lf`, yielding its result in batches."""
        msg = f"`collect_batches` is not supported by {type(self).__name__}"
        raise NotImplementedError(msg)

    def collect_all(
        self, lfs: Iterable[LazyFrame], *, optimizations: QueryOptFlags
    ) -> list[DataFrame]:
        """Execute several queries, potentially in parallel."""
        msg = f"`collect_all` is not supported by {type(self).__name__}"
        raise NotImplementedError(msg)

    def collect_all_async(
        self,
        lfs: Iterable[LazyFrame],
        *,
        optimizations: QueryOptFlags,
        gevent: bool = False,
    ) -> AsyncResult[list[DataFrame]]:
        """Execute several queries asynchronously."""
        msg = f"`collect_all_async` is not supported by {type(self).__name__}"
        raise NotImplementedError(msg)

View on GitHub (pinned to fe841f959e)