pola-rs/polars · error · NotImplementedError

`collect_all` is not supported by {type(self).__name__}

Error message

`collect_all` is not supported by {type(self).__name__}

What it means

`Engine.collect_all` (engine.py:211) is the optional multi-query execution hook used by `pl.collect_all`; its default stub raises `NotImplementedError` naming the engine. It executes several `LazyFrame`s potentially in parallel and is implemented only by the local engine family. `RemoteEngine` and custom `Engine` subclasses that define just `collect`/`execute` hit the stub.

Source

Thrown at py-polars/src/polars/lazyframe/engine.py:211

    def collect_batches(
        self,
        lf: LazyFrame,
        *,
        optimizations: QueryOptFlags,
        maintain_order: bool = True,
        chunk_size: int | None = None,
        lazy: bool = False,
    ) -> Iterator[DataFrame]:
        """Execute `lf`, yielding its result in batches."""
        msg = f"`collect_batches` is not supported by {type(self).__name__}"
        raise NotImplementedError(msg)

    def collect_all(
        self, lfs: Iterable[LazyFrame], *, optimizations: QueryOptFlags
    ) -> list[DataFrame]:
        """Execute several queries, potentially in parallel."""
        msg = f"`collect_all` is not supported by {type(self).__name__}"
        raise NotImplementedError(msg)

    def collect_all_async(
        self,
        lfs: Iterable[LazyFrame],
        *,
        optimizations: QueryOptFlags,
        gevent: bool = False,
    ) -> AsyncResult[list[DataFrame]]:
        """Execute several queries asynchronously."""
        msg = f"`collect_all_async` is not supported by {type(self).__name__}"
        raise NotImplementedError(msg)

    def sink_parquet(
        self,
        lf: LazyFrame,
        path: str | Path | IO[bytes] | PartitionBy,
        *,
        compression: ParquetCompression,

View on GitHub (pinned to df599052da)

Solutions

1. Collect sequentially with the engine's supported path: `[lf.collect(engine=remote) for lf in lfs]`
2. Use a local engine: `pl.collect_all(lfs, engine='streaming')` (parallel multi-query works there)
3. Implement `collect_all` on your custom `Engine` subclass
4. Check engine capability before dispatching when the engine is configurable

Example fix

# before
dfs = pl.collect_all([lf1, lf2], engine=pl.RemoteEngine())  # NotImplementedError

# after
dfs = [lf.collect(engine=remote) for lf in (lf1, lf2)]

from polars.lazyframe.engine import Engine

def supports_collect_all(engine: pl.Engine) -> bool:
    return type(engine).collect_all is not Engine.collect_all

if supports_collect_all(engine):
    dfs = pl.collect_all(lfs, engine=engine)
else:
    dfs = [lf.collect(engine=engine) for lf in lfs]

try:
    dfs = pl.collect_all(lfs, engine=engine)
except NotImplementedError as e:
    if 'collect_all' in str(e):
        dfs = [lf.collect(engine=engine) for lf in lfs]
    else:
        raise

When it happens

Trigger: `pl.collect_all([lf1, lf2], engine=pl.RemoteEngine())`, or `pl.collect_all(lfs, engine=my_engine)` with a minimal custom engine. Also occurs when a non-local engine is set as the global affinity and `pl.collect_all` inherits it.

Common situations: Fan-out ETL code that executes a list of built queries with one engine argument; switching a codebase from local to remote execution by only changing the engine; half-finished custom backends used across a shared utility function.

Related errors

• `collect_async` is not supported by {type(self).__name__}
• `collect_batches` is not supported by {type(self).__name__}
• `collect_all_async` is not supported by {type(self).__name__
• `sink_parquet` is not supported by {type(self).__name__}
• `sink_ipc` is not supported by {type(self).__name__}

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/529d7313865abd5c. Report an issue: GitHub.