pola-rs/polars · error · NotImplementedError

`collect_async` is not supported by {type(self).__name__}

Error message

`collect_async` is not supported by {type(self).__name__}

What it means

`Engine` is the abstract base class for pluggable Polars execution backends; only `collect` and `execute` are abstract, while optional operations like `collect_async` ship a default stub that raises `NotImplementedError` naming the engine class. The built-in local engines (`in-memory`, `streaming`, `gpu`, `auto`) override it, but `RemoteEngine` and any minimal custom subclass do not. The error therefore identifies a capability gap of the specific engine instance you passed.

Source

Thrown at py-polars/src/polars/lazyframe/engine.py:191

        This method of materializing a `LazyFrame` makes no guarantees as to where
        the result is materialized. This can be on the GPU for the GPU-engine,
        on the cluster or remote storage for the distributed engine and the streaming
        engine could spill the result if it needed to.

        The `QueryResult` can always be consumed as a new `LazyFrame` by calling `.lazy`
        """

    def collect_async(
        self,
        lf: LazyFrame,
        *,
        optimizations: QueryOptFlags,
        gevent: bool = False,
    ) -> AsyncResult[DataFrame]:
        """Execute `lf` asynchronously."""
        msg = f"`collect_async` is not supported by {type(self).__name__}"
        raise NotImplementedError(msg)

    def collect_batches(
        self,
        lf: LazyFrame,
        *,
        optimizations: QueryOptFlags,
        maintain_order: bool = True,
        chunk_size: int | None = None,
        lazy: bool = False,
    ) -> Iterator[DataFrame]:
        """Execute `lf`, yielding its result in batches."""
        msg = f"`collect_batches` is not supported by {type(self).__name__}"
        raise NotImplementedError(msg)

    def collect_all(
        self, lfs: Iterable[LazyFrame], *, optimizations: QueryOptFlags
    ) -> list[DataFrame]:
        """Execute several queries, potentially in parallel."""

View on GitHub (pinned to df599052da)

Solutions

1. Use a local engine that supports it: `lf.collect_async(engine='streaming')` (the default) or `engine='in-memory'`
2. For remote work use the supported background API: `handle = lf.collect(engine=remote, background=True)` or `lf.execute(...)` plus a sink
3. If you own the engine, implement `collect_async` on your `Engine` subclass
4. Catch `NotImplementedError` and degrade gracefully when the engine is chosen at runtime

Example fix

# before
res = lf.collect_async(engine=pl.RemoteEngine())  # NotImplementedError

# after
handle = lf.collect(engine=remote, background=True)  # InProcessQuery
df = handle.fetch()

from polars.lazyframe.engine import Engine

def engine_supports(engine: pl.Engine, method: str) -> bool:
    # True only when the engine overrides the Engine stub
    return getattr(type(engine), method, None) is not getattr(Engine, method)

remote = pl.RemoteEngine()
assert engine_supports(remote, 'collect_async') is False
assert engine_supports(pl.StreamingEngine(), 'collect_async') is True

try:
    ar = lf.collect_async(engine=engine)
except NotImplementedError as e:
    # message names the engine class, e.g. 'RemoteEngine'
    handle = lf.collect(engine=engine, background=True)

When it happens

Trigger: `lf.collect_async(engine=pl.RemoteEngine())`, or `lf.collect_async(engine=my_engine)` where `my_engine` subclasses `pl.Engine` and only implements `collect`/`execute`. The error is raised synchronously at call time, before any work is scheduled.

Common situations: Migrating a pipeline to Polars Cloud and reusing gevent/async orchestration code unchanged; writing a custom engine (test double, alternative backend) and forgetting that optional `Engine` methods are not abstract; assuming every engine supports every `LazyFrame` entry point.

Related errors

• `collect_batches` is not supported by {type(self).__name__}
• `collect_all` is not supported by {type(self).__name__}
• `collect_all_async` is not supported by {type(self).__name__
• `sink_parquet` is not supported by {type(self).__name__}
• `sink_ipc` is not supported by {type(self).__name__}

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/ddd5498b31fb8d6c. Report an issue: GitHub.