{"record":{"id":"4080584180179406","repo":"pola-rs/polars","slug":"collect-batches-is-not-supported-by-type-self","errorCode":null,"errorMessage":"`collect_batches` is not supported by {type(self).__name__}","messagePattern":"`collect_batches` is not supported by (.+?)","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"py-polars/src/polars/lazyframe/engine.py","lineNumber":204,"sourceCode":"        optimizations: QueryOptFlags,\n        gevent: bool = False,\n    ) -> AsyncResult[DataFrame]:\n        \"\"\"Execute `lf` asynchronously.\"\"\"\n        msg = f\"`collect_async` is not supported by {type(self).__name__}\"\n        raise NotImplementedError(msg)\n\n    def collect_batches(\n        self,\n        lf: LazyFrame,\n        *,\n        optimizations: QueryOptFlags,\n        maintain_order: bool = True,\n        chunk_size: int | None = None,\n        lazy: bool = False,\n    ) -> Iterator[DataFrame]:\n        \"\"\"Execute `lf`, yielding its result in batches.\"\"\"\n        msg = f\"`collect_batches` is not supported by {type(self).__name__}\"\n        raise NotImplementedError(msg)\n\n    def collect_all(\n        self, lfs: Iterable[LazyFrame], *, optimizations: QueryOptFlags\n    ) -> list[DataFrame]:\n        \"\"\"Execute several queries, potentially in parallel.\"\"\"\n        msg = f\"`collect_all` is not supported by {type(self).__name__}\"\n        raise NotImplementedError(msg)\n\n    def collect_all_async(\n        self,\n        lfs: Iterable[LazyFrame],\n        *,\n        optimizations: QueryOptFlags,\n        gevent: bool = False,\n    ) -> AsyncResult[list[DataFrame]]:\n        \"\"\"Execute several queries asynchronously.\"\"\"\n        msg = f\"`collect_all_async` is not supported by {type(self).__name__}\"\n        raise NotImplementedError(msg)","sourceCodeStart":186,"sourceCodeEnd":222,"githubUrl":"https://github.com/pola-rs/polars/blob/df599052daf96e7a9cc30a3b0c6bd25d6947e3c0/py-polars/src/polars/lazyframe/engine.py#L186-L222","documentation":"`Engine.collect_batches` (engine.py:204) is an optional method on the `Engine` ABC whose default stub raises `NotImplementedError` with the engine's class name. It backs `LazyFrame.collect_batches`, which streams a query result in `DataFrame` chunks. Only the `_LocalEngine` family (`in-memory`, `streaming`, `gpu`, `auto`) implements it; `RemoteEngine` and minimal custom engines do not.","triggerScenarios":"`lf.collect_batches(chunk_size=..., engine=pl.RemoteEngine())`, or the same call with a custom `pl.Engine` subclass that implements only the abstract `collect`/`execute`. Also triggered indirectly by code that iterates `lf.collect_batches(...)` while a non-local engine affinity is configured via `pl.Config.set_engine_affinity`.","commonSituations":"Memory-conscious batch-processing pipelines moved to Polars Cloud; custom engines written as thin wrappers around `execute`; configurable engine arguments resolved at runtime where one branch picks an engine without batching support.","solutions":["Use a local engine: `lf.collect_batches(engine='streaming')` or `engine='in-memory'`","For remote execution, run the query to a cloud sink (`sink_parquet` is supported by RemoteEngine) and read it back in batches locally","Implement `collect_batches` on your custom `Engine` subclass","Guard the call with a capability check on `type(engine)` before invoking it"],"exampleFix":"# before\nfor batch in lf.collect_batches(engine=pl.RemoteEngine()):  # NotImplementedError\n    ...\n\n# after\nfor batch in lf.collect_batches(engine='streaming'):\n    ...","handlingStrategy":"validation","validationCode":"from polars.lazyframe.engine import Engine\n\ndef supports_batches(engine: pl.Engine) -> bool:\n    return type(engine).collect_batches is not Engine.collect_batches\n\nif not supports_batches(engine):\n    raise ValueError(f'{type(engine).__name__} cannot stream batches; use a local engine')","typeGuard":null,"tryCatchPattern":"try:\n    for batch in lf.collect_batches(engine=engine):\n        handle_batch(batch)\nexcept NotImplementedError:\n    df = lf.collect(engine=engine)  # fall back to full materialization","preventionTips":["Route batch-streaming pipelines to local engines only","Remote workflows should sink to cloud storage and re-read in batches instead","Add capability checks wherever engine= is configurable","Document which entry points each engine supports next to engine selection code"],"tags":["polars","engine","batching","not-implemented","plugin"],"backgroundTag":null,"analyzedSha":"df599052daf96e7a9cc30a3b0c6bd25d6947e3c0","analyzedAt":"2026-08-16T12:10:03.978Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}