{"record":{"id":"31813e95cad02b62","repo":"pola-rs/polars","slug":"sink-batches-is-not-supported-by-type-self-n","errorCode":null,"errorMessage":"`sink_batches` is not supported by {type(self).__name__}","messagePattern":"`sink_batches` is not supported by (.+?)","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"py-polars/src/polars/lazyframe/engine.py","lineNumber":341,"sourceCode":"        optimizations: QueryOptFlags,\n    ) -> LazyFrame | None:\n        \"\"\"See :meth:`polars.LazyFrame.sink_ndjson`.\"\"\"\n        msg = f\"`sink_ndjson` is not supported by {type(self).__name__}\"\n        raise NotImplementedError(msg)\n\n    def sink_batches(\n        self,\n        lf: LazyFrame,\n        function: Callable[[DataFrame], bool | None],\n        *,\n        chunk_size: int | None,\n        maintain_order: bool,\n        lazy: bool,\n        optimizations: QueryOptFlags,\n    ) -> LazyFrame | None:\n        \"\"\"See :meth:`polars.LazyFrame.sink_batches`.\"\"\"\n        msg = f\"`sink_batches` is not supported by {type(self).__name__}\"\n        raise NotImplementedError(msg)\n\n\nclass _LocalEngine(Engine):\n    \"\"\"Base for in-process engines backed by `PyLazyFrame`.\"\"\"\n\n    _name: ClassVar[str]\n\n    @property\n    def name(self) -> str:\n        \"\"\"Name of the engine.\"\"\"\n        return self._name\n\n    def execute(self, lf: LazyFrame, *, optimizations: QueryOptFlags) -> QueryResult:\n        df = self.collect(lf, optimizations=optimizations)\n        return SingleNodeQueryResult(df)  # type: ignore[arg-type]\n\n    def _post_opt_callback(\n        self,","sourceCodeStart":323,"sourceCodeEnd":359,"githubUrl":"https://github.com/pola-rs/polars/blob/df599052daf96e7a9cc30a3b0c6bd25d6947e3c0/py-polars/src/polars/lazyframe/engine.py#L323-L359","documentation":"`Engine.sink_batches` (engine.py:341) is the optional `Engine` hook behind `LazyFrame.sink_batches`, which streams result batches to a Python callback `function(DataFrame) -> bool | None`. Only the local `_LocalEngine` family implements it; `RemoteEngine` does not, and neither do minimal custom engines. The stub raises `NotImplementedError` naming the engine class.","triggerScenarios":"`lf.sink_batches(fn, engine=pl.RemoteEngine())` or `lf.sink_batches(fn, engine=my_engine)` with a custom `Engine` subclass that implements only `collect`/`execute`. Because the callback runs in-process, no remote implementation exists.","commonSituations":"Feeding query output chunk-by-chunk into another in-process system (vector DBs, HTTP uploaders) and then pointing the query at Polars Cloud; custom backends where batch callbacks were never wired; shared utility code that assumes every engine accepts `sink_batches`.","solutions":["Use a local engine: `lf.sink_batches(fn, engine='streaming')` (default)","For remote data movement, sink to a cloud URI with `sink_parquet`/`sink_ipc`/`sink_csv` on RemoteEngine and process the files separately","If you own the engine, implement `sink_batches` with the stub's signature (function, chunk_size, maintain_order, lazy, optimizations)","Collect first as a stopgap: iterate `lf.collect().iter_slices(n)`"],"exampleFix":"# before\nlf.sink_batches(consume, engine=pl.RemoteEngine())  # NotImplementedError\n\n# after\nlf.sink_batches(consume, engine='streaming')","handlingStrategy":"validation","validationCode":"from polars.lazyframe.engine import Engine\n\ndef engine_can_sink_batches(engine: pl.Engine) -> bool:\n    return type(engine).sink_batches is not Engine.sink_batches\n\nengine = engine if engine_can_sink_batches(engine) else pl.StreamingEngine()\nlf.sink_batches(consume, engine=engine)","typeGuard":null,"tryCatchPattern":"try:\n    lf.sink_batches(fn, engine=engine)\nexcept NotImplementedError as e:\n    if 'sink_batches' in str(e):\n        for chunk in lf.collect(engine=engine).iter_slices(100_000):\n            fn(chunk)\n    else:\n        raise","preventionTips":["Batch callbacks are inherently in-process: pair them with local engines","For remote execution use URI sinks plus a separate processing step","Add a capability check wherever engine= is user- or config-selectable"],"tags":["polars","engine","sink","batches","remote","not-implemented"],"backgroundTag":null,"analyzedSha":"df599052daf96e7a9cc30a3b0c6bd25d6947e3c0","analyzedAt":"2026-08-16T12:10:03.978Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}