{"record":{"id":"93979b60326f7721","repo":"pola-rs/polars","slug":"cannot-set-iter-batches-without-also-setting-a-n","errorCode":null,"errorMessage":"Cannot set `iter_batches` without also setting a non-zero `batch_size`","messagePattern":"Cannot set `iter_batches` without also setting a non-zero `batch_size`","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"py-polars/src/polars/io/database/_executor.py","lineNumber":286,"sourceCode":"\n        return None\n\n    def _from_rows(\n        self,\n        *,\n        batch_size: int | None,\n        iter_batches: bool,\n        schema_overrides: SchemaDict | None,\n        infer_schema_length: int | None,\n    ) -> DataFrame | Iterator[DataFrame] | None:\n        \"\"\"Return resultset data row-wise for frame init.\"\"\"\n        from polars import DataFrame\n\n        if iter_batches and not batch_size:\n            msg = (\n                \"Cannot set `iter_batches` without also setting a non-zero `batch_size`\"\n            )\n            raise ValueError(msg)\n\n        if is_async := isinstance(original_result := self.result, Coroutine):\n            self.result = _run_async(self.result)\n        try:\n            if hasattr(self.result, \"fetchall\"):\n                if is_alchemy := (self.driver_name == \"sqlalchemy\"):\n                    if hasattr(self.result, \"cursor\"):\n                        cursor_desc = [\n                            (d[0], d[1:]) for d in self.result.cursor.description\n                        ]\n                    elif hasattr(self.result, \"_metadata\"):\n                        cursor_desc = [(k, None) for k in self.result._metadata.keys]\n                    else:\n                        msg = f\"Unable to determine metadata from query result; {self.result!r}\"\n                        raise ValueError(msg)\n\n                elif hasattr(self.result, \"description\"):\n                    cursor_desc = [(d[0], d[1:]) for d in self.result.description]","sourceCodeStart":268,"sourceCodeEnd":304,"githubUrl":"https://github.com/pola-rs/polars/blob/df599052daf96e7a9cc30a3b0c6bd25d6947e3c0/py-polars/src/polars/io/database/_executor.py#L268-L304","documentation":"On the row-wise fetch path (_from_rows, used when the driver has no Arrow support), batched iteration is implemented via repeated result.fetchmany(batch_size). fetchmany requires an explicit size, so iter_batches=True with a missing or zero batch_size is rejected immediately with this ValueError. Note this is the generic message (no driver name) - it applies to any driver that ends up on the row-wise path.","triggerScenarios":"pl.read_database('SELECT ...', connection=<plain DBAPI cursor>, iter_batches=True) with batch_size=None (default) or 0; drivers without Arrow registry entries falling back to row-wise fetching.","commonSituations":"Streaming large tables via sqlite3/psycopg2 cursors; shared helper functions where iter_batches is a parameter but batch_size was never wired through; tutorials that show iter_batches without the required companion argument.","solutions":["Set a concrete batch_size alongside iter_batches, e.g. batch_size=10_000","Read everything at once (iter_batches=False) when memory permits","For cursor-level control, iterate manually with cursor.fetchmany(n) and build pl.DataFrame per chunk"],"exampleFix":"# before\nframes = pl.read_database(query, connection=cursor, iter_batches=True)\n\n# after\nframes = pl.read_database(query, connection=cursor, iter_batches=True,\n                          batch_size=10_000)\nfor df in frames:\n    process(df)","handlingStrategy":"validation","validationCode":"if iter_batches:\n    assert batch_size and batch_size > 0, (\n        'iter_batches requires an explicit positive batch_size'\n    )\nframes = pl.read_database(query, connection=cursor,\n                          iter_batches=iter_batches, batch_size=batch_size)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always pass a concrete batch_size wherever iter_batches=True appears","Wrap read_database in one team utility that enforces the iter_batches/batch_size invariant","Size batches with fetchmany in mind: large enough for throughput, small enough to bound memory"],"tags":["polars","database","row-wise","iter-batches","batch-size","valueerror"],"backgroundTag":null,"analyzedSha":"df599052daf96e7a9cc30a3b0c6bd25d6947e3c0","analyzedAt":"2026-08-16T12:10:03.978Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}