pola-rs/polars · error · ValueError

Cannot set `iter_batches` without also setting a non-zero `b

Error message

Cannot set `iter_batches` without also setting a non-zero `batch_size`

What it means

On the row-wise fetch path (_from_rows, used when the driver has no Arrow support), batched iteration is implemented via repeated result.fetchmany(batch_size). fetchmany requires an explicit size, so iter_batches=True with a missing or zero batch_size is rejected immediately with this ValueError. Note this is the generic message (no driver name) - it applies to any driver that ends up on the row-wise path.

Source

Thrown at py-polars/src/polars/io/database/_executor.py:286

        return None

    def _from_rows(
        self,
        *,
        batch_size: int | None,
        iter_batches: bool,
        schema_overrides: SchemaDict | None,
        infer_schema_length: int | None,
    ) -> DataFrame | Iterator[DataFrame] | None:
        """Return resultset data row-wise for frame init."""
        from polars import DataFrame

        if iter_batches and not batch_size:
            msg = (
                "Cannot set `iter_batches` without also setting a non-zero `batch_size`"
            )
            raise ValueError(msg)

        if is_async := isinstance(original_result := self.result, Coroutine):
            self.result = _run_async(self.result)
        try:
            if hasattr(self.result, "fetchall"):
                if is_alchemy := (self.driver_name == "sqlalchemy"):
                    if hasattr(self.result, "cursor"):
                        cursor_desc = [
                            (d[0], d[1:]) for d in self.result.cursor.description
                        ]
                    elif hasattr(self.result, "_metadata"):
                        cursor_desc = [(k, None) for k in self.result._metadata.keys]
                    else:
                        msg = f"Unable to determine metadata from query result; {self.result!r}"
                        raise ValueError(msg)

                elif hasattr(self.result, "description"):
                    cursor_desc = [(d[0], d[1:]) for d in self.result.description]

View on GitHub (pinned to df599052da)

Solutions

Set a concrete batch_size alongside iter_batches, e.g. batch_size=10_000
Read everything at once (iter_batches=False) when memory permits
For cursor-level control, iterate manually with cursor.fetchmany(n) and build pl.DataFrame per chunk

Example fix

# before
frames = pl.read_database(query, connection=cursor, iter_batches=True)

# after
frames = pl.read_database(query, connection=cursor, iter_batches=True,
                          batch_size=10_000)
for df in frames:
    process(df)

Defensive patterns

Strategy: validation

Validate before calling

if iter_batches:
    assert batch_size and batch_size > 0, (
        'iter_batches requires an explicit positive batch_size'
    )
frames = pl.read_database(query, connection=cursor,
                          iter_batches=iter_batches, batch_size=batch_size)

Prevention

Always pass a concrete batch_size wherever iter_batches=True appears
Wrap read_database in one team utility that enforces the iter_batches/batch_size invariant
Size batches with fetchmany in mind: large enough for throughput, small enough to bound memory

When it happens

Trigger: pl.read_database('SELECT ...', connection=<plain DBAPI cursor>, iter_batches=True) with batch_size=None (default) or 0; drivers without Arrow registry entries falling back to row-wise fetching.

Common situations: Streaming large tables via sqlite3/psycopg2 cursors; shared helper functions where iter_batches is a parameter but batch_size was never wired through; tutorials that show iter_batches without the required companion argument.

Related errors

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/93979b60326f7721. Report an issue: GitHub.