pola-rs/polars · error · TypeError

source must implement the Arrow PyCapsule Interface (__arrow

Error message

source must implement the Arrow PyCapsule Interface (__arrow_c_stream__)

What it means

pl.scan_arrow_c_stream (py-polars/src/polars/io/arrow_c_stream.py:21) builds a LazyFrame from any object implementing the Arrow PyCapsule Interface, i.e. exposing a __arrow_c_stream__ method (e.g. a pyarrow RecordBatchReader or a nanoarrow stream). The first thing it does is hasattr(source, '__arrow_c_stream__'); anything else - a pyarrow Table, a pandas DataFrame, a path string, or a pre-PyArrow-14 object lacking the method - raises this TypeError immediately.

Source

Thrown at py-polars/src/polars/io/arrow_c_stream.py:76

    ... ]
    >>> reader = pa.RecordBatchReader.from_batches(schema, batches)
    >>> pl.scan_arrow_c_stream(reader).collect()
    shape: (5, 2)
    ┌─────┬─────┐
    │ a   ┆ b   │
    │ --- ┆ --- │
    │ i64 ┆ str │
    ╞═════╪═════╡
    │ 1   ┆ x   │
    │ 2   ┆ y   │
    │ 3   ┆ z   │
    │ 4   ┆ a   │
    │ 5   ┆ b   │
    └─────┴─────┘
    """
    if not hasattr(source, "__arrow_c_stream__"):
        msg = "source must implement the Arrow PyCapsule Interface (__arrow_c_stream__)"
        raise TypeError(msg)

    import polars._plr as plr

    reader = plr.PyArrowCStreamReader(source)

    def io_source(
        with_columns: list[str] | None,
        predicate: Expr | None,
        n_rows: int | None,
        batch_size: int | None,  # noqa: ARG001
    ) -> Iterator[DataFrame]:
        remaining = n_rows
        while (batch := reader.next_batch(with_columns)) is not None:
            df = pl.DataFrame._from_pydf(batch)
            if predicate is not None:
                df = df.filter(predicate)
            if remaining is not None:
                df = df.head(remaining)

View on GitHub (pinned to df599052da)

Solutions

Pass a capsule-capable stream object: reader = tbl.to_reader() (pyarrow >= 14) and scan that
Upgrade pyarrow to >= 14 so Table/RecordBatchReader expose __arrow_c_stream__
For non-capsule Arrow inputs use the eager pl.from_arrow(source) instead
Branch in generic ingestion code on hasattr(source, '__arrow_c_stream__') before choosing the API

Example fix

# before
lf = pl.scan_arrow_c_stream(pa.table({"a": [1]}))  # TypeError
# after
reader = pa.table({"a": [1]}).to_reader()
lf = pl.scan_arrow_c_stream(reader)

Defensive patterns

Strategy: type-guard

Validate before calling

def to_c_stream_source(obj):
    if hasattr(obj, "__arrow_c_stream__"):
        return obj
    if hasattr(obj, "to_reader"):  # pyarrow Table
        return obj.to_reader()
    raise TypeError(f"cannot adapt {type(obj).__name__} to an Arrow C stream")

Type guard

def supports_arrow_c_stream(source: object) -> bool:
    return hasattr(source, "__arrow_c_stream__") and callable(source.__arrow_c_stream__)

Try / catch

try:
    lf = pl.scan_arrow_c_stream(source)
except TypeError as e:
    if "PyCapsule" in str(e):
        lf = pl.from_arrow(source).lazy()  # eager fallback for legacy Arrow inputs
    else:
        raise

Prevention

Standardize on pyarrow >= 14 when using capsule-based APIs
Gate generic ingestion behind hasattr(source, '__arrow_c_stream__')
Remember scan_arrow_c_stream is @unstable; pin polars versions where it is used

When it happens

Trigger: pl.scan_arrow_c_stream(pa.table({'a': [1]})); passing a pandas DataFrame, numpy array, or file path; passing a pyarrow.RecordBatchReader created by pyarrow < 14, which does not yet expose the capsule method.

Common situations: Generic interop code that accepts 'anything arrow-ish'; CI pinned to an older pyarrow than the developer machine; users assuming this unstable API is a drop-in replacement for pl.from_arrow.

Related errors

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/4f753ed9b2ddaede. Report an issue: GitHub.