pola-rs/polars · error · ValueError

cannot parse Python data type {dtype!r} into Arrow data type

Error message

cannot parse Python data type {dtype!r} into Arrow data type

What it means

py_type_to_arrow_type maps a Python type to a pyarrow DataType through a fixed dictionary. A Python type outside that dictionary — complex, custom classes, numpy scalar types instead of Python types — raises ValueError. It means schema inference was handed a Python type polars cannot express in Arrow.

Source

Thrown at py-polars/src/polars/datatypes/convert.py:279

def dtype_to_py_type(dtype: PolarsDataType) -> PythonDataType:
    """Convert a Polars dtype to a Python dtype."""
    try:
        dtype = dtype.base_type()
        return DataTypeMappings.DTYPE_TO_PY_TYPE[dtype]
    except KeyError:  # pragma: no cover
        msg = f"conversion of polars data type {dtype!r} to Python type not implemented"
        raise NotImplementedError(msg) from None


def py_type_to_arrow_type(dtype: PythonDataType) -> pa.DataType:
    """Convert a Python dtype to an Arrow dtype."""
    try:
        return DataTypeMappings.PY_TYPE_TO_ARROW_TYPE[dtype]
    except KeyError:  # pragma: no cover
        msg = f"cannot parse Python data type {dtype!r} into Arrow data type"
        raise ValueError(msg) from None


def dtype_short_repr_to_dtype(dtype_string: str | None) -> PolarsDataType | None:
    """Map a PolarsDataType short repr (eg: 'i64', 'list[str]') back into a dtype."""
    if dtype_string is None:
        return None

    m = re.match(r"^(\w+)(?:\[(.+)\])?$", dtype_string)
    if m is None:
        return None

    dtype_base, subtype = m.groups()
    dtype = DataTypeMappings.REPR_TO_DTYPE.get(dtype_base)
    if dtype and subtype:
        # TODO: further-improve handling for nested types (such as List,Struct)
        try:
            if dtype == Decimal:
                subtype = (None, int(subtype))

View on GitHub (pinned to df599052da)

Solutions

1. Pass plain Python types (int, float, str, bool, bytes, datetime.datetime, datetime.date, datetime.time, datetime.timedelta)
2. Convert numpy scalar types to Python types first (np.int64 -> int)
3. For custom classes, map them explicitly to a supported type (usually str) before schema creation

Example fix

# before
arrow_t = py_type_to_arrow_type(complex)  # ValueError

# after
class Money:
    ...
arrow_t = py_type_to_arrow_type(str)  # serialise Money as its string form

import datetime as dt

SUPPORTED_PY_TYPES = {bool, int, float, str, bytes, dt.date, dt.datetime, dt.time, dt.timedelta}

if py_type not in SUPPORTED_PY_TYPES:
    raise ValueError(f'unsupported Python type for Arrow conversion: {py_type!r}')
arrow_t = py_type_to_arrow_type(py_type)

import datetime as dt

def is_arrow_mappable_py_type(t: type) -> bool:
    return t in {bool, int, float, str, bytes, dt.date, dt.datetime, dt.time, dt.timedelta}

from polars.datatypes.convert import py_type_to_arrow_type

try:
    arrow_t = py_type_to_arrow_type(py_type)
except ValueError:
    arrow_t = pa.string()  # degrade custom/exotic types to string

When it happens

Trigger: Calling polars.datatypes.convert.py_type_to_arrow_type with unsupported types: complex, user-defined classes, np.int64/np.float32 used as type objects, or other types missing from PY_TYPE_TO_ARROW_TYPE.

Common situations: Building a pyarrow schema from type hints that include complex numbers or custom classes; passing numpy scalar types where Python types were intended; generic schema-inference utilities.

Related errors

• dimensions of columns arg must match data dimensions
• `schema_overrides` should be of type list or dict, got {qual
• dtypes must be fully-specified, got: {tp!r}
• not implemented
• not implemented

AI-assisted analysis of pola-rs/polars@df599052da (2026-08-16). Data as JSON: /api/errors/f601cc87739a3caa. Report an issue: GitHub.