pola-rs/polars · error · ValueError

cannot parse Python data type

Error message

cannot parse Python data type {dtype!r} into Arrow data type

What it means

py_type_to_arrow_type maps a Python builtin type to a pyarrow DataType via PY_TYPE_TO_ARROW_TYPE; unknown types raise ValueError. It is the Python-type -> Arrow direction of the datatype bridge used when exporting/annotating Arrow schemas.

Solutions

Map the type to a supported one first (int, float, str, bool, bytes, datetime.date, datetime.datetime, datetime.timedelta).
Use pyarrow directly: pa.array(values) infers the Arrow type for arbitrary types.
Upgrade polars to get a broader PY_TYPE_TO_ARROW_TYPE table.

Example fix

// before
py_type_to_arrow_type(mycustomlib.Foo)
// after
py_type_to_arrow_type(str)  # or let pa.array(values) infer

Defensive patterns

Strategy: type-guard

Validate before calling

ARROW_MAPPABLE = {int, float, str, bool, bytes, list, dict}
def arrow_mappable(t: type) -> bool:
    return t in ARROW_MAPPABLE or getattr(t, '__module__', '').startswith('datetime')

Type guard

def is_arrow_mappable_type(t: object) -> TypeGuard[type[int | float | str | bool | bytes]]:
    return isinstance(t, type) and t in {int, float, str, bool, bytes}

Try / catch

try:
    arrow_type = py_type_to_arrow_type(py_t)
except ValueError:
    arrow_type = pa.array([], dtype=object).type  # let pyarrow infer instead

Prevention

Restrict schema annotations to Python types polars maps to Arrow
Let pyarrow infer types from sample values for exotic types
Normalize user-supplied type hints at the config boundary

When it happens

Trigger: Calling py_type_to_arrow_type with a type not in the mapping, e.g. bytes on versions lacking a binary entry, Decimal, NoneType, or custom/user classes; passing a polars dtype where a Python type was expected.

Common situations: Building Arrow schemas from user-supplied Python type annotations (pydantic/dataclass fields with exotic types); version skew where the mapping table lacks a newer type.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of pola-rs/polars@fe841f959e (2026-09-18). Data as JSON: /api/errors/f601cc87739a3caa. Report an issue: GitHub.

Appendix: source

Thrown at py-polars/src/polars/datatypes/convert.py:288


def dtype_to_py_type(dtype: PolarsDataType) -> PythonDataType:
    """Convert a Polars dtype to a Python dtype."""
    try:
        dtype = dtype.base_type()
        return DataTypeMappings.DTYPE_TO_PY_TYPE[dtype]
    except KeyError:  # pragma: no cover
        msg = f"conversion of polars data type {dtype!r} to Python type not implemented"
        raise NotImplementedError(msg) from None


def py_type_to_arrow_type(dtype: PythonDataType) -> pa.DataType:
    """Convert a Python dtype to an Arrow dtype."""
    try:
        return DataTypeMappings.PY_TYPE_TO_ARROW_TYPE[dtype]
    except KeyError:  # pragma: no cover
        msg = f"cannot parse Python data type {dtype!r} into Arrow data type"
        raise ValueError(msg) from None


def dtype_short_repr_to_dtype(dtype_string: str | None) -> PolarsDataType | None:
    """Map a PolarsDataType short repr (eg: 'i64', 'list[str]') back into a dtype."""
    if dtype_string is None:
        return None

    m = re.match(r"^(\w+)(?:\[(.+)\])?$", dtype_string)
    if m is None:
        return None

    dtype_base, subtype = m.groups()
    dtype = DataTypeMappings.REPR_TO_DTYPE.get(dtype_base)
    if dtype and subtype:
        # TODO: further-improve handling for nested types (such as List,Struct)
        try:
            if dtype == Decimal:
                subtype = (None, int(subtype))

View on GitHub (pinned to fe841f959e)