pandas-dev/pandas · critical · RuntimeError

Disallowed deserialization of 'arrow.py_extension_type': sto

Error message

Disallowed deserialization of 'arrow.py_extension_type':
storage_type = {storage_type}
serialized = {serialized}
pickle disassembly:
{pickle_disassembly}

Reading of untrusted Parquet or Feather files with a PyExtensionType column
allows arbitrary code execution.
If you trust this file, you can enable reading the extension type by one of:

- upgrading to pyarrow >= 14.0.1, and call `pa.PyExtensionType.set_auto_load(True)`
- install pyarrow-hotfix (`pip install pyarrow-hotfix`) and disable it by running
  `import pyarrow_hotfix; pyarrow_hotfix.uninstall()`

We strongly recommend updating your Parquet/Feather files to use extension types
derived from `pyarrow.ExtensionType` instead, and register this type explicitly.

What it means

Raised by the ForbiddenExtensionType injected by pandas.core.arrays.arrow.extension_types.patch_pyarrow() when deserializing a column whose pyarrow extension type is 'arrow.py_extension_type' on pyarrow < 14.0.1 (without pyarrow-hotfix). This is the pandas-side mitigation for CVE-2023-47248: deserializing a PyExtensionType column from Parquet/Feather executes arbitrary pickle code, so pandas blocks it with a RuntimeError that includes a pickle disassembly and remediation steps. It fires only on the old, vulnerable pyarrow versions; pyarrow >= 14.0.1 has its own auto-load gate.

Source

Thrown at pandas/core/arrays/arrow/extension_types.py:158

    if not pa_version_under14p1:
        return

    # if https://github.com/pitrou/pyarrow-hotfix was installed and enabled
    if getattr(pyarrow, "_hotfix_installed", False):
        return

    class ForbiddenExtensionType(pyarrow.ExtensionType):
        def __arrow_ext_serialize__(self) -> bytes:
            return b""

        @classmethod
        def __arrow_ext_deserialize__(cls, storage_type, serialized):
            import io
            import pickletools

            out = io.StringIO()
            pickletools.dis(serialized, out)
            raise RuntimeError(
                _ERROR_MSG.format(
                    storage_type=storage_type,
                    serialized=serialized,
                    pickle_disassembly=out.getvalue(),
                )
            )

    pyarrow.unregister_extension_type("arrow.py_extension_type")
    pyarrow.register_extension_type(
        ForbiddenExtensionType(pyarrow.null(), "arrow.py_extension_type")
    )

    pyarrow._hotfix_installed = True


patch_pyarrow()

View on GitHub (pinned to 71959b8cb9)

Solutions

1. Upgrade pyarrow to >= 14.0.1 (preferred); then optionally enable trusted loading via pa.PyExtensionType.set_auto_load(True).
2. If you must stay on old pyarrow, install pyarrow-hotfix (`pip install pyarrow-hotfix`) for general protection; to read a TRUSTED file, run `import pyarrow_hotfix; pyarrow_hotfix.uninstall()` (understand the risk first).
3. Rewrite the source Parquet/Feather file using extension types derived from pyarrow.ExtensionType and register them explicitly instead of PyExtensionType.
4. If the file is untrusted, do NOT bypass the guard; treat it as untrusted input and sanitize at ingestion.

Example fix

# before (triggers on pyarrow < 14.0.1)
df = pd.read_parquet("legacy_with_pyext.parquet")  # RuntimeError

# after (preferred): upgrade pyarrow then optionally allow trusted loading
# pip install -U "pyarrow>=14.0.1"
import pyarrow as pa
pa.PyExtensionType.set_auto_load(True)
df = pd.read_parquet("legacy_with_pyext.parquet")

# after (legacy pyarrow, trusted file only):
# pip install pyarrow-hotfix
import pyarrow_hotfix
pyarrow_hotfix.uninstall()
df = pd.read_parquet("legacy_with_pyext.parquet")

import pyarrow as pa
from packaging.version import Version

def safe_read_parquet(path):
    if Version(pa.__version__) < Version("14.0.1"):
        # check the file's schema for arrow.py_extension_type columns before reading
        schema = pa.parquet.read_schema(path) if hasattr(pa, "parquet") else None
        if schema is not None and "py_extension_type" in str(schema):
            raise RuntimeError(
                "Refusing to read file with PyExtensionType column on pyarrow < 14.0.1 "
                "(CVE-2023-47248). Upgrade pyarrow or only bypass with a TRUSTED file."
            )
    return pd.read_parquet(path)

import pyarrow as pa
from packaging.version import Version

def is_safe_pyarrow_version() -> bool:
    try:
        from packaging.version import Version
        return Version(pa.__version__) >= Version("14.0.1")
    except Exception:
        return False

try:
    df = pd.read_parquet(path)
except RuntimeError as e:
    if "arrow.py_extension_type" in str(e):
        # only for TRUSTED files; upgrade pyarrow or rewrite file instead
        import pyarrow_hotfix
        pyarrow_hotfix.uninstall()
        df = pd.read_parquet(path)
    else:
        raise

When it happens

Trigger: Calling pd.read_parquet / pd.read_feather on a file that contains an `arrow.py_extension_type` column, while running pyarrow < 14.0.1 and without pyarrow-hotfix installed/enabled. The file may be untrusted or simply produced by an older writer that used PyExtensionType.

Common situations: CI/CD pinned to an old pyarrow (<14.0.1); reading Parquet/Feather written by legacy pyarrow or third-party tools that registered PyExtensionType; shared datasets downloaded from untrusted sources. Security-sensitive environments (regulated, multi-tenant) must treat this as an explicit block, not a nuisance.

Related errors

• Invalid side: {side}. Side must be one of 'left', 'right', '
• invalid normalization form
• replace is not supported with a re.Pattern, callable repl, c
• contains not implemented with {flags=}
• Can only use the '.list' accessor with 'list[pyarrow]' dtype

AI-assisted analysis of pandas-dev/pandas@71959b8cb9 (2026-08-07). Data as JSON: /api/errors/7e078468d6762a15. Report an issue: GitHub.