pandas-dev/pandas · critical · RuntimeError

Disallowed deserialization of…

Error message

Disallowed deserialization of 'arrow.py_extension_type':
storage_type = {storage_type}
serialized = {serialized}
pickle disassembly:\n{pickle_disassembly}

Reading of untrusted Parquet or Feather files with a PyExtensionType column
allows arbitrary code execution.
If you trust this file, you can enable reading the extension type by one of:

- upgrading to pyarrow >= 14.0.1, and call `pa.PyExtensionType.set_auto_load(True)`
- install pyarrow-hotfix (`pip install pyarrow-hotfix`) and disable it by running
  `import pyarrow_hotfix; pyarrow_hotfix.uninstall()`

We strongly recommend updating your Parquet/Feather files to use extension types
derived from `pyarrow.ExtensionType` instead, and register this type explicitly.

What it means

Raised (as RuntimeError) when pandas detects an attempt to deserialize a Parquet/Feather/IPC column whose logical type is pyarrow's legacy `PyExtensionType` ('arrow.py_extension_type'). Pandas deliberately registers a ForbiddenExtensionType whose deserialize hook rejects the pickle payload, printing its pickle disassembly. This blocks CVE-2023-47248-style arbitrary code execution from untrusted files. The message lists remediation: upgrade pyarrow (>=14.0.1) and opt in via PyExtensionType.set_auto_load, or install pyarrow-hotfix, and migrate files to ExtensionType-based types.

Solutions

  1. Do NOT blindly enable loading for untrusted files — that re-enables arbitrary code execution. If and only if you trust the source: upgrade to pyarrow >= 14.0.1 and call `pa.PyExtensionType.set_auto_load(True)` before reading.
  2. Re-write the file using a type derived from pyarrow.ExtensionType (registered explicitly) so no Python-pickle payload is involved, or store the column as a plain storage type and reconstruct the extension on read.
  3. If you cannot upgrade pyarrow, `pip install pyarrow-hotfix` and (for trusted files only) `import pyarrow_hotfix; pyarrow_hotfix.uninstall()`.

Example fix

# before (untrusted file rejected)
pd.read_parquet('legacy_extension.parquet')
# after — TRUSTED source only
import pyarrow as pa
pa.PyExtensionType.set_auto_load(True)
pd.read_parquet('legacy_extension.parquet')
Defensive patterns

Strategy: validation

Validate before calling

# Treat any unknown extension type as untrusted by default
import pyarrow as pa
schema = pa.parquet.read_schema(path)
for field in schema:
    if str(field.type) == 'extension<arrow.py_extension_type>':
        raise SecurityError(f"{path} contains a PyExtensionType column; refusing to load without explicit trust")

Type guard

def is_safe_arrow_type(t) -> bool:
    import pyarrow as pa
    return str(t) != 'extension<arrow.py_extension_type>'

Try / catch

try:
    df = pd.read_parquet(path)
except RuntimeError as e:
    if "py_extension_type" in str(e):
        # ONLY for trusted sources:
        # pa.PyExtensionType.set_auto_load(True); df = pd.read_parquet(path)
        raise SecurityError('Refusing to deserialize untrusted PyExtensionType')
    raise

Prevention

When it happens

Trigger: Reading a Parquet/Feather file produced by an older pyarrow that serialized a PyExtensionType column; `pd.read_parquet(...)` or `pa.ipc.open_stream(...).read_pandas()` on such a file triggers the forbidden deserialize hook.

Common situations: Loading legacy Arrow files from untrusted or third-party sources; CI reading fixtures written with pyarrow < 14; sharing notebooks that depend on pickled Python extension types.

Related errors

AI-assisted analysis of pandas-dev/pandas@3b7651241d (2026-08-11). Data as JSON: /api/errors/f44b1b472917439b. Report an issue: GitHub.

Appendix: source

Thrown at pandas/core/arrays/arrow/extension_types.py:158

    if not pa_version_under14p1:
        return

    # if https://github.com/pitrou/pyarrow-hotfix was installed and enabled
    if getattr(pyarrow, "_hotfix_installed", False):
        return

    class ForbiddenExtensionType(pyarrow.ExtensionType):
        def __arrow_ext_serialize__(self) -> bytes:
            return b""

        @classmethod
        def __arrow_ext_deserialize__(cls, storage_type, serialized):
            import io
            import pickletools

            out = io.StringIO()
            pickletools.dis(serialized, out)
            raise RuntimeError(
                _ERROR_MSG.format(
                    storage_type=storage_type,
                    serialized=serialized,
                    pickle_disassembly=out.getvalue(),
                )
            )

    pyarrow.unregister_extension_type("arrow.py_extension_type")
    pyarrow.register_extension_type(
        ForbiddenExtensionType(pyarrow.null(), "arrow.py_extension_type")
    )

    pyarrow._hotfix_installed = True


patch_pyarrow()

View on GitHub (pinned to 3b7651241d)