{"record":{"id":"f44b1b472917439b","repo":"pandas-dev/pandas","slug":"disallowed-deserialization-of-arrow-py-extension","errorCode":null,"errorMessage":"Disallowed deserialization of 'arrow.py_extension_type':\nstorage_type = {storage_type}\nserialized = {serialized}\npickle disassembly:\\n{pickle_disassembly}\n\nReading of untrusted Parquet or Feather files with a PyExtensionType column\nallows arbitrary code execution.\nIf you trust this file, you can enable reading the extension type by one of:\n\n- upgrading to pyarrow >= 14.0.1, and call `pa.PyExtensionType.set_auto_load(True)`\n- install pyarrow-hotfix (`pip install pyarrow-hotfix`) and disable it by running\n  `import pyarrow_hotfix; pyarrow_hotfix.uninstall()`\n\nWe strongly recommend updating your Parquet/Feather files to use extension types\nderived from `pyarrow.ExtensionType` instead, and register this type explicitly.","messagePattern":"Disallowed deserialization of 'arrow\\.py_extension_type':\nstorage_type = \\{storage_type\\}\nserialized = \\{serialized\\}\npickle disassembly:\\\\n\\{pickle_disassembly\\}\n\nReading of untrusted Parquet or Feather files with a PyExtensionType column\nallows arbitrary code execution\\.\nIf you trust this file, you can enable reading the extension type by one of:\n\n- upgrading to pyarrow >= 14\\.0\\.1, and call `pa\\.PyExtensionType\\.set_auto_load\\(True\\)`\n- install pyarrow-hotfix \\(`pip install pyarrow-hotfix`\\) and disable it by running\n  `import pyarrow_hotfix; pyarrow_hotfix\\.uninstall\\(\\)`\n\nWe strongly recommend updating your Parquet/Feather files to use extension types\nderived from `pyarrow\\.ExtensionType` instead, and register this type explicitly\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"pandas/core/arrays/arrow/extension_types.py","lineNumber":158,"sourceCode":"    if not pa_version_under14p1:\n        return\n\n    # if https://github.com/pitrou/pyarrow-hotfix was installed and enabled\n    if getattr(pyarrow, \"_hotfix_installed\", False):\n        return\n\n    class ForbiddenExtensionType(pyarrow.ExtensionType):\n        def __arrow_ext_serialize__(self) -> bytes:\n            return b\"\"\n\n        @classmethod\n        def __arrow_ext_deserialize__(cls, storage_type, serialized):\n            import io\n            import pickletools\n\n            out = io.StringIO()\n            pickletools.dis(serialized, out)\n            raise RuntimeError(\n                _ERROR_MSG.format(\n                    storage_type=storage_type,\n                    serialized=serialized,\n                    pickle_disassembly=out.getvalue(),\n                )\n            )\n\n    pyarrow.unregister_extension_type(\"arrow.py_extension_type\")\n    pyarrow.register_extension_type(\n        ForbiddenExtensionType(pyarrow.null(), \"arrow.py_extension_type\")\n    )\n\n    pyarrow._hotfix_installed = True\n\n\npatch_pyarrow()\n","sourceCodeStart":140,"sourceCodeEnd":175,"githubUrl":"https://github.com/pandas-dev/pandas/blob/3b7651241d4da534b3559b60ef128e1c34f54116/pandas/core/arrays/arrow/extension_types.py#L140-L175","documentation":"Raised (as RuntimeError) when pandas detects an attempt to deserialize a Parquet/Feather/IPC column whose logical type is pyarrow's legacy `PyExtensionType` ('arrow.py_extension_type'). Pandas deliberately registers a ForbiddenExtensionType whose deserialize hook rejects the pickle payload, printing its pickle disassembly. This blocks CVE-2023-47248-style arbitrary code execution from untrusted files. The message lists remediation: upgrade pyarrow (>=14.0.1) and opt in via PyExtensionType.set_auto_load, or install pyarrow-hotfix, and migrate files to ExtensionType-based types.","triggerScenarios":"Reading a Parquet/Feather file produced by an older pyarrow that serialized a PyExtensionType column; `pd.read_parquet(...)` or `pa.ipc.open_stream(...).read_pandas()` on such a file triggers the forbidden deserialize hook.","commonSituations":"Loading legacy Arrow files from untrusted or third-party sources; CI reading fixtures written with pyarrow < 14; sharing notebooks that depend on pickled Python extension types.","solutions":["Do NOT blindly enable loading for untrusted files — that re-enables arbitrary code execution. If and only if you trust the source: upgrade to pyarrow >= 14.0.1 and call `pa.PyExtensionType.set_auto_load(True)` before reading.","Re-write the file using a type derived from pyarrow.ExtensionType (registered explicitly) so no Python-pickle payload is involved, or store the column as a plain storage type and reconstruct the extension on read.","If you cannot upgrade pyarrow, `pip install pyarrow-hotfix` and (for trusted files only) `import pyarrow_hotfix; pyarrow_hotfix.uninstall()`."],"exampleFix":"# before (untrusted file rejected)\npd.read_parquet('legacy_extension.parquet')\n# after — TRUSTED source only\nimport pyarrow as pa\npa.PyExtensionType.set_auto_load(True)\npd.read_parquet('legacy_extension.parquet')","handlingStrategy":"validation","validationCode":"# Treat any unknown extension type as untrusted by default\nimport pyarrow as pa\nschema = pa.parquet.read_schema(path)\nfor field in schema:\n    if str(field.type) == 'extension<arrow.py_extension_type>':\n        raise SecurityError(f\"{path} contains a PyExtensionType column; refusing to load without explicit trust\")","typeGuard":"def is_safe_arrow_type(t) -> bool:\n    import pyarrow as pa\n    return str(t) != 'extension<arrow.py_extension_type>'","tryCatchPattern":"try:\n    df = pd.read_parquet(path)\nexcept RuntimeError as e:\n    if \"py_extension_type\" in str(e):\n        # ONLY for trusted sources:\n        # pa.PyExtensionType.set_auto_load(True); df = pd.read_parquet(path)\n        raise SecurityError('Refusing to deserialize untrusted PyExtensionType')\n    raise","preventionTips":["Never enable PyExtensionType auto-load for untrusted files","Inspect schemas with pa.parquet.read_schema before reading","Migrate legacy files to registered ExtensionType-based types","Keep pyarrow >= 14.0.1 and apply the hotfix"],"tags":["security","pyarrow","parquet","feather","pickle","rce","deserialization"],"backgroundTag":null,"analyzedSha":"3b7651241d4da534b3559b60ef128e1c34f54116","analyzedAt":"2026-08-11T22:10:44.015Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}