cocoindex-io/cocoindex · error · DeserializationError

Failed to deserialize pickle payload ({_error_context()})

Error message

Failed to deserialize pickle payload ({_error_context()})

What it means

Raised when a pickle-routed payload (routing byte 0x80) cannot be unpickled through cocoindex's restricted unpickler. Serialization deliberately limits unpickling to an allow-list of builtin/stdlib types plus classes registered via `unpickle_safe` / `serialize_by_pickle`; any other global reference, or any malformed/truncated pickle bytes, causes the load to fail and is wrapped in this DeserializationError with the type hint and source label as context.

Source

Thrown at python/cocoindex/_internal/serde.py:429

                        if pydantic_adapter is None:
                            import pydantic

                            pydantic_adapter = pydantic.TypeAdapter(type_hint)
                raw = msgspec.msgpack.decode(mv[1:], ext_hook=_ext_hook)
                if type_hint is Any:
                    return raw
                return pydantic_adapter.validate_python(raw)
            except Exception as e:
                raise DeserializationError(
                    f"Failed to deserialize pydantic payload ({_error_context()})"
                ) from e

        # C: Pickle (legacy and @serialize_by_pickle)
        if routing_byte == 0x80:
            try:
                return _RestrictedUnpickler(io.BytesIO(bytes(mv))).load()
            except Exception as e:
                raise DeserializationError(
                    f"Failed to deserialize pickle payload ({_error_context()})"
                ) from e

        raise DeserializationError(
            f"Unknown routing byte: {routing_byte:#x} ({_error_context()})"
        )

    return _deserialize


# ---------------------------------------------------------------------------
# Top-level serialize / deserialize
# ---------------------------------------------------------------------------


def serialize(value: Any) -> bytes:
    """Serialize a value using the routing-byte protocol (C → B → A priority)."""
    # C: Explicit pickle (user opted in — highest priority)

View on GitHub (pinned to e84aa99b32)

Solutions

1. Read `e.__cause__`: if 'Forbidden global during unpickling: module.Name', decorate the class with @cocoindex.unpickle_safe (or @serialize_by_pickle) so it is allow-listed, then re-serialize
2. Restore the original module path/qualname of the pickled class, or re-generate the payload after the refactor
3. Install the missing package that defines the pickled class in the current environment
4. If bytes are truncated/corrupted, delete the stale persisted state and regenerate it

Example fix

// before
class MyMatrix: ...  # not registered -> Forbidden global during unpickling
// after
import cocoindex as coco
from cocoindex._internal.serde import serialize_by_pickle
@serialize_by_pickle
class MyMatrix: ...  # registered as unpickle-safe and pickled on serialize

from cocoindex._internal.serde import _UNPICKLE_SAFE_GLOBALS
def type_is_unpickle_safe(cls: type) -> bool:
    return (cls.__module__, cls.__qualname__) in _UNPICKLE_SAFE_GLOBALS
# call before serializing: assert type_is_unpickle_safe(MyClass)

try:
    value = serde.deserialize(data, type_hint=MyType)
except serde.DeserializationError as e:
    if isinstance(e.__cause__, pickle.UnpicklingError):
        value = rebuild_payload_manually()
    else:
        raise

When it happens

Trigger: `deserialize(data, type_hint)` with 0x80-routed bytes where: the pickled object references a class not in the unpickle allow-list ('Forbidden global during unpickling'); the class was renamed/moved between serialize and deserialize; the data is truncated or corrupted; or the pickle was produced with objects from a module not importable in the current environment.

Common situations: Serializing a custom class then refactoring its module/qualname before the memoized value is read; deserializing on a machine where the pickled class's package is not installed; numpy version changes altering internal ndarray reconstruction globals; payloads saved under a virtualenv where the class differs.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• @serialize_by_pickle on {cls.__qualname__} (a {'dataclass' i
• Unsupported type for memoization key: {type(obj)!r}. Provide
• Cannot serialize {type(obj).__name__}
• Unknown extension code: {code}
• Cannot deserialize msgspec payload ({_error_context()})

AI-assisted analysis of cocoindex-io/cocoindex@e84aa99b32 (2026-09-08). Data as JSON: /api/errors/b8b4edeb55764c61. Report an issue: GitHub.