cocoindex-io/cocoindex · error · DeserializationError

Failed to deserialize pydantic payload ({_error_context()})

Error message

Failed to deserialize pydantic payload ({_error_context()})

What it means

Raised when a pydantic-routed payload (routing byte 0x02) fails during the raw msgpack decode or the subsequent `pydantic.TypeAdapter.validate_python` step. CocoIndex encodes Pydantic models as msgpack dicts of their JSON dump; this error means those bytes could not be decoded into raw Python, or the resulting dict/list did not match the declared type hint per pydantic validation. The original exception is chained as cause.

Source

Thrown at python/cocoindex/_internal/serde.py:420

                raise DeserializationError(
                    f"Failed to deserialize msgspec payload ({_error_context()})"
                ) from e

        # B: Pydantic
        if routing_byte == 0x02:
            try:
                if pydantic_adapter is None:
                    with pydantic_lock:
                        if pydantic_adapter is None:
                            import pydantic

                            pydantic_adapter = pydantic.TypeAdapter(type_hint)
                raw = msgspec.msgpack.decode(mv[1:], ext_hook=_ext_hook)
                if type_hint is Any:
                    return raw
                return pydantic_adapter.validate_python(raw)
            except Exception as e:
                raise DeserializationError(
                    f"Failed to deserialize pydantic payload ({_error_context()})"
                ) from e

        # C: Pickle (legacy and @serialize_by_pickle)
        if routing_byte == 0x80:
            try:
                return _RestrictedUnpickler(io.BytesIO(bytes(mv))).load()
            except Exception as e:
                raise DeserializationError(
                    f"Failed to deserialize pickle payload ({_error_context()})"
                ) from e

        raise DeserializationError(
            f"Unknown routing byte: {routing_byte:#x} ({_error_context()})"
        )

    return _deserialize

View on GitHub (pinned to e84aa99b32)

Solutions

1. Read `e.__cause__` for the exact pydantic ValidationError and align the type hint or the model fields with what was serialized
2. Clear stale persisted state / memoization entries so payloads are regenerated with the current model definition
3. Give removed/renamed pydantic fields defaults or `Optional` types so old payloads still validate
4. Verify pydantic v2 is installed — the adapter path uses TypeAdapter/model_dump(mode='json') and fails with incompatible pydantic versions

Example fix

// before: renamed field breaks old payloads
class Row(BaseModel):
    user_name: str
// after: keep old payloads validating
class Row(BaseModel):
    user_name: str = ""
    # or migrate: user_name: str = Field(validation_alias=AliasChoices('user_name','name'))

import pydantic
try:
    pydantic.TypeAdapter(type_hint)
except Exception as e:
    raise TypeError(f"hint not pydantic-compatible: {type_hint!r}") from e

def looks_like_pydantic_payload(data: bytes) -> bool:
    return bool(data) and data[0] == 0x02

try:
    value = serde.deserialize(data, type_hint=MyModel)
except serde.DeserializationError as e:
    # e.__cause__ is the pydantic ValidationError with per-field details
    for err in getattr(e.__cause__, 'errors', lambda: [])():
        print(err['loc'], err['type'])
    value = MyModel.model_validate(new_raw_dict)

When it happens

Trigger: `deserialize(data, type_hint)` with 0x02-routed bytes where: the model fields changed since serialization (missing/extra fields failing validation); type_hint is incompatible with the decoded raw data; pydantic import or TypeAdapter construction fails inside the lazy adapter; or the bytes are corrupted/truncated msgpack.

Common situations: Evolving a Pydantic BaseModel used in a @coco.fn (removing/renaming fields with old memoized payloads persisted); storing payloads with an older pydantic v1-style model; passing a type_hint like `list[str]` where the payload contains a model dict; unpickling errors inside nested ext-quarantined values during raw decode.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• expected None{loc}, got {type(value).__name__}
• expected {tp}{loc}, got {type(value).__name__}: {value!r}
• expected tuple{loc}, got {type(value).__name__}
• expected {tp}{loc}, got {type(value).__name__}
• expected {tp.__name__}{loc}, got {type(value).__name__}: {va

AI-assisted analysis of cocoindex-io/cocoindex@e84aa99b32 (2026-09-08). Data as JSON: /api/errors/75c341c9c8b859cc. Report an issue: GitHub.