cocoindex-io/cocoindex · error · ValueError

Unknown extension code: {code}

Error message

Unknown extension code: {code}

What it means

Raised by the msgspec extension hook `_ext_hook` in cocoindex's serde layer. CocoIndex encodes custom types inside msgpack payloads as msgspec extension blobs, all of which must use extension code 100 (its pickle quarantine channel). If a msgpack payload contains an extension record with any other code, the decoder cannot interpret it and raises this ValueError, indicating the payload was not produced by cocoindex's serializer or was corrupted/hand-crafted.

Source

Thrown at python/cocoindex/_internal/serde.py:296

    # B: Bridge Pydantic into msgspec
    if _is_pydantic_instance(obj):
        return obj.model_dump(mode="json")
    raise NotImplementedError(f"Cannot serialize {type(obj).__name__}")


_msgspec_encoder = msgspec.msgpack.Encoder(enc_hook=_enc_hook)


# ---------------------------------------------------------------------------
# Deserialization hooks
# ---------------------------------------------------------------------------


def _ext_hook(code: int, data: memoryview) -> Any:  # type: ignore[type-arg]
    """Un-quarantine pickle inside msgspec payloads."""
    if code == 100:
        return _RestrictedUnpickler(io.BytesIO(bytes(data))).load()
    raise ValueError(f"Unknown extension code: {code}")


def _dec_hook(type_hint: Any, obj: Any) -> Any:
    """Handle custom types during msgspec decoding.

    Called when msgspec encounters a type it doesn't natively support.
    Only two cases reach here:

    1. Pydantic models — enc_hook serialized via model_dump(mode="json"),
       so *obj* is a dict that needs model_validate to reconstruct.
    2. Pickle-quarantined values — ext_hook already reconstructed the
       correct object; just pass through.
    """
    if _is_pydantic_model_type(type_hint):
        return type_hint.model_validate(obj)
    return obj

View on GitHub (pinned to e84aa99b32)

Solutions

1. Regenerate the payload with cocoindex's own `serialize()` so extension values are encoded with code 100
2. Verify the bytes being deserialized actually came from this library/version (check for stale or foreign persisted state files and delete/rebuild them)
3. Ensure you are deserializing the correct field — a corrupted offset can make non-ext bytes be read as an ext record
4. If you craft msgpack in tests, wrap custom objects with msgspec.msgpack.Ext(100, pickle_bytes) instead of other codes

Example fix

// before (hand-crafted payload with wrong ext code)
msgpack.encode({"x": Ext(7, b"...")})  # -> Unknown extension code: 7
// after
from cocoindex._internal import serde
payload = serde.serialize(my_value)  # ext values encoded with code 100
serde.deserialize(payload, type_hint=MyType)

# only safe if you know the producer
if data[0] != 0x01:
    raise ValueError("not a msgspec-routed payload; ext codes not applicable")

def is_cocoindex_payload(data: bytes) -> bool:
    return bool(data) and data[0] in (0x01, 0x02, 0x80)

try:
    value = serde.deserialize(data, type_hint=MyType)
except serde.DeserializationError as e:
    if "Unknown extension code" in str(e.__cause__):
        data = regenerate_payload()  # re-serialize with cocoindex.serialize
        value = serde.deserialize(data, type_hint=MyType)
    else:
        raise

When it happens

Trigger: Decoding a msgpack buffer (via `deserialize`, `make_deserialize_fn`, or a msgspec Decoder built with ext_hook=_ext_hook) whose bytes contain a msgpack ext extension with a code other than 100 — e.g. data serialized by another library's msgpack encoder that emitted ext records, or a payload hand-modified/corrupted.

Common situations: Feeding payloads produced by a different msgpack encoder into cocoindex's deserializer; mixing versions where an older/experimental cocoindex build used a different ext code; manual binary editing of persisted state (memoization/state files); tests crafting raw msgpack bytes with custom ext codes.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• Cannot serialize {type(obj).__name__}
• Cannot deserialize msgspec payload ({_error_context()})
• Failed to deserialize msgspec payload ({_error_context()})
• Failed to deserialize pydantic payload ({_error_context()})
• Failed to deserialize pickle payload ({_error_context()})

AI-assisted analysis of cocoindex-io/cocoindex@e84aa99b32 (2026-09-08). Data as JSON: /api/errors/fd8a6d1fe2f01558. Report an issue: GitHub.