lancedb/lancedb · error · ValueError

Unknown table pickle state kind

Error message

Unknown table pickle state kind: {kind}

What it means

Raised by _table_from_pickle_state (used by Permutation.__setstate__) when the deserialized state's 'kind' field is not one of 'remote', 'memory', or 'local'. This indicates the pickle payload was not produced by a recognized version of _table_to_pickle_state.

Solutions

1. Regenerate the pickle with the same lancedb version that created it (align versions across environments).
2. Ensure the Permutation was serialized via __getstate__ of a matching lancedb release, not manually constructed state.
3. If you cannot regenerate, reconstruct the Permutation from the table directly instead of unpickling.

Example fix

// before
perm = pickle.load(open("perm.pkl", "rb"))  # pickled by forked lancedb
// after
# pip install 'lancedb==<same version as producer>'
perm = pickle.load(open("perm.pkl", "rb"))

state = pickle.loads(raw) if isinstance(raw, dict) else None
if isinstance(state, dict) and state.get("kind") not in {"remote", "memory", "local"}:
    raise ValueError(f"pickle produced by incompatible lancedb version: kind={state.get('kind')!r}")

try:
    perm = pickle.load(f)
except ValueError as e:
    if "Unknown table pickle state kind" in str(e):
        raise RuntimeError("Pickle written by incompatible lancedb version; align versions and regenerate") from e
    raise

When it happens

Trigger: Unpickling a Permutation whose state dict has an unrecognized or corrupted kind value; unpickling data produced by an incompatible library version or a hand-crafted state dict.

Common situations: Pickling with a patched/forked version of lancedb and unpickling with stock lancedb; manually editing pickle state; version drift between training and serving environments.

Understand the failure class

Background: "invalid response format", "malformed payload", "missing data field": when an API returns 200 but the response shape is wrong — this error's family across 23 libraries.

Related errors

• Cannot pickle table of type
• Cannot create a permutation on split
• Cannot create a permutation on split
• Cannot remove all columns
• Cannot rename column

AI-assisted analysis of lancedb/lancedb@c7b051aff7 (2026-09-08). Data as JSON: /api/errors/e2deaa5d2d0db787. Report an issue: GitHub.

Appendix: source

Thrown at python/python/lancedb/permutation.py:414

    metadata = dict(permutation_data.schema.metadata or {})
    if metadata.pop(b"base_version", None) is None:
        return permutation_data
    metadata.pop(b"base_branch", None)
    return permutation_data.replace_schema_metadata(metadata)


def _table_from_pickle_state(state: dict[str, Any]) -> Table:
    from . import connect

    kind = state["kind"]
    if kind == "remote":
        return state["table"]
    if kind == "memory":
        return connect("memory://").create_table(state["name"], state["data"])
    if kind == "local":
        db = connect(state["uri"], storage_options=state["storage_options"])
        return db.open_table(state["name"], namespace_path=state["namespace"] or None)
    raise ValueError(f"Unknown table pickle state kind: {kind}")


class Permutation:
    """
    A Permutation is a view of a dataset that can be used as input to model training
    and evaluation.

    A Permutation fulfills the pytorch Dataset contract and is loosely modeled after the
    huggingface Dataset so it should be easy to use with existing code.

    A permutation is not a "materialized view" or copy of the underlying data.  It is
    calculated on the fly from the base table.  As a result, it is truly "lazy" and does
    not require materializing the entire dataset in memory.
    """

    def __init__(
        self,
        base_table: Table,

View on GitHub (pinned to c7b051aff7)