lancedb/lancedb · error · ValueError

shuffle_seed mismatch: checkpoint has

Error message

shuffle_seed mismatch: checkpoint has {state['shuffle_seed']}, current dataset has {self._shuffle_seed}

What it means

load_state_dict also pins shuffle_seed: resuming with a different seed raises ValueError because the shuffle permutation differs, making saved positions_consumed_per_split point at unrelated rows. The docstring explicitly documents this constraint.

Solutions

Pass the checkpoint's shuffle_seed to the StreamingDataset constructor before calling load_state_dict.
Persist shuffle_seed alongside the checkpoint file and restore it on resume.
If the seed intentionally changed, restart from the beginning of the epoch rather than resuming.

Example fix

// before
ds = StreamingDataset(table)  # random seed
ds.load_state_dict(state)  # ValueError: shuffle_seed mismatch
// after
ds = StreamingDataset(table, shuffle_seed=state['shuffle_seed'])
ds.load_state_dict(state)

Defensive patterns

Strategy: validation

Validate before calling

assert saved_state["shuffle_seed"] == ds._shuffle_seed, (
    "recreate the dataset with shuffle_seed=%r" % saved_state["shuffle_seed"]
)

Try / catch

try:
    ds.load_state_dict(state)
except ValueError as e:
    if "shuffle_seed mismatch" in str(e):
        seed = state["shuffle_seed"]
        ds = StreamingDataset(table, shuffle_seed=seed, num_splits=ds._num_splits)
        ds.load_state_dict(state)
    else:
        raise

Prevention

Always pass shuffle_seed explicitly (never rely on the random default) and store it with the checkpoint.
Use a fixed seed in configs so experiments are reproducible across processes.
Construct the resume dataset from the checkpoint's config dict, not from freshly written code.

When it happens

Trigger: Calling load_state_dict with a checkpoint whose shuffle_seed differs from the dataset's shuffle_seed constructor argument — typically after recreating the dataset without passing the original seed (defaulting to a new random seed).

Common situations: Rebuilding the dataset in a new script/process without persisting and replaying shuffle_seed; changing the seed intentionally between epochs but trying to resume mid-epoch.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of lancedb/lancedb@c7b051aff7 (2026-09-08). Data as JSON: /api/errors/3acb05c00e1402d6. Report an issue: GitHub.

Appendix: source

Thrown at python/python/lancedb/streaming.py:1559

        )
        self._consumer_checkpoint_requires_uniform |= require_uniform

    def load_state_dict(self, state: dict) -> None:
        """Resume from a previously snapshotted state.

        Raises ``ValueError`` if ``num_splits`` or ``shuffle_seed`` differ
        from the checkpoint, since a different split structure or shuffle order
        makes mid-epoch resumption meaningless.  Packed checkpoints
        pin ``pack_sequences``, ``eos_id``, ``pad_id``,
        ``blocks_per_epoch``, and ``epoch``.
        """
        if state["num_splits"] != self._num_splits:
            raise ValueError(
                f"num_splits mismatch: checkpoint has {state['num_splits']}, "
                f"current dataset has {self._num_splits}"
            )
        if state["shuffle_seed"] != self._shuffle_seed:
            raise ValueError(
                f"shuffle_seed mismatch: checkpoint has {state['shuffle_seed']}, "
                f"current dataset has {self._shuffle_seed}"
            )
        self._consumer_checkpoint_requires_uniform = False

        if "pack_buffers" in state or self._pack_sequences is not None:
            for key in (
                "pack_sequences",
                "eos_id",
                "pad_id",
                "blocks_per_epoch",
                "epoch",
            ):
                ours = getattr(self, f"_{key}")
                if state.get(key) != ours:
                    raise ValueError(
                        f"{key} mismatch: checkpoint has {state.get(key)}, "
                        f"current dataset has {ours}"

View on GitHub (pinned to c7b051aff7)