{"record":{"id":"3acb05c00e1402d6","repo":"lancedb/lancedb","slug":"shuffle-seed-mismatch-checkpoint-has-state-shuf","errorCode":null,"errorMessage":"shuffle_seed mismatch: checkpoint has {state['shuffle_seed']}, current dataset has {self._shuffle_seed}","messagePattern":"shuffle_seed mismatch: checkpoint has (.+?), current dataset has (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/python/lancedb/streaming.py","lineNumber":1559,"sourceCode":"        )\n        self._consumer_checkpoint_requires_uniform |= require_uniform\n\n    def load_state_dict(self, state: dict) -> None:\n        \"\"\"Resume from a previously snapshotted state.\n\n        Raises ``ValueError`` if ``num_splits`` or ``shuffle_seed`` differ\n        from the checkpoint, since a different split structure or shuffle order\n        makes mid-epoch resumption meaningless.  Packed checkpoints\n        pin ``pack_sequences``, ``eos_id``, ``pad_id``,\n        ``blocks_per_epoch``, and ``epoch``.\n        \"\"\"\n        if state[\"num_splits\"] != self._num_splits:\n            raise ValueError(\n                f\"num_splits mismatch: checkpoint has {state['num_splits']}, \"\n                f\"current dataset has {self._num_splits}\"\n            )\n        if state[\"shuffle_seed\"] != self._shuffle_seed:\n            raise ValueError(\n                f\"shuffle_seed mismatch: checkpoint has {state['shuffle_seed']}, \"\n                f\"current dataset has {self._shuffle_seed}\"\n            )\n        self._consumer_checkpoint_requires_uniform = False\n\n        if \"pack_buffers\" in state or self._pack_sequences is not None:\n            for key in (\n                \"pack_sequences\",\n                \"eos_id\",\n                \"pad_id\",\n                \"blocks_per_epoch\",\n                \"epoch\",\n            ):\n                ours = getattr(self, f\"_{key}\")\n                if state.get(key) != ours:\n                    raise ValueError(\n                        f\"{key} mismatch: checkpoint has {state.get(key)}, \"\n                        f\"current dataset has {ours}\"","sourceCodeStart":1541,"sourceCodeEnd":1577,"githubUrl":"https://github.com/lancedb/lancedb/blob/c7b051aff7039333a3f61b79217246c27676806a/python/python/lancedb/streaming.py#L1541-L1577","documentation":"load_state_dict also pins shuffle_seed: resuming with a different seed raises ValueError because the shuffle permutation differs, making saved positions_consumed_per_split point at unrelated rows. The docstring explicitly documents this constraint.","triggerScenarios":"Calling load_state_dict with a checkpoint whose shuffle_seed differs from the dataset's shuffle_seed constructor argument — typically after recreating the dataset without passing the original seed (defaulting to a new random seed).","commonSituations":"Rebuilding the dataset in a new script/process without persisting and replaying shuffle_seed; changing the seed intentionally between epochs but trying to resume mid-epoch.","solutions":["Pass the checkpoint's shuffle_seed to the StreamingDataset constructor before calling load_state_dict.","Persist shuffle_seed alongside the checkpoint file and restore it on resume.","If the seed intentionally changed, restart from the beginning of the epoch rather than resuming."],"exampleFix":"// before\nds = StreamingDataset(table)  # random seed\nds.load_state_dict(state)  # ValueError: shuffle_seed mismatch\n// after\nds = StreamingDataset(table, shuffle_seed=state['shuffle_seed'])\nds.load_state_dict(state)","handlingStrategy":"validation","validationCode":"assert saved_state[\"shuffle_seed\"] == ds._shuffle_seed, (\n    \"recreate the dataset with shuffle_seed=%r\" % saved_state[\"shuffle_seed\"]\n)","typeGuard":null,"tryCatchPattern":"try:\n    ds.load_state_dict(state)\nexcept ValueError as e:\n    if \"shuffle_seed mismatch\" in str(e):\n        seed = state[\"shuffle_seed\"]\n        ds = StreamingDataset(table, shuffle_seed=seed, num_splits=ds._num_splits)\n        ds.load_state_dict(state)\n    else:\n        raise","preventionTips":["Always pass shuffle_seed explicitly (never rely on the random default) and store it with the checkpoint.","Use a fixed seed in configs so experiments are reproducible across processes.","Construct the resume dataset from the checkpoint's config dict, not from freshly written code."],"tags":["python","checkpoint","shuffle","resume"],"backgroundTag":"invalid-config-value","analyzedSha":"c7b051aff7039333a3f61b79217246c27676806a","analyzedAt":"2026-09-08T23:42:37.579Z","contentChangedAt":"2026-09-08T23:42:37.579Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}