{"record":{"id":"798ababac5b8b647","repo":"invoke-ai/InvokeAI","slug":"duplicate-keys-across-sdnq-shards-model-model-pa","errorCode":null,"errorMessage":"Duplicate keys across SDNQ shards (model={model_path}): {sorted(overlap)[:3]}","messagePattern":"Duplicate keys across SDNQ shards \\(model=(.+?)\\): (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"invokeai/backend/quantization/sdnq/loaders.py","lineNumber":248,"sourceCode":"    # Build a reverse map for dynamic-mixed-precision models. SDNQ stores\n    # ``modules_dtype_dict`` as ``{dtype_name: [list of layer keys]}``; we flip it to\n    # ``{layer_key: dtype_name}`` for O(1) lookup during the per-tensor type inference.\n    per_tensor_dtype_map: dict[str, str] = {}\n    modules_dtype_dict = quant_config.get(\"modules_dtype_dict\") or {}\n    if isinstance(modules_dtype_dict, dict):\n        for dtype_name, layer_keys in modules_dtype_dict.items():\n            if isinstance(layer_keys, list):\n                for layer_key in layer_keys:\n                    per_tensor_dtype_map[layer_key] = dtype_name\n\n    # Load and merge all safetensors shards.\n    raw_sd: dict[str, torch.Tensor] = {}\n    for shard in safetensors_files:\n        shard_sd = load_file(shard)\n        # Detect accidental key collisions between shards — would indicate a corrupted bundle.\n        overlap = set(shard_sd).intersection(raw_sd)\n        if overlap:\n            raise ValueError(f\"Duplicate keys across SDNQ shards (model={model_path}): {sorted(overlap)[:3]}\")\n        raw_sd.update(shard_sd)\n\n    # Group related tensors (weight, scale, zero_point, svd_up, svd_down)\n    sd: dict[str, Union[SDNQTensor, torch.Tensor]] = {}\n    processed_keys: set[str] = set()\n\n    for key in raw_sd.keys():\n        if key in processed_keys:\n            continue\n\n        # Check if this is a base weight tensor\n        if key.endswith(\".weight\"):\n            base_key = key[:-7]  # Remove \".weight\"\n\n            weight = raw_sd[key]\n            scale_key = f\"{base_key}.scale\"\n            zero_point_key = f\"{base_key}.zero_point\"\n            svd_up_key = f\"{base_key}.svd_up\"","sourceCodeStart":230,"sourceCodeEnd":266,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/quantization/sdnq/loaders.py#L230-L266","documentation":"When merging sharded SDNQ safetensors files, any key appearing in more than one shard indicates a corrupted or mis-assembled bundle, so the loader raises ValueError showing the first few duplicated keys. Clean bundles partition keys disjointly across shards.","triggerScenarios":"Loading a multi-shard model where two *-NNNNN-of-MMMMM.safetensors files contain the same tensor key — duplicated shards, a shard copied over another, or concatenated mismatched downloads.","commonSituations":"Resume-interrupted downloads producing duplicate/overlapping files; manually mixing shards from different revisions; mis-numbered shard filenames after a manual copy.","solutions":["Re-download all shards from a single consistent revision and overwrite existing files","Verify shard filenames/counts match the model's index (each NNNNN-of-MMMMM appears exactly once)","Compare file hashes against the repo manifest to find the corrupted duplicate"],"exampleFix":"// before\nfiles = sorted(path.glob(\"*.safetensors\"))  # contains model-00001 twice via copy\n// after\nfiles = sorted(set(path.glob(\"*.safetensors\")))  # after removing duplicate shard copies","handlingStrategy":"validation","validationCode":"files = sorted(Path(model_path).glob(\"*.safetensors\"))\nseen: set = set()\nfor f in files:\n    keys = set(load_file(f).keys())\n    dup = keys & seen\n    assert not dup, f\"duplicate keys across shards: {sorted(dup)[:3]}\"\n    seen |= keys","typeGuard":"def shards_are_disjoint(shard_key_sets: list[set]) -> bool:\n    seen: set = set()\n    for ks in shard_key_sets:\n        if ks & seen:\n            return False\n        seen |= ks\n    return True","tryCatchPattern":"try:\n    model = _load_sdnq_transformer_checkpoint(path)\nexcept ValueError as e:\n    if \"Duplicate keys\" in str(e):\n        verify_and_redownload_shards(path)\n    raise","preventionTips":["Download all shards from one revision in one operation","Validate shard filenames follow *-NNNNN-of-MMMMM with no repeats","Checksum shards against the repo index before loading"],"tags":["valueerror","corrupt-model","sharded-checkpoint"],"backgroundTag":"duplicate-checkpoint-shard-keys","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}