invoke-ai/InvokeAI · error · ValueError

Duplicate keys across SDNQ shards (model={model_path}): {sor

Error message

Duplicate keys across SDNQ shards (model={model_path}): {sorted(overlap)[:3]}

What it means

When merging sharded SDNQ safetensors files, any key appearing in more than one shard indicates a corrupted or mis-assembled bundle, so the loader raises ValueError showing the first few duplicated keys. Clean bundles partition keys disjointly across shards.

Source

Thrown at invokeai/backend/quantization/sdnq/loaders.py:248

    # Build a reverse map for dynamic-mixed-precision models. SDNQ stores
    # ``modules_dtype_dict`` as ``{dtype_name: [list of layer keys]}``; we flip it to
    # ``{layer_key: dtype_name}`` for O(1) lookup during the per-tensor type inference.
    per_tensor_dtype_map: dict[str, str] = {}
    modules_dtype_dict = quant_config.get("modules_dtype_dict") or {}
    if isinstance(modules_dtype_dict, dict):
        for dtype_name, layer_keys in modules_dtype_dict.items():
            if isinstance(layer_keys, list):
                for layer_key in layer_keys:
                    per_tensor_dtype_map[layer_key] = dtype_name

    # Load and merge all safetensors shards.
    raw_sd: dict[str, torch.Tensor] = {}
    for shard in safetensors_files:
        shard_sd = load_file(shard)
        # Detect accidental key collisions between shards — would indicate a corrupted bundle.
        overlap = set(shard_sd).intersection(raw_sd)
        if overlap:
            raise ValueError(f"Duplicate keys across SDNQ shards (model={model_path}): {sorted(overlap)[:3]}")
        raw_sd.update(shard_sd)

    # Group related tensors (weight, scale, zero_point, svd_up, svd_down)
    sd: dict[str, Union[SDNQTensor, torch.Tensor]] = {}
    processed_keys: set[str] = set()

    for key in raw_sd.keys():
        if key in processed_keys:
            continue

        # Check if this is a base weight tensor
        if key.endswith(".weight"):
            base_key = key[:-7]  # Remove ".weight"

            weight = raw_sd[key]
            scale_key = f"{base_key}.scale"
            zero_point_key = f"{base_key}.zero_point"
            svd_up_key = f"{base_key}.svd_up"

View on GitHub (pinned to 0b6a024f2f)

Solutions

1. Re-download all shards from a single consistent revision and overwrite existing files
2. Verify shard filenames/counts match the model's index (each NNNNN-of-MMMMM appears exactly once)
3. Compare file hashes against the repo manifest to find the corrupted duplicate

Example fix

// before
files = sorted(path.glob("*.safetensors"))  # contains model-00001 twice via copy
// after
files = sorted(set(path.glob("*.safetensors")))  # after removing duplicate shard copies

files = sorted(Path(model_path).glob("*.safetensors"))
seen: set = set()
for f in files:
    keys = set(load_file(f).keys())
    dup = keys & seen
    assert not dup, f"duplicate keys across shards: {sorted(dup)[:3]}"
    seen |= keys

def shards_are_disjoint(shard_key_sets: list[set]) -> bool:
    seen: set = set()
    for ks in shard_key_sets:
        if ks & seen:
            return False
        seen |= ks
    return True

try:
    model = _load_sdnq_transformer_checkpoint(path)
except ValueError as e:
    if "Duplicate keys" in str(e):
        verify_and_redownload_shards(path)
    raise

When it happens

Trigger: Loading a multi-shard model where two *-NNNNN-of-MMMMM.safetensors files contain the same tensor key — duplicated shards, a shard copied over another, or concatenated mismatched downloads.

Common situations: Resume-interrupted downloads producing duplicate/overlapping files; manually mixing shards from different revisions; mis-numbered shard filenames after a manual copy.

Related errors

• Invalid or expired token
• User not found or inactive
• Missing authentication credentials
• Invalid or expired authentication token
• Authentication required

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/798ababac5b8b647. Report an issue: GitHub.