invoke-ai/InvokeAI · error · ValueError

Expected 2D embed_tokens weight tensor, got shape {embed_sha

Error message

Expected 2D embed_tokens weight tensor, got shape {embed_shape}.

What it means

Companion to the missing-key error: 'model.embed_tokens.weight' was found but its shape is not 2D, so hidden_size/vocab_size cannot be derived from it. Indicates a malformed, corrupted, or unconventionally quantized SDNQ file.

Source

Thrown at invokeai/backend/model_manager/load/model_loaders/z_image.py:1532

        layer_count = 0
        for key in sd.keys():
            if isinstance(key, str) and key.startswith("model.layers."):
                parts = key.split(".")
                if len(parts) > 2:
                    try:
                        layer_idx = int(parts[2])
                        layer_count = max(layer_count, layer_idx + 1)
                    except ValueError:
                        pass

        # Get hidden size from embed_tokens weight shape
        embed_weight = sd.get("model.embed_tokens.weight")
        if embed_weight is None:
            raise ValueError("Could not find model.embed_tokens.weight in state dict")

        embed_shape = embed_weight.shape if hasattr(embed_weight, "shape") else embed_weight.tensor_shape
        if len(embed_shape) != 2:
            raise ValueError(f"Expected 2D embed_tokens weight tensor, got shape {embed_shape}.")
        hidden_size = embed_shape[1]
        vocab_size = embed_shape[0]

        # Detect attention configuration from layer 0 weights
        q_proj_weight = sd.get("model.layers.0.self_attn.q_proj.weight")
        k_proj_weight = sd.get("model.layers.0.self_attn.k_proj.weight")
        gate_proj_weight = sd.get("model.layers.0.mlp.gate_proj.weight")

        if q_proj_weight is None or k_proj_weight is None or gate_proj_weight is None:
            raise ValueError("Could not find attention/mlp weights in state dict to determine configuration")

        q_shape = q_proj_weight.shape if hasattr(q_proj_weight, "shape") else q_proj_weight.tensor_shape
        k_shape = k_proj_weight.shape if hasattr(k_proj_weight, "shape") else k_proj_weight.tensor_shape
        gate_shape = gate_proj_weight.shape if hasattr(gate_proj_weight, "shape") else gate_proj_weight.tensor_shape

        head_dim = 128  # Standard head dimension for Qwen3 models
        num_attention_heads = q_shape[0] // head_dim
        num_kv_heads = k_shape[0] // head_dim

View on GitHub (pinned to 0b6a024f2f)

Solutions

Re-quantize the model keeping embed_tokens.weight in a standard 2D (bf16/f16) layout.
Re-download and verify file integrity.
Use a known-good SDNQ release of the Qwen3 encoder instead of a hand-converted one.
If a custom packing scheme is intentional, unpack/dequantize the tensor before this loader reads the state dict.

Example fix

// before
'model.embed_tokens.weight' shape [151669*2048] (1D packed)
// after
'model.embed_tokens.weight' shape [151669, 2048]

Defensive patterns

Strategy: validation

Validate before calling

w = sd["model.embed_tokens.weight"]
shape = w.shape if hasattr(w, "shape") else w.tensor_shape
if len(shape) != 2:
    raise ValueError(f"{path}: embed_tokens must be 2D, got {shape} — re-quantize with standard embedding layout")

Type guard

def has_2d_embed(sd: dict) -> bool:
    w = sd.get("model.embed_tokens.weight")
    if w is None:
        return False
    s = w.shape if hasattr(w, "shape") else getattr(w, "tensor_shape", None)
    return s is not None and len(s) == 2

Try / catch

try:
    model = load_text_encoder(cfg)
except ValueError as e:
    if "Expected 2D embed_tokens" in str(e):
        raise ModelIntegrityError(f"SDNQ file {cfg.path} malformed; use a known-good release.") from e
    raise

Prevention

Keep embed_tokens.weight unquantized/standard 2D when quantizing.
Validate tensor shapes right after any SDNQ conversion.
Prefer official SDNQ releases over custom-packed formats.
Hash-verify large quantized downloads.

When it happens

Trigger: Loading an SDNQ model whose embedding tensor was flattened or packed to >2D by a custom quantization scheme, or whose file is truncated so tensor metadata is wrong.

Common situations: Custom/experimental SDNQ quant formats applied to the embedding layer; corrupted download; conversion bug in a third-party SDNQ exporter.

Understand the failure class

Background: Tensor shape mismatch errors ("must have shape", "expected shape ... got ..."): when tensor dimensions disagree with what an op or layer was told to expect — this error's family across 6 libraries.

Related errors

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/681985fdc720b875. Report an issue: GitHub.