invoke-ai/InvokeAI · error · ValueError

Expected 2D embed_tokens weight tensor, got shape {embed_sha

Error message

Expected 2D embed_tokens weight tensor, got shape {embed_shape}. The model file may be corrupted or incompatible.

What it means

Raised when 'model.embed_tokens.weight' exists in the GGUF state dict but is not a 2D (vocab, hidden) tensor. Because GGUF tensors carry GGML shapes, the loader reads .shape (or .tensor_shape) to derive vocab_size; a malformed or quantized-off embedding tensor yields an unexpected rank, so the file is treated as corrupted/incompatible.

Source

Thrown at invokeai/backend/model_manager/load/model_loaders/z_image.py:1243

        for key in sd.keys():
            if isinstance(key, str) and key.startswith("model.layers."):
                parts = key.split(".")
                if len(parts) > 2:
                    try:
                        layer_idx = int(parts[2])
                        layer_count = max(layer_count, layer_idx + 1)
                    except ValueError:
                        pass

        # Get vocab size from embed_tokens weight shape
        embed_weight = sd.get("model.embed_tokens.weight")
        if embed_weight is None:
            raise ValueError("Could not find model.embed_tokens.weight in state dict")

        # Handle GGMLTensor shape access
        embed_shape = embed_weight.shape if hasattr(embed_weight, "shape") else embed_weight.tensor_shape
        if len(embed_shape) != 2:
            raise ValueError(
                f"Expected 2D embed_tokens weight tensor, got shape {embed_shape}. "
                "The model file may be corrupted or incompatible."
            )
        vocab_size = embed_shape[0]

        # Detect attention configuration from layer weights
        # IMPORTANT: Use layer 1 (not layer 0) because some models like FLUX 2 Klein have a special
        # first layer with different dimensions (input projection layer) while the rest of the
        # transformer layers have a different hidden_size. Using a middle layer ensures we get
        # the representative hidden_size for the bulk of the model.
        # Fall back to layer 0 if layer 1 doesn't exist.
        q_proj_weight = sd.get("model.layers.1.self_attn.q_proj.weight")
        k_proj_weight = sd.get("model.layers.1.self_attn.k_proj.weight")
        gate_proj_weight = sd.get("model.layers.1.mlp.gate_proj.weight")

        # Fall back to layer 0 if layer 1 doesn't exist (single-layer model edge case)
        if q_proj_weight is None:
            q_proj_weight = sd.get("model.layers.0.self_attn.q_proj.weight")

View on GitHub (pinned to 0b6a024f2f)

Solutions

  1. Re-download or re-convert the GGUF with an up-to-date converter that preserves 2D embedding shapes.
  2. Verify the tensor rank with gguf-dump; if it is 1D/3D, re-export from the original safetensors checkpoint.
  3. If quantization packs the tensor, load a F16/FP32 variant of the text-encoder GGUF instead.
  4. Check disk/full-download health: a truncated file can corrupt tensor headers producing wrong shapes.

Example fix

// before: hand-rolled conversion flattened embeddings
tensor 'model.embed_tokens.weight' shape [2315296]
// after: re-convert preserving rank
tensor 'model.embed_tokens.weight' shape [151669, 2048]
Defensive patterns

Strategy: validation

Validate before calling

import gguf
reader = gguf.GGUFReader(path)
for t in reader.tensors:
    if t.name == "model.embed_tokens.weight":
        shape = t.shape
        if shape is None or len(shape) != 2:
            raise ValueError(f"Bad embed shape {shape} in {path}; re-convert the GGUF.")

Type guard

def has_2d_embedding(path: str) -> bool:
    try:
        import gguf
        for t in gguf.GGUFReader(path).tensors:
            if t.name == "model.embed_tokens.weight":
                return len(t.shape) == 2
    except Exception:
        pass
    return False

Try / catch

try:
    model = load_text_encoder(cfg)
except ValueError as e:
    if "Expected 2D embed_tokens" in str(e):
        raise ModelValidationError(f"GGUF {cfg.path} corrupt; re-download/re-convert.") from e
    raise

Prevention

When it happens

Trigger: Loading a GGUF text encoder where the embed_tokens tensor was flattened (1D), stored transposed with extra dims (3D), or quantized to a format whose metadata loses the 2D shape during conversion.

Common situations: Hand-converted GGUF from safetensors with a broken conversion script; corrupted download where tensor metadata is garbled; incompatible model generation whose embed tensor was fused or reshaped.

Understand the failure class

Background: Tensor shape mismatch errors ("must have shape", "expected shape ... got ..."): when tensor dimensions disagree with what an op or layer was told to expect — this error's family across 6 libraries.

Related errors

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/af4a9228d1c6479b. Report an issue: GitHub.