invoke-ai/InvokeAI · error · ValueError

Operation changed the dtype of GGMLTensor unexpectedly.

Error message

Operation changed the dtype of GGMLTensor unexpectedly.

What it means

GGMLTensor wraps quantized GGUF data; torch functional dispatches routed through apply_to_quantized_tensor must preserve the underlying quantized_data dtype because dequantization assumes it. If an operation (e.g. .to(torch.float32)) changes the dtype, ValueError is raised since casting quantized GGML data is unsupported.

Source

Thrown at invokeai/backend/quantization/gguf/ggml_tensor.py:71

    return func(*dequantized_args, **dequantized_kwargs)


def apply_to_quantized_tensor(func, args, kwargs):
    """A helper function to apply a function to a quantized GGML tensor, and re-wrap the result in a GGMLTensor.

    Assumes that the first argument is a GGMLTensor.
    """
    # We expect the first argument to be a GGMLTensor, and all other arguments to be non-GGMLTensors.
    ggml_tensor = args[0]
    assert isinstance(ggml_tensor, GGMLTensor)
    assert all(not isinstance(a, GGMLTensor) for a in args[1:])
    assert all(not isinstance(v, GGMLTensor) for v in kwargs.values())

    new_data = func(ggml_tensor.quantized_data, *args[1:], **kwargs)

    if new_data.dtype != ggml_tensor.quantized_data.dtype:
        # This is intended to catch calls such as `.to(dtype-torch.float32)`, which are not supported on GGMLTensors.
        raise ValueError("Operation changed the dtype of GGMLTensor unexpectedly.")

    return GGMLTensor(
        new_data, ggml_tensor._ggml_quantization_type, ggml_tensor.tensor_shape, ggml_tensor.compute_dtype
    )


GGML_TENSOR_OP_TABLE = {
    # Ops to run on the quantized tensor.
    torch.ops.aten.detach.default: apply_to_quantized_tensor,  # pyright: ignore
    torch.ops.aten._to_copy.default: apply_to_quantized_tensor,  # pyright: ignore
    torch.ops.aten.clone.default: apply_to_quantized_tensor,  # pyright: ignore
    # Ops to run on dequantized tensors.
    torch.ops.aten.t.default: dequantize_and_run,  # pyright: ignore
    torch.ops.aten.addmm.default: dequantize_and_run,  # pyright: ignore
    torch.ops.aten.mul.Tensor: dequantize_and_run,  # pyright: ignore
    torch.ops.aten.add.Tensor: dequantize_and_run,  # pyright: ignore
    torch.ops.aten.sub.Tensor: dequantize_and_run,  # pyright: ignore
    torch.ops.aten.allclose.default: dequantize_and_run,  # pyright: ignore

View on GitHub (pinned to 0b6a024f2f)

Solutions

1. Do not cast GGMLTensor to another dtype; convert to compute_dtype at dequantization time instead
2. Cast only non-quantized tensors, or dequantize to a plain torch.Tensor first then cast
3. When moving devices, pass only device= to .to() (device moves preserve dtype)

Example fix

// before
model.to(device="cuda", dtype=torch.float16)
// after
model.to(device="cuda")  # dtype changes are unsupported on GGML quantized tensors

def safe_to(module, *, device=None):
    for p in module.parameters():
        if isinstance(p, GGMLTensor):
            assert device is not None and not hasattr(p, '_dtype_override'), "GGML tensors must not be dtype-cast"
    module.to(device=device)

def is_ggml(t: torch.Tensor) -> bool:
    return isinstance(t, GGMLTensor)

try:
    out = tensor.to(dtype=torch.float32)
except ValueError:
    out = tensor.dequantize().to(dtype=torch.float32)  # plain tensor path

When it happens

Trigger: Calling .to(dtype=...) with a different dtype, or any torch op on a GGMLTensor that alters the quantized payload's dtype.

Common situations: Code that generically casts all model tensors to float32/fp16; moving pipeline components with .to() specifying both device and dtype; interoperating with libraries that call .float() on every parameter.

Related errors

• Operation changed the dtype of SDNQTensor unexpectedly.
• Invalid or expired token
• state dict does not look like GGUF quantized
• state dict looks like GGUF quantized
• Unexpected keys loading {model_name}: {unexpected}

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/51cab66710a37dc0. Report an issue: GitHub.