invoke-ai/InvokeAI · error · ValueError

Operation changed the dtype of GGMLTensor unexpectedly.

Error message

Operation changed the dtype of GGMLTensor unexpectedly.

What it means

GGMLTensor wraps quantized GGUF data; torch functional dispatches routed through apply_to_quantized_tensor must preserve the underlying quantized_data dtype because dequantization assumes it. If an operation (e.g. .to(torch.float32)) changes the dtype, ValueError is raised since casting quantized GGML data is unsupported.

Solutions

1. Do not cast GGMLTensor to another dtype; convert to compute_dtype at dequantization time instead
2. Cast only non-quantized tensors, or dequantize to a plain torch.Tensor first then cast
3. When moving devices, pass only device= to .to() (device moves preserve dtype)

Example fix

// before
model.to(device="cuda", dtype=torch.float16)
// after
model.to(device="cuda")  # dtype changes are unsupported on GGML quantized tensors

def safe_to(module, *, device=None):
    for p in module.parameters():
        if isinstance(p, GGMLTensor):
            assert device is not None and not hasattr(p, '_dtype_override'), "GGML tensors must not be dtype-cast"
    module.to(device=device)

def is_ggml(t: torch.Tensor) -> bool:
    return isinstance(t, GGMLTensor)

try:
    out = tensor.to(dtype=torch.float32)
except ValueError:
    out = tensor.dequantize().to(dtype=torch.float32)  # plain tensor path

When it happens

Trigger: Calling .to(dtype=...) with a different dtype, or any torch op on a GGMLTensor that alters the quantized payload's dtype.

Common situations: Code that generically casts all model tensors to float32/fp16; moving pipeline components with .to() specifying both device and dtype; interoperating with libraries that call .float() on every parameter.

Related errors

• Operation changed the dtype of SDNQTensor unexpectedly.
• Invalid or expired token
• Missing keys loading
• Refusing to record as an FP8 compute dtype; it is a…
• state dict does not look like GGUF quantized

AI-assisted analysis of invoke-ai/InvokeAI@0b6a024f2f (2026-08-29). Data as JSON: /api/errors/51cab66710a37dc0. Report an issue: GitHub.

Appendix: source

Thrown at invokeai/backend/quantization/gguf/ggml_tensor.py:71

    return func(*dequantized_args, **dequantized_kwargs)


def apply_to_quantized_tensor(func, args, kwargs):
    """A helper function to apply a function to a quantized GGML tensor, and re-wrap the result in a GGMLTensor.

    Assumes that the first argument is a GGMLTensor.
    """
    # We expect the first argument to be a GGMLTensor, and all other arguments to be non-GGMLTensors.
    ggml_tensor = args[0]
    assert isinstance(ggml_tensor, GGMLTensor)
    assert all(not isinstance(a, GGMLTensor) for a in args[1:])
    assert all(not isinstance(v, GGMLTensor) for v in kwargs.values())

    new_data = func(ggml_tensor.quantized_data, *args[1:], **kwargs)

    if new_data.dtype != ggml_tensor.quantized_data.dtype:
        # This is intended to catch calls such as `.to(dtype-torch.float32)`, which are not supported on GGMLTensors.
        raise ValueError("Operation changed the dtype of GGMLTensor unexpectedly.")

    return GGMLTensor(
        new_data, ggml_tensor._ggml_quantization_type, ggml_tensor.tensor_shape, ggml_tensor.compute_dtype
    )


GGML_TENSOR_OP_TABLE = {
    # Ops to run on the quantized tensor.
    torch.ops.aten.detach.default: apply_to_quantized_tensor,  # pyright: ignore
    torch.ops.aten._to_copy.default: apply_to_quantized_tensor,  # pyright: ignore
    torch.ops.aten.clone.default: apply_to_quantized_tensor,  # pyright: ignore
    # Ops to run on dequantized tensors.
    torch.ops.aten.t.default: dequantize_and_run,  # pyright: ignore
    torch.ops.aten.addmm.default: dequantize_and_run,  # pyright: ignore
    torch.ops.aten.mul.Tensor: dequantize_and_run,  # pyright: ignore
    torch.ops.aten.add.Tensor: dequantize_and_run,  # pyright: ignore
    torch.ops.aten.sub.Tensor: dequantize_and_run,  # pyright: ignore
    torch.ops.aten.allclose.default: dequantize_and_run,  # pyright: ignore

View on GitHub (pinned to 0b6a024f2f)