{"record":{"id":"51cab66710a37dc0","repo":"invoke-ai/InvokeAI","slug":"operation-changed-the-dtype-of-ggmltensor-unexpect","errorCode":null,"errorMessage":"Operation changed the dtype of GGMLTensor unexpectedly.","messagePattern":"Operation changed the dtype of GGMLTensor unexpectedly\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"invokeai/backend/quantization/gguf/ggml_tensor.py","lineNumber":71,"sourceCode":"    return func(*dequantized_args, **dequantized_kwargs)\n\n\ndef apply_to_quantized_tensor(func, args, kwargs):\n    \"\"\"A helper function to apply a function to a quantized GGML tensor, and re-wrap the result in a GGMLTensor.\n\n    Assumes that the first argument is a GGMLTensor.\n    \"\"\"\n    # We expect the first argument to be a GGMLTensor, and all other arguments to be non-GGMLTensors.\n    ggml_tensor = args[0]\n    assert isinstance(ggml_tensor, GGMLTensor)\n    assert all(not isinstance(a, GGMLTensor) for a in args[1:])\n    assert all(not isinstance(v, GGMLTensor) for v in kwargs.values())\n\n    new_data = func(ggml_tensor.quantized_data, *args[1:], **kwargs)\n\n    if new_data.dtype != ggml_tensor.quantized_data.dtype:\n        # This is intended to catch calls such as `.to(dtype-torch.float32)`, which are not supported on GGMLTensors.\n        raise ValueError(\"Operation changed the dtype of GGMLTensor unexpectedly.\")\n\n    return GGMLTensor(\n        new_data, ggml_tensor._ggml_quantization_type, ggml_tensor.tensor_shape, ggml_tensor.compute_dtype\n    )\n\n\nGGML_TENSOR_OP_TABLE = {\n    # Ops to run on the quantized tensor.\n    torch.ops.aten.detach.default: apply_to_quantized_tensor,  # pyright: ignore\n    torch.ops.aten._to_copy.default: apply_to_quantized_tensor,  # pyright: ignore\n    torch.ops.aten.clone.default: apply_to_quantized_tensor,  # pyright: ignore\n    # Ops to run on dequantized tensors.\n    torch.ops.aten.t.default: dequantize_and_run,  # pyright: ignore\n    torch.ops.aten.addmm.default: dequantize_and_run,  # pyright: ignore\n    torch.ops.aten.mul.Tensor: dequantize_and_run,  # pyright: ignore\n    torch.ops.aten.add.Tensor: dequantize_and_run,  # pyright: ignore\n    torch.ops.aten.sub.Tensor: dequantize_and_run,  # pyright: ignore\n    torch.ops.aten.allclose.default: dequantize_and_run,  # pyright: ignore","sourceCodeStart":53,"sourceCodeEnd":89,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/quantization/gguf/ggml_tensor.py#L53-L89","documentation":"GGMLTensor wraps quantized GGUF data; torch functional dispatches routed through apply_to_quantized_tensor must preserve the underlying quantized_data dtype because dequantization assumes it. If an operation (e.g. .to(torch.float32)) changes the dtype, ValueError is raised since casting quantized GGML data is unsupported.","triggerScenarios":"Calling .to(dtype=...) with a different dtype, or any torch op on a GGMLTensor that alters the quantized payload's dtype.","commonSituations":"Code that generically casts all model tensors to float32/fp16; moving pipeline components with .to() specifying both device and dtype; interoperating with libraries that call .float() on every parameter.","solutions":["Do not cast GGMLTensor to another dtype; convert to compute_dtype at dequantization time instead","Cast only non-quantized tensors, or dequantize to a plain torch.Tensor first then cast","When moving devices, pass only device= to .to() (device moves preserve dtype)"],"exampleFix":"// before\nmodel.to(device=\"cuda\", dtype=torch.float16)\n// after\nmodel.to(device=\"cuda\")  # dtype changes are unsupported on GGML quantized tensors","handlingStrategy":"type-guard","validationCode":"def safe_to(module, *, device=None):\n    for p in module.parameters():\n        if isinstance(p, GGMLTensor):\n            assert device is not None and not hasattr(p, '_dtype_override'), \"GGML tensors must not be dtype-cast\"\n    module.to(device=device)","typeGuard":"def is_ggml(t: torch.Tensor) -> bool:\n    return isinstance(t, GGMLTensor)","tryCatchPattern":"try:\n    out = tensor.to(dtype=torch.float32)\nexcept ValueError:\n    out = tensor.dequantize().to(dtype=torch.float32)  # plain tensor path","preventionTips":["Never include dtype= in .to() calls on GGUF pipelines","Filter quantized parameters out of generic casting loops","Keep compute_dtype conversion inside the dequantization path only"],"tags":["valueerror","quantization","gguf","dtype"],"backgroundTag":"quantized-dtype-change","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}