{"record":{"id":"fa9bd223d540f592","repo":"invoke-ai/InvokeAI","slug":"refusing-to-record-compute-dtype-as-an-fp8-compu","errorCode":null,"errorMessage":"Refusing to record {compute_dtype} as an FP8 compute dtype; it is a storage-only dtype. This usually means the compute dtype was derived from an already-fp8-cast model.","messagePattern":"Refusing to record (.+?) as an FP8 compute dtype; it is a storage-only dtype\\. This usually means the compute dtype was derived from an already-fp8-cast model\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"invokeai/backend/util/fp8.py","lineNumber":38,"sourceCode":"\n# Storage-only float8 dtypes. Weights may be held in these, but no math may be done in them.\nFP8_STORAGE_DTYPES: tuple[torch.dtype, ...] = (torch.float8_e4m3fn, torch.float8_e5m2)\n\n# Attribute set on a model by the loader when FP8 layerwise casting is applied. It lives in the\n# module's `__dict__` (torch.dtype is not a Parameter/Module), so it survives the deepcopy of the\n# meta shell in the shared-CPU-weights adoption path.\nFP8_COMPUTE_DTYPE_ATTR = \"_invokeai_fp8_compute_dtype\"\n\n\ndef set_fp8_compute_dtype(model: torch.nn.Module, compute_dtype: torch.dtype) -> None:\n    \"\"\"Record the dtype that `model`'s fp8-cast layers compute in.\"\"\"\n    if compute_dtype in FP8_STORAGE_DTYPES:\n        # A float8 compute dtype is never valid, and recording one would silently reintroduce the\n        # very crash this module exists to prevent: `get_model_compute_dtype` trusts the marker, so\n        # every downstream tensor would be built in a dtype torch has no arithmetic kernels for.\n        # The realistic way to get here is deriving the compute dtype from a model that is already\n        # cast (i.e. casting twice) — fail loudly at the source instead.\n        raise ValueError(\n            f\"Refusing to record {compute_dtype} as an FP8 compute dtype; it is a storage-only dtype. \"\n            \"This usually means the compute dtype was derived from an already-fp8-cast model.\"\n        )\n    setattr(model, FP8_COMPUTE_DTYPE_ATTR, compute_dtype)\n\n\ndef get_model_compute_dtype(model: torch.nn.Module) -> torch.dtype:\n    \"\"\"Return the dtype that `model` actually computes in.\n\n    Equivalent to `model.dtype` for normally-loaded models. For models with FP8 storage it returns\n    the compute dtype (fp16/bf16) rather than the float8 storage dtype.\n    \"\"\"\n    marked = getattr(model, FP8_COMPUTE_DTYPE_ATTR, None)\n    if isinstance(marked, torch.dtype):\n        return marked\n\n    dtype = getattr(model, \"dtype\", None)\n    if not isinstance(dtype, torch.dtype):","sourceCodeStart":20,"sourceCodeEnd":56,"githubUrl":"https://github.com/invoke-ai/InvokeAI/blob/0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06/invokeai/backend/util/fp8.py#L20-L56","documentation":"set_fp8_compute_dtype() in invokeai/backend/util/fp8.py records, as an attribute on an nn.Module, the dtype that FP8 layerwise-cast layers compute in (fp16/bf16). It refuses the record if the given compute_dtype is one of FP8_STORAGE_DTYPES (torch.float8_e4m3fn / float8_e5m2), because those are storage-only dtypes with no CUDA arithmetic kernels — downstream code trusts this marker when building tensors, so recording float8 would silently reintroduce 'pow_cuda not implemented'-style crashes. The typical cause is deriving the compute dtype from model.dtype of a model that is already fp8-cast (double casting).","triggerScenarios":"Calling set_fp8_compute_dtype(model, model.dtype) or any path (_apply_fp8_to_nn_module / ModelLoader._apply_fp8_layerwise_casting) where the compute dtype was taken from an already-fp8-cast model (model.dtype returns float8_e4m3fn because it is derived from the first parameter), instead of using get_model_compute_dtype(model) or an explicit fp16/bf16 dtype.","commonSituations":"Custom loader/plugin code reading model.dtype after an initial FP8 cast and passing it into a second cast pass; casting a quantized checkpoint twice; copying dtype from a source model that was loaded with FP8 layerwise casting; building LoRA patch weights from the wrong dtype source.","solutions":["Use get_model_compute_dtype(model) instead of model.dtype whenever the dtype is needed for compute-side tensors or a re-cast.","If applying FP8 to an already-fp8 model, skip the second cast (check for the FP8_COMPUTE_DTYPE_ATTR marker first).","Pass an explicit fp16/bf16 dtype (e.g. the pipeline's configured precision) to set_fp8_compute_dtype.","Assert with a guard before casting: if getattr(model, FP8_COMPUTE_DTYPE_ATTR, None) is not None: return."],"exampleFix":"// before\ncompute_dtype = model.dtype  # float8_e4m3fn on an already-cast model\nset_fp8_compute_dtype(model, compute_dtype)\n// after\nfrom invokeai.backend.util.fp8 import get_model_compute_dtype\ncompute_dtype = get_model_compute_dtype(model)  # fp16/bf16\nset_fp8_compute_dtype(model, compute_dtype)","handlingStrategy":"validation","validationCode":"from invokeai.backend.util.fp8 import FP8_STORAGE_DTYPES, get_model_compute_dtype\n\ndef safe_fp8_cast(model, compute_dtype=None):\n    if compute_dtype is None:\n        compute_dtype = get_model_compute_dtype(model)\n    assert compute_dtype not in FP8_STORAGE_DTYPES, (\n        f'{compute_dtype} is storage-only; use get_model_compute_dtype(model) instead of model.dtype'\n    )\n    set_fp8_compute_dtype(model, compute_dtype)","typeGuard":"def is_compute_safe_dtype(dtype: torch.dtype) -> bool:\n    return dtype not in (torch.float8_e4m3fn, torch.float8_e5m2)","tryCatchPattern":"try:\n    set_fp8_compute_dtype(model, compute_dtype)\nexcept ValueError as e:\n    if 'storage-only dtype' in str(e):\n        compute_dtype = get_model_compute_dtype(model)\n        set_fp8_compute_dtype(model, compute_dtype)\n    else:\n        raise","preventionTips":["Never read model.dtype on FP8-cast models; always use get_model_compute_dtype(model).","Skip casting entirely if the FP8_COMPUTE_DTYPE_ATTR marker is already present on the module.","Centralize all dtype selection in one helper instead of scattering model.dtype reads.","Add a unit test asserting the chosen compute dtype is not in FP8_STORAGE_DTYPES before any cast."],"tags":["pytorch","dtype","fp8","quantization"],"backgroundTag":"storage-only-dtype-as-compute","analyzedSha":"0b6a024f2ff6a86bfb953dcdb9cc504ef7397a06","analyzedAt":"2026-08-29T04:46:49.967Z","schemaVersion":2},"datasetVersion":"2026-08-29T07:17:48.351Z"}