p-e-w/heretic · critical · Exception

Failed to load model with all configured dtypes.

Error message

Failed to load model with all configured dtypes.

What it means

Model loading iterates the configured dtypes (dtype fallback loop) and assigns self.model on the first success. If every dtype attempt fails (OOM, missing weights, unsupported dtype/quantization), self.model stays None and the constructor raises this generic failure.

Source

Thrown at src/heretic/model.py:169

            except Exception as error:
                self.model = None  # ty:ignore[invalid-assignment]
                empty_cache()

                formatted = format_exception(error)
                if "\n" in formatted:
                    print(f"* [red]Failed:\n{formatted}[/]")
                else:
                    print(f"* [red]Failed ({formatted})[/]")

                continue

            if settings.quantization == QuantizationMethod.BNB_4BIT:
                print("* Quantized to 4-bit precision")

            break

        if self.model is None:
            raise Exception("Failed to load model with all configured dtypes.")

        self._apply_lora()

        # LoRA B matrices are initialized to zero by default in PEFT,
        # so we don't need to do anything manually.

        print(f"* Transformer model with [bold]{len(self.get_layers())}[/] layers")

        all_components = {}
        for layer_index in range(len(self.get_layers())):
            for component, modules in self.get_layer_modules(layer_index).items():
                if component not in all_components:
                    all_components[component] = 0
                all_components[component] += len(modules)

        print("* Abliterable components:")
        for component, count in all_components.items():
            print(f"  * [bold]{component}[/]: [bold]{count}[/] modules total")

View on GitHub (pinned to bedb94ef11)

Solutions

1. Read the underlying per-dtype exception printed during the loop (OOM, missing package, etc.) and address that root cause
2. Free GPU memory or use a smaller model / more quantization (bnb_4bit)
3. Remove bfloat16 from dtypes if your GPU doesn't support it
4. Install missing optional deps (bitsandbytes) or fix the model path in settings

Example fix

// before (config.toml)
dtypes = ["bfloat16", "float16"]  # GPU has no bf16
// after (config.toml)
dtypes = ["float16"]
quantization = "bnb_4bit"

import torch
if settings.quantization == "bnb_4bit":
    import bitsandbytes  # noqa: F401  (raises if missing)
free, _ = torch.cuda.mem_get_info()
if free < estimated_model_bytes:
    raise RuntimeError("Insufficient GPU memory for configured model/dtypes")

def gpu_supports(dtype: str) -> bool:
    cap = torch.cuda.get_device_capability()
    return not (dtype == "bfloat16" and cap < (8, 0))

try:
    model = Model(settings)
except Exception as e:
    logger.exception("Model load failed across all dtypes")
    settings.quantization = "bnb_4bit"  # fallback to 4-bit
    model = Model(settings)

When it happens

Trigger: Constructing the Model wrapper where every attempted torch dtype fails — e.g. model doesn't fit in GPU memory at float16/float32, bfloat16 unsupported by the GPU, or 4-bit quantization requested without bitsandbytes installed.

Common situations: Loading large models on small GPUs (CUDA OOM at all dtypes), older GPUs lacking bf16 support, misconfigured dtype list in config, or quantization=bnb_4bit without the bitsandbytes dependency.

Related errors

• prompts must not be empty

AI-assisted analysis of p-e-w/heretic@bedb94ef11 (2026-08-29). Data as JSON: /api/errors/8ec7a0a0ec3d09d6. Report an issue: GitHub.