{"record":{"id":"8ec7a0a0ec3d09d6","repo":"p-e-w/heretic","slug":"failed-to-load-model-with-all-configured-dtypes","errorCode":null,"errorMessage":"Failed to load model with all configured dtypes.","messagePattern":"Failed to load model with all configured dtypes\\.","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"critical","filePath":"src/heretic/model.py","lineNumber":169,"sourceCode":"            except Exception as error:\n                self.model = None  # ty:ignore[invalid-assignment]\n                empty_cache()\n\n                formatted = format_exception(error)\n                if \"\\n\" in formatted:\n                    print(f\"* [red]Failed:\\n{formatted}[/]\")\n                else:\n                    print(f\"* [red]Failed ({formatted})[/]\")\n\n                continue\n\n            if settings.quantization == QuantizationMethod.BNB_4BIT:\n                print(\"* Quantized to 4-bit precision\")\n\n            break\n\n        if self.model is None:\n            raise Exception(\"Failed to load model with all configured dtypes.\")\n\n        self._apply_lora()\n\n        # LoRA B matrices are initialized to zero by default in PEFT,\n        # so we don't need to do anything manually.\n\n        print(f\"* Transformer model with [bold]{len(self.get_layers())}[/] layers\")\n\n        all_components = {}\n        for layer_index in range(len(self.get_layers())):\n            for component, modules in self.get_layer_modules(layer_index).items():\n                if component not in all_components:\n                    all_components[component] = 0\n                all_components[component] += len(modules)\n\n        print(\"* Abliterable components:\")\n        for component, count in all_components.items():\n            print(f\"  * [bold]{component}[/]: [bold]{count}[/] modules total\")","sourceCodeStart":151,"sourceCodeEnd":187,"githubUrl":"https://github.com/p-e-w/heretic/blob/bedb94ef117a271532ac2058447fbc165d5051bd/src/heretic/model.py#L151-L187","documentation":"Model loading iterates the configured dtypes (dtype fallback loop) and assigns self.model on the first success. If every dtype attempt fails (OOM, missing weights, unsupported dtype/quantization), self.model stays None and the constructor raises this generic failure.","triggerScenarios":"Constructing the Model wrapper where every attempted torch dtype fails — e.g. model doesn't fit in GPU memory at float16/float32, bfloat16 unsupported by the GPU, or 4-bit quantization requested without bitsandbytes installed.","commonSituations":"Loading large models on small GPUs (CUDA OOM at all dtypes), older GPUs lacking bf16 support, misconfigured dtype list in config, or quantization=bnb_4bit without the bitsandbytes dependency.","solutions":["Read the underlying per-dtype exception printed during the loop (OOM, missing package, etc.) and address that root cause","Free GPU memory or use a smaller model / more quantization (bnb_4bit)","Remove bfloat16 from dtypes if your GPU doesn't support it","Install missing optional deps (bitsandbytes) or fix the model path in settings"],"exampleFix":"// before (config.toml)\ndtypes = [\"bfloat16\", \"float16\"]  # GPU has no bf16\n// after (config.toml)\ndtypes = [\"float16\"]\nquantization = \"bnb_4bit\"\n","handlingStrategy":"fallback","validationCode":"import torch\nif settings.quantization == \"bnb_4bit\":\n    import bitsandbytes  # noqa: F401  (raises if missing)\nfree, _ = torch.cuda.mem_get_info()\nif free < estimated_model_bytes:\n    raise RuntimeError(\"Insufficient GPU memory for configured model/dtypes\")","typeGuard":"def gpu_supports(dtype: str) -> bool:\n    cap = torch.cuda.get_device_capability()\n    return not (dtype == \"bfloat16\" and cap < (8, 0))","tryCatchPattern":"try:\n    model = Model(settings)\nexcept Exception as e:\n    logger.exception(\"Model load failed across all dtypes\")\n    settings.quantization = \"bnb_4bit\"  # fallback to 4-bit\n    model = Model(settings)","preventionTips":["Check the per-dtype exception logged inside the loop for the real cause","Match dtypes to GPU capability (bf16 needs sm_80+)","Prefer 4-bit quantization for large models on limited VRAM","Install optional deps (bitsandbytes, accelerate) before loading"],"tags":["model-loading","gpu","memory","torch"],"backgroundTag":"model-load-failed","analyzedSha":"bedb94ef117a271532ac2058447fbc165d5051bd","analyzedAt":"2026-08-29T08:38:06.692Z","schemaVersion":2},"datasetVersion":"2026-08-29T12:17:43.993Z"}