labmlai/annotated_deep_learning_paper_implementations · error · ImportError

Please install `bitsandbytes` with `pip install bitsandbytes

Error message

Please install `bitsandbytes` with `pip install bitsandbytes -U`

What it means

labml-nn's LLM.int8() support delegates 8-bit linear layers to the third-party bitsandbytes package. The module does a bare 'from bitsandbytes.nn import Linear8bitLt, Int8Params' at import time and converts any ImportError into an actionable ImportError telling you to install the package. It is purely an environment problem, not a code or data problem.

Source

Thrown at labml_nn/neox/utils/llm_int8.py:37

These features get clamped in 8-bit integer space which causes the model performance to degrade.
As a solution they pick these outliers (greater than a specified threshold)
and compute their multiplications separately in float16 space.
Since the percentage of outliers is around 0.01% this doesn't increase memory usage,
and prevents the model from degrading performance.

The code to transform GPT-NoeX layers is defined in [model.py](../model.html#post_load_prepare).

Here are example uses of GPT-NeoX with int8 quantization.

* [Generate Text](../samples/llm_int8.html)
* [Run Evaluation Tests](../evaluation/llm_int8.html)
"""

# Import [`bitsandbytes`](https://github.com/timdettmers/bitsandbytes) package
try:
    from bitsandbytes.nn import Linear8bitLt, Int8Params
except ImportError:
    raise ImportError('''Please install `bitsandbytes` with `pip install bitsandbytes -U`''')

import torch
from torch import nn


def make_llm_int8_linear(linear_module: nn.Linear, device: torch.device, threshold: float = 6.0):
    """
    ## Transform a `nn.Linear` layer to LLM.int8() linear layer

    :param linear_module: is the `nn.Linear` layer to transform
    :param device: is the device of the model
    :param threshold: is the threshold $\alpha$ to use for outlier detection
    """

    #
    assert isinstance(linear_module, nn.Linear)

    # Create an empty Linear8bitLt module

View on GitHub (pinned to 33ab02281c)

Solutions

1. pip install bitsandbytes -U (ideally pip install bitsandbytes -U --no-cache-dir)
2. Verify the install matches your torch/CUDA build; if import still fails, reinstall torch and bitsandbytes into the same environment
3. If you do not need int8 quantization, stop importing/making llm_int8 layers so the module is never loaded
4. Pre-flight check in your entrypoint: importlib.util.find_spec('bitsandbytes') before importing llm_int8

Example fix

# before: ImportError at import time
from labml_nn.neox.utils.llm_int8 import make_llm_int8_linear

# after: guard the optional dependency
import importlib.util
if importlib.util.find_spec('bitsandbytes') is None:
    raise SystemExit('bitsandbytes required: pip install bitsandbytes -U')
from labml_nn.neox.utils.llm_int8 import make_llm_int8_linear

import importlib.util
if importlib.util.find_spec('bitsandbytes') is None:
    raise SystemExit('This script requires bitsandbytes: pip install bitsandbytes -U')

# only now import the int8 module
from labml_nn.neox.utils.llm_int8 import make_llm_int8_linear

def has_bitsandbytes() -> bool:
    import importlib.util
    return importlib.util.find_spec('bitsandbytes') is not None

try:
    from labml_nn.neox.utils.llm_int8 import make_llm_int8_linear
except ImportError as e:
    raise SystemExit(f'bitsandbytes missing or broken: {e}. Install with: pip install bitsandbytes -U') from e

When it happens

Trigger: Importing labml_nn.neox.utils.llm_int8 (or a module that imports it) in an environment where bitsandbytes is not installed or is broken enough that bitsandbytes.nn fails to import.

Common situations: Fresh conda/venv without extras installed; CI image that only installs labml-nn core deps; a bitsandbytes binary built for a different CUDA/torch version raising ImportError on import; dependency resolver having uninstalled it during an upgrade.

Related errors

• Invalid learning rate: {lr}
• Invalid epsilon value: {eps}
• Invalid beta parameter at index 0: {betas[0]}
• Invalid beta parameter at index 1: {betas[1]}
• GenericAdaptiveOptimizer does not support sparse gradients,

AI-assisted analysis of labmlai/annotated_deep_learning_paper_implementations@33ab02281c (2026-08-25). Data as JSON: /api/errors/ee3500151495ce19. Report an issue: GitHub.