docling-project/docling · error · RuntimeError

Nemotron OCR requires CUDA at initialization time, but `torc

Error message

Nemotron OCR requires CUDA at initialization time, but `torch.cuda.is_available()` is false.

What it means

Beyond the configured device, validate_runtime verifies a CUDA runtime is actually usable via torch.cuda.is_available(). If torch cannot initialize CUDA (no driver, mismatched torch build, GPU in exclusive use), a RuntimeError is raised at initialization time. This catches configs that claim CUDA while the process cannot get it.

Source

Thrown at docling/models/stages/ocr/nemotron_ocr_model.py:155

                    'via `pip install "docling[feat-ocr-nemotron]"` on Linux x86_64 with '
                    "Python 3.12 and CUDA 13.x."
                ) from exc

            # Resolve the request language
            language = resolve_nemotronocr_language(options.lang)

            # Initialize the model
            model_dir = self._resolve_model_dir(language, artifacts_path=artifacts_path)

            self.reader = NemotronOCRV2(
                model_dir=None if model_dir is None else str(model_dir),
                lang=language,
            )

    @staticmethod
    def _fail_runtime(message: str) -> None:
        _log.error(message)
        raise RuntimeError(message)

    @classmethod
    def validate_runtime(cls, accelerator_options: AcceleratorOptions) -> None:
        if sys.platform != "linux":
            cls._fail_runtime("Nemotron OCR is only supported on Linux.")

        if platform.machine() != "x86_64":
            cls._fail_runtime("Nemotron OCR is only supported on x86_64 machines.")

        if sys.version_info[:2] != (3, 12):
            cls._fail_runtime("Nemotron OCR requires Python 3.12.")

        requested_device = decide_device(accelerator_options.device)
        if not requested_device.startswith("cuda"):
            cls._fail_runtime(
                "Nemotron OCR requires a CUDA accelerator. Set "
                "`pipeline_options.accelerator_options.device` to CUDA or AUTO on a "
                "CUDA-enabled machine."

View on GitHub (pinned to 61d76f1ff3)

Solutions

Install GPU-enabled PyTorch matching your CUDA 13.x driver, and verify `python -c "import torch; print(torch.cuda.is_available())"` returns True.
For containers, pass the GPU through (docker run --gpus all ...) and install nvidia-container-toolkit.
If the host truly has no usable CUDA, run on a CUDA machine or fall back to a CPU OCR engine instead of Nemotron.

Example fix

# before: CPU-only torch on a GPU host
pip install torch  # defaults to CPU wheel -> cuda.is_available() False

# after
pip install torch --index-url https://download.pytorch.org/whl/cu130
python -c "import torch; assert torch.cuda.is_available()"

Defensive patterns

Strategy: validation

Validate before calling

import torch

if not torch.cuda.is_available():
    raise SystemExit("CUDA unavailable: fix driver/torch before using Nemotron OCR")

Prevention

Smoke-test torch.cuda.is_available() in CI and at deploy time, not just at import time.
Install GPU torch builds (cu13x index) matching the NVIDIA driver.
Expose GPUs to containers with --gpus all and nvidia-container-toolkit.

When it happens

Trigger: device=CUDA/AUTO on a machine without the NVIDIA driver; CPU-only PyTorch wheel installed; driver/toolkit version too old for the torch build; GPU busy in exclusive compute mode.

Common situations: Containers without GPU passthrough (missing --gpus all); pip torch built for CPU only; CUDA driver version older than torch requires; CI runners without GPUs.

Related errors

AI-assisted analysis of docling-project/docling@61d76f1ff3 (2026-08-14). Data as JSON: /api/errors/527823b7e2a1681e. Report an issue: GitHub.