Lightning-AI/pytorch-lightning · error · RuntimeError

You requested to find {num_devices} devices but only {len(av

Error message

You requested to find {num_devices} devices but only {len(available_devices)} are currently available. The devices {unavailable_devices} are occupied by other processes and can't be used at the moment.

What it means

val_check_interval can be given as a string in strict 'DD:HH:MM:SS' format. _parse_time_interval_seconds splits on ':' and requires exactly 4 parts (days, hours, minutes, seconds). Strings with a different number of colon-separated segments (e.g. 'HH:MM' or '00:00:30') raise this error.

Source

Thrown at src/lightning/fabric/accelerators/cuda.py:130

        )

    available_devices = []
    unavailable_devices = []

    for gpu_idx in visible_devices:
        try:
            torch.tensor(0, device=torch.device("cuda", gpu_idx))
        except RuntimeError:
            unavailable_devices.append(gpu_idx)
            continue

        available_devices.append(gpu_idx)
        if len(available_devices) == num_devices:
            # exit early if we found the right number of GPUs
            break

    if num_devices != -1 and len(available_devices) != num_devices:
        raise RuntimeError(
            f"You requested to find {num_devices} devices but only {len(available_devices)} are currently available."
            f" The devices {unavailable_devices} are occupied by other processes and can't be used at the moment."
        )
    return available_devices


def _get_all_visible_cuda_devices() -> list[int]:
    """Returns a list of all visible CUDA GPU devices.

    Devices masked by the environment variabale ``CUDA_VISIBLE_DEVICES`` won't be returned here. For example, assume you
    have 8 physical GPUs. If ``CUDA_VISIBLE_DEVICES="1,3,6"``, then this function will return the list ``[0, 1, 2]``
    because these are the three visible GPUs after applying the mask ``CUDA_VISIBLE_DEVICES``.

    """
    return list(range(num_cuda_devices()))


def num_cuda_devices() -> int:

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Format the string as exactly four colon-separated integers: 'DD:HH:MM:SS', e.g. '0:0:30' is invalid, use '0:0:0:30'
2. Prefer a timedelta or dict: Trainer(val_check_interval=timedelta(minutes=30)) or {'minutes': 30}
3. Or pass an int/float number of batches / fraction

Example fix

# before
trainer = Trainer(val_check_interval="0:0:30")

# after
trainer = Trainer(val_check_interval="0:0:0:30")
# or
from datetime import timedelta
trainer = Trainer(val_check_interval=timedelta(seconds=30))

import re

TIME_RE = re.compile(r"^\d+:\d+:\d+:\d+$")

def valid_interval_str(s: str) -> bool:
    return bool(TIME_RE.match(s))

def is_dd_hh_mm_ss(v) -> bool:
    if not isinstance(v, str):
        return False
    parts = v.split(":")
    return len(parts) == 4 and all(p.isdigit() for p in parts)

When it happens

Trigger: Trainer(val_check_interval='0:0:30') (3 parts); Trainer(val_check_interval='10:00') (2 parts); Trainer(val_check_interval='1:2:3:4:5') (5 parts).

Common situations: Naturally writing an HH:MM:SS duration like '00:01:00' (3 segments) instead of DD-prefixed '0:00:01:00'; adapting ISO-8601 durations; passing clock times rather than durations.

Related errors

• Device should be MPS, got {device} instead.
• Device should be CUDA, got {device} instead.
• You requested to find {num_devices} devices but there are no
• You requested to find {num_devices} devices but this machine
• `name` must be a str, found {name}

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/d1d8b343a66f1a2a. Report an issue: GitHub.