huggingface/transformers · error · ImportError

cpu_offload_space=None requires psutil to auto-size the CPU

Error message

cpu_offload_space=None requires psutil to auto-size the CPU swap pool. Install psutil or pass an explicit GiB value.

What it means

Raised in OffloadingManager._compute_num_cpu_blocks when cpu_offload_space is None (auto-size requested) but psutil is not installed. Auto-sizing reads available RAM via psutil.virtual_memory(); without it there is no safe way to pick a pool size, so it fails rather than guessing.

Source

Thrown at src/transformers/generation/continuous_batching/offloading_manager.py:156

                clamped_gib = max_bytes / (1024**3)
                logger.warning(
                    f"cpu_offload_space={cpu_offload_space_gib:.1f} GiB exceeds {safety_threshold:.0%} of total RAM "
                    f"({total_ram / (1024**3):.1f} GiB). Clamping to {clamped_gib:.1f} GiB."
                )
                offload_bytes = max_bytes
        # Else if the max is None, throw a warning and accept the requested number of bytes as is
        elif offload_bytes is not None:
            logger.warning(
                "psutil is not available — cpu_offload_space_safety_threshold cannot be enforced. "
                "Install psutil to enable the safety cap."
            )
        # Else if the requested number of bytes is None, we use the max number of bytes as the requested number of bytes
        elif max_bytes is not None:
            offload_bytes = max_bytes
            logger.warning(f"Auto-sizing CPU swap pool from safety threshold: {max_bytes / (1024**3):.2f} GiB.")
        # Otherwise, it means the pool was supposed to be sized using psutil but it is not available
        else:
            raise ImportError(
                "cpu_offload_space=None requires psutil to auto-size the CPU swap pool. Install psutil or pass an "
                "explicit GiB value."
            )

        # Compute how many blocks fit in CPU pool
        bytes_per_block = (
            2                                 # one for key, one for value
            * len(self.cache.key_cache)       # number of layers in a layer group
            * self.cache.block_size           # block size
            * self.cache.num_key_value_heads  # number of key value heads
            * self.cache.head_dim             # head dimension
            * self.cache.dtype.itemsize       # data type size in bytes
        )  # fmt: skip
        if bytes_per_block == 0:
            raise ValueError("The number of bytes per block is 0. This is not possible.")
        return offload_bytes // bytes_per_block

    def _stream_ctx(self):

View on GitHub (pinned to a597f97485)

Solutions

1. pip install psutil (or install transformers with the extra that includes it)
2. Pass an explicit value: ContinuousBatchingConfig(cpu_offload_space=8.0) (GiB)
3. For Dockerfiles, add psutil next to transformers

Example fix

# before (no psutil installed)
cfg = ContinuousBatchingConfig(cpu_offload_space=None)

# after
# option A: pip install psutil
# option B:
cfg = ContinuousBatchingConfig(cpu_offload_space=8.0)

try:
    import psutil  # noqa
    cpu_offload_space = None          # auto-size is safe
except ImportError:
    cpu_offload_space = 8.0           # explicit GiB fallback
cfg = ContinuousBatchingConfig(cpu_offload_space=cpu_offload_space)

try:
    manager = model.continuous_batching(config=cfg)
except ImportError as e:
    if 'psutil' in str(e):
        cfg.cpu_offload_space = 8.0
        manager = model.continuous_batching(config=cfg)
    else:
        raise

When it happens

Trigger: Constructing a ContinuousBatchingConfig/manager with cpu_offload_space=None (the auto default) on an environment where import psutil fails — slim Docker images, minimal CI images, or transformers installed without the psutil extra.

Common situations: Docker images without psutil; new venv where transformers[torch] was installed but not psutil; CI pipeline using the same config as a production box that has psutil.

Related errors

• You need to install optimum-quanto in order to use KV cache
• You need to install `HQQ` in order to use KV cache quantizat
• Audio transcription requires sequential generation (not CB)
• CB worker is dead and cannot accept request {request_id}: {s
• CB worker died during request {request_id}: {result.error}

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/e683c906f8196412. Report an issue: GitHub.