vllm-project/vllm · error · ValueError

offload_num_in_group ({self.prefetch.offload_num_in_group})

Error message

offload_num_in_group ({self.prefetch.offload_num_in_group}) must be <= offload_group_size ({self.prefetch.offload_group_size})

What it means

OffloadConfig.validate_offload_config rejects prefetch-offload settings where offload_num_in_group exceeds offload_group_size. The group defines how many KV-cache blocks form an offload group; you cannot offload more blocks per group than the group contains. Checked whenever offload_backend == 'prefetch' or offload_group_size > 0.

Source

Thrown at vllm/config/offload.py:102

    """The backend for weight offloading. Options:
    - "auto": Selects based on which sub-config has non-default values
      (prefetch if offload_group_size > 0, uva if cpu_offload_gb > 0).
    - "uva": UVA (Unified Virtual Addressing) zero-copy offloading.
    - "prefetch": Async prefetch with group-based layer offloading.
    """

    uva: UVAOffloadConfig = Field(default_factory=UVAOffloadConfig)
    """Parameters for UVA offloading backend."""

    prefetch: PrefetchOffloadConfig = Field(default_factory=PrefetchOffloadConfig)
    """Parameters for prefetch offloading backend."""

    @model_validator(mode="after")
    def validate_offload_config(self) -> "OffloadConfig":
        """Validate offload configuration constraints."""
        if self.offload_backend == "prefetch" or self.prefetch.offload_group_size > 0:
            if self.prefetch.offload_num_in_group > self.prefetch.offload_group_size:
                raise ValueError(
                    f"offload_num_in_group ({self.prefetch.offload_num_in_group})"
                    f" must be <= offload_group_size"
                    f" ({self.prefetch.offload_group_size})"
                )
            if self.prefetch.offload_prefetch_step < 1:
                raise ValueError(
                    f"offload_prefetch_step"
                    f" ({self.prefetch.offload_prefetch_step})"
                    f" must be >= 1 when prefetch offloading is enabled"
                    f" (offload_group_size > 0)"
                )

        # Warn if both backends have non-default values
        uva_active = self.uva.cpu_offload_gb > 0
        prefetch_active = self.prefetch.offload_group_size > 0
        if self.offload_backend == "uva" and prefetch_active:
            warnings.warn(
                "Prefetch offload fields are set but offload_backend='uva'. "

View on GitHub (pinned to c794754062)

Solutions

Lower offload_num_in_group to at most offload_group_size (e.g. group 4, num_in_group 4).
Or raise offload_group_size to be >= offload_num_in_group.
Re-read the prefetch offloader docs to pick a sane ratio (commonly num_in_group == group_size or half).

Example fix

# before
vllm serve model --offload-backend prefetch --offload-num-in-group 8 --offload-group-size 4

# after
vllm serve model --offload-backend prefetch --offload-num-in-group 4 --offload-group-size 4

Defensive patterns

Strategy: validation

Validate before calling

def check_offload_group(num_in_group: int, group_size: int) -> None:
    if group_size > 0 and num_in_group > group_size:
        raise SystemExit(f"offload_num_in_group ({num_in_group}) must be <= offload_group_size ({group_size})")

Prevention

Derive num_in_group from group_size in your config generator instead of hardcoding both.
When tuning offloading, change group_size first and recompute num_in_group.

When it happens

Trigger: Passing --offload-num-in-group 8 with --offload-group-size 4 (or defaults/CLI combos where num_in_group > group_size), either with backend 'prefetch' explicitly or merely a positive group size.

Common situations: Tuning CPU offloading knobs independently without understanding their relationship; setting num_in_group from an old benchmark note while shrinking group_size to save memory.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/f56c538f6aa629cd. Report an issue: GitHub.