hiyouga/LlamaFactory · error · ValueError

Global batch size must be divisible by DP size and micro bat

Error message

Global batch size must be divisible by DP size and micro batch size. Got {global_batch_size} % ({dp_size} * {micro_batch_size}) != 0.

What it means

The v1 batching module derives num_micro_batch = global_batch_size / dp_size / micro_batch_size and requires exact divisibility. If global_batch_size % (dp_size * micro_batch_size) != 0 the gradient-accumulation arithmetic is non-integral, which the scheduler cannot honor, so it raises at construction time.

Source

Thrown at src/llamafactory/v1/core/utils/batching.py:192

        self.micro_batch_size = micro_batch_size
        self.global_batch_size = global_batch_size
        self.cutoff_len = cutoff_len
        self.batching_workers = batching_workers
        self.batching_strategy = batching_strategy
        self.pin_memory = pin_memory
        self.drop_last = drop_last
        self.seed = seed
        self._warned_truncation = False  # warn once when dropping fully-truncated (zero-loss) samples
        # TODO: support length and infinity
        dp_size = DistributedInterface().get_world_size(Dim.DP)

        if self.global_batch_size is None:
            self.global_batch_size = dp_size * micro_batch_size
            self.num_micro_batch = 1
        elif self.global_batch_size % (dp_size * micro_batch_size) == 0:
            self.num_micro_batch = global_batch_size // dp_size // micro_batch_size
        else:
            raise ValueError(
                "Global batch size must be divisible by DP size and micro batch size. "
                f"Got {global_batch_size} % ({dp_size} * {micro_batch_size}) != 0."
            )

        if not self.drop_last:
            raise ValueError("Drop last must be True.")

        self._batch_info: BatchInfo = {
            "micro_batch_size": self.micro_batch_size,
            "num_micro_batch": self.num_micro_batch,
            "cutoff_len": self.cutoff_len,
        }

        self._init_data_provider()

        self._is_resuming: bool = False
        self._data_iter = iter(self._data_provider)
        self._buffer = StatefulBuffer()

View on GitHub (pinned to f28afaf635)

Solutions

1. Leave global_batch_size unset (None) — it defaults to dp_size * micro_batch_size with num_micro_batch=1
2. Or pick global_batch_size = k * dp_size * micro_batch_size for integer k (e.g. 2 * 4 * 32 = 256)
3. Adjust micro_batch_size so the product divides your desired global batch

Example fix

# before (dp=4, micro=32)
scheduler = Batching(global_batch_size=100, micro_batch_size=32, drop_last=True)  # 100 % 128 != 0

# after
scheduler = Batching(global_batch_size=128, micro_batch_size=32, drop_last=True)  # num_micro_batch=1

def batch_sizes_consistent(global_batch_size, dp_size, micro_batch_size) -> bool:
    return global_batch_size % (dp_size * micro_batch_size) == 0

When it happens

Trigger: Constructing the v1 batch scheduler with explicit global_batch_size (e.g. 96) while dp_size * micro_batch_size (e.g. 8 * 14 = 112, or 4 * 32 = 128) does not divide it; changing per_device_train_batch_size or world size without adjusting global batch.

Common situations: Scaling a working single-GPU config to multi-GPU (DP size changes the divisor); odd micro batch sizes from packing/dynamic strategies; setting global_batch_size equal to token-count-derived values.

Related errors

• Drop last must be True.
• Unsupported model type: {getattr(config, 'model_type')}.
• Template is required for MultiModalDataCollator.
• Cannot specify `val_size` if `eval_dataset` is not None.
• `tensor_model_parallel_size` must be >= 1.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/a454125b9b9c56bd. Report an issue: GitHub.