{"record":{"id":"a454125b9b9c56bd","repo":"hiyouga/LlamaFactory","slug":"global-batch-size-must-be-divisible-by-dp-size-and","errorCode":null,"errorMessage":"Global batch size must be divisible by DP size and micro batch size. Got {global_batch_size} % ({dp_size} * {micro_batch_size}) != 0.","messagePattern":"Global batch size must be divisible by DP size and micro batch size\\. Got (.+?) % \\((.+?) \\* (.+?)\\) != 0\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/core/utils/batching.py","lineNumber":192,"sourceCode":"        self.micro_batch_size = micro_batch_size\n        self.global_batch_size = global_batch_size\n        self.cutoff_len = cutoff_len\n        self.batching_workers = batching_workers\n        self.batching_strategy = batching_strategy\n        self.pin_memory = pin_memory\n        self.drop_last = drop_last\n        self.seed = seed\n        self._warned_truncation = False  # warn once when dropping fully-truncated (zero-loss) samples\n        # TODO: support length and infinity\n        dp_size = DistributedInterface().get_world_size(Dim.DP)\n\n        if self.global_batch_size is None:\n            self.global_batch_size = dp_size * micro_batch_size\n            self.num_micro_batch = 1\n        elif self.global_batch_size % (dp_size * micro_batch_size) == 0:\n            self.num_micro_batch = global_batch_size // dp_size // micro_batch_size\n        else:\n            raise ValueError(\n                \"Global batch size must be divisible by DP size and micro batch size. \"\n                f\"Got {global_batch_size} % ({dp_size} * {micro_batch_size}) != 0.\"\n            )\n\n        if not self.drop_last:\n            raise ValueError(\"Drop last must be True.\")\n\n        self._batch_info: BatchInfo = {\n            \"micro_batch_size\": self.micro_batch_size,\n            \"num_micro_batch\": self.num_micro_batch,\n            \"cutoff_len\": self.cutoff_len,\n        }\n\n        self._init_data_provider()\n\n        self._is_resuming: bool = False\n        self._data_iter = iter(self._data_provider)\n        self._buffer = StatefulBuffer()","sourceCodeStart":174,"sourceCodeEnd":210,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/core/utils/batching.py#L174-L210","documentation":"The v1 batching module derives num_micro_batch = global_batch_size / dp_size / micro_batch_size and requires exact divisibility. If global_batch_size % (dp_size * micro_batch_size) != 0 the gradient-accumulation arithmetic is non-integral, which the scheduler cannot honor, so it raises at construction time.","triggerScenarios":"Constructing the v1 batch scheduler with explicit global_batch_size (e.g. 96) while dp_size * micro_batch_size (e.g. 8 * 14 = 112, or 4 * 32 = 128) does not divide it; changing per_device_train_batch_size or world size without adjusting global batch.","commonSituations":"Scaling a working single-GPU config to multi-GPU (DP size changes the divisor); odd micro batch sizes from packing/dynamic strategies; setting global_batch_size equal to token-count-derived values.","solutions":["Leave global_batch_size unset (None) — it defaults to dp_size * micro_batch_size with num_micro_batch=1","Or pick global_batch_size = k * dp_size * micro_batch_size for integer k (e.g. 2 * 4 * 32 = 256)","Adjust micro_batch_size so the product divides your desired global batch"],"exampleFix":"# before (dp=4, micro=32)\nscheduler = Batching(global_batch_size=100, micro_batch_size=32, drop_last=True)  # 100 % 128 != 0\n\n# after\nscheduler = Batching(global_batch_size=128, micro_batch_size=32, drop_last=True)  # num_micro_batch=1","handlingStrategy":"validation","validationCode":"def batch_sizes_consistent(global_batch_size, dp_size, micro_batch_size) -> bool:\n    return global_batch_size % (dp_size * micro_batch_size) == 0","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Omit global_batch_size unless you specifically need gradient accumulation","Re-validate batch config whenever world size or micro batch changes"],"tags":["batching","distributed","config","training"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}