{"record":{"id":"bfabf5f6f218761c","repo":"huggingface/pytorch-image-models","slug":"rank-self-rank-canonical-schedule-accounts-for","errorCode":null,"errorMessage":"Rank {self.rank}: Canonical schedule accounts for {total_scheduled_samples} samples, but expected {num_samples_per_rank} samples per rank. This might happen if min_batch_size or batch_divisor constraints prevent utilizing all samples. Check parameters. Remaining samples: {remaining_samples}","messagePattern":"Rank (.+?): Canonical schedule accounts for (.+?) samples, but expected (.+?) samples per rank\\. This might happen if min_batch_size or batch_divisor constraints prevent utilizing all samples\\. Check parameters\\. Remaining samples: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"timm/data/naflex_dataset.py","lineNumber":379,"sourceCode":"                max_size=remaining_samples,\n                divisor=self.batch_divisor,\n                rounding='floor',\n            )\n            # Ensure batch size is positive and doesn't exceed remaining samples\n            batch_size = max(1, batch_size)\n            batch_size = min(batch_size, remaining_samples)\n\n            if batch_size <= 0:\n                 warnings.warn(f\"Calculated batch size <= 0 (seq_len={seq_len}, remaining={remaining_samples}). Stopping schedule generation early.\")\n                 break # Avoid infinite loop if something goes wrong\n\n            current_schedule.append((seq_len, batch_size))\n            remaining_samples -= batch_size\n            total_scheduled_samples += batch_size\n\n        # Sanity check: Ensure the schedule covers all samples for the rank\n        if total_scheduled_samples != num_samples_per_rank:\n            warnings.warn(\n                f\"Rank {self.rank}: Canonical schedule accounts for {total_scheduled_samples} samples, \"\n                f\"but expected {num_samples_per_rank} samples per rank. \"\n                f\"This might happen if min_batch_size or batch_divisor constraints prevent utilizing all samples. \"\n                f\"Check parameters. Remaining samples: {remaining_samples}\"\n            )\n            # Adjust if needed? Could add a final small batch, but might violate constraints.\n            # Current behavior: some samples might be dropped if schedule logic fails.\n\n        self._canonical_batch_schedule = current_schedule\n        self._num_batches_per_rank = len(current_schedule)\n        print(f\"Rank {self.rank}: Created canonical schedule with {self._num_batches_per_rank} batches for {self._padded_samples_per_rank} samples/rank.\")\n\n\n    def _prepare_epoch_batches(self, epoch: int) -> List[Tuple[int, int, List[int]]]:\n        \"\"\"\n        Prepares the batches for the current epoch by:\n        1. Shuffling the full dataset indices (using epoch seed).\n        2. Applying padding if in distributed mode.","sourceCodeStart":361,"sourceCodeEnd":397,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/data/naflex_dataset.py#L361-L397","documentation":"After building the per-rank batch schedule, NaFlexDataset checks that total scheduled samples equals num_samples_per_rank. A warning means constraints (min_batch_size, batch_divisor, max_seq_len packing) made it impossible to schedule every sample — e.g. leftover samples fewer than min_batch_size that can't form a batch — so some data is dropped from the epoch.","triggerScenarios":"num_samples_per_rank not divisible by batch_divisor, leaving a remainder smaller than min_batch_size; very large min_batch_size relative to per-rank dataset size; max_seq_len forcing tiny batches that can't absorb the remainder.","commonSituations":"Small datasets or large world sizes where per-rank samples are few; odd dataset sizes like 1001 samples with min_batch_size=8. Impact is usually negligible (a few samples skipped per epoch).","solutions":["Accept it if only a handful of samples are dropped (expected behavior)","Choose batch_divisor/min_batch_size so per-rank samples divide evenly, or pad the dataset length to a multiple","Reduce min_batch_size or adjust max_seq_len so a final small batch is possible"],"exampleFix":"# before\nds = NaFlexDataset(ds, ..., min_batch_size=8)  # 1001 samples/rank -> 1 sample unscheduled\n# after\nds = NaFlexDataset(ds, ..., min_batch_size=1)  # remainder can be scheduled","handlingStrategy":"validation","validationCode":"per_rank = math.ceil(len(ds) / world_size)\\nassert per_rank % batch_divisor == 0 or min_batch_size == 1, 'samples will be dropped from schedule'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pick dataset sizes/divisors so per-rank samples pack evenly","Accept small remainders for large datasets"],"tags":["naflex","batch-schedule","distributed","timm"],"backgroundTag":"uneven-data-sharding","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}