{"record":{"id":"0725f114ae95bb7e","repo":"huggingface/pytorch-image-models","slug":"rank-self-rank-ran-out-of-samples-idx-pos-e","errorCode":null,"errorMessage":"Rank {self.rank}: Ran out of samples ({idx_pos}/{effective_samples_this_rank}) before processing entire schedule. Check schedule generation.","messagePattern":"Rank (.+?): Ran out of samples \\((.+?)/(.+?)\\) before processing entire schedule\\. Check schedule generation\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"timm/data/naflex_dataset.py","lineNumber":467,"sourceCode":"        # 4. Shuffle the order of the canonical batch schedule for this epoch\n        if self.shuffle:\n            schedule_perm = torch.randperm(self._num_batches_per_rank, generator=g).tolist()\n            shuffled_schedule = [self._canonical_batch_schedule[i] for i in schedule_perm]\n        else:\n            shuffled_schedule = list(self._canonical_batch_schedule) # Keep original order\n\n        # 5. Assign indices and patch-size choices to the shuffled batches\n        epoch_batches = []\n        patch_size_probs = torch.tensor(self.patch_size_probs)\n        idx_pos = 0\n        scheduled_samples_count = 0\n        for seq_len, bs in shuffled_schedule:\n            # Ensure we don't try to grab more indices than available for the rank\n            actual_bs = min(bs, effective_samples_this_rank - idx_pos)\n            if actual_bs <= 0:\n                 if scheduled_samples_count < effective_samples_this_rank:\n                     # This indicates mismatch between schedule total and actual samples\n                     warnings.warn(f\"Rank {self.rank}: Ran out of samples ({idx_pos}/{effective_samples_this_rank}) before processing entire schedule. Check schedule generation.\")\n                 break # Stop if no more indices or batch size is zero\n\n            batch_indices = indices_this_rank[idx_pos : idx_pos + actual_bs]\n            patch_idx = 0\n            if self.variable_patch_size:\n                patch_idx = torch.multinomial(patch_size_probs, 1, generator=g).item()\n            epoch_batches.append((seq_len, patch_idx, batch_indices))\n            idx_pos += actual_bs\n            scheduled_samples_count += actual_bs\n\n        # Final check\n        if scheduled_samples_count != effective_samples_this_rank:\n             warnings.warn(\n                f\"Rank {self.rank}: Assigned {scheduled_samples_count} samples to batches, \"\n                f\"but expected {effective_samples_this_rank} effective samples this epoch. \"\n                f\"Indices remaining: {effective_samples_this_rank - scheduled_samples_count}.\"\n             )\n        return epoch_batches","sourceCodeStart":449,"sourceCodeEnd":485,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/data/naflex_dataset.py#L449-L485","documentation":"While slicing the per-rank index list into batches according to the (shuffled) schedule, NaFlexDataset detects that the schedule's batches exhaust the available indices before all effective samples are covered — actual_bs clamped to <= 0 while scheduled_samples_count < effective_samples_this_rank. This indicates the schedule total is smaller than the number of indices (schedule/indices mismatch) and iteration stops early.","triggerScenarios":"Following warnings 267/268: schedule total < effective samples; shuffled schedule where a large batch consumed indices early leaving schedule entries with nothing to take; epoch regeneration with persistent workers reading stale shared state.","commonSituations":"Distributed NaFlex training with non-divisible sample counts; persistent DataLoader workers reusing a shared epoch state that drifted from a newly computed schedule.","solutions":["Resolve upstream schedule/indices mismatch (align dataset size, world_size, min_batch_size, batch_divisor so the schedule covers all samples)","If using persistent workers, ensure the shared epoch state is regenerated consistently every epoch","Upgrade timm if the schedule shuffle math was patched"],"exampleFix":"# before\nNaFlexDataset(ds, ..., min_batch_size=8, world_size=7)  # uneven\n# after\nNaFlexDataset(ds, ..., min_batch_size=1, world_size=7)","handlingStrategy":"validation","validationCode":"sched_total = sum(bs for _, bs in schedule)\\nassert sched_total == effective_samples, f'schedule {sched_total} != samples {effective_samples}'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Fix upstream warnings 267/268 first — this is a symptom","Regenerate epoch schedules atomically with persistent workers"],"tags":["naflex","distributed","schedule-mismatch","timm"],"backgroundTag":"uneven-data-sharding","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}