huggingface/pytorch-image-models · warning

Rank {self.rank}: Ran out of samples ({idx_pos}/{effective_s

Error message

Rank {self.rank}: Ran out of samples ({idx_pos}/{effective_samples_this_rank}) before processing entire schedule. Check schedule generation.

What it means

While slicing the per-rank index list into batches according to the (shuffled) schedule, NaFlexDataset detects that the schedule's batches exhaust the available indices before all effective samples are covered — actual_bs clamped to <= 0 while scheduled_samples_count < effective_samples_this_rank. This indicates the schedule total is smaller than the number of indices (schedule/indices mismatch) and iteration stops early.

Source

Thrown at timm/data/naflex_dataset.py:467

        # 4. Shuffle the order of the canonical batch schedule for this epoch
        if self.shuffle:
            schedule_perm = torch.randperm(self._num_batches_per_rank, generator=g).tolist()
            shuffled_schedule = [self._canonical_batch_schedule[i] for i in schedule_perm]
        else:
            shuffled_schedule = list(self._canonical_batch_schedule) # Keep original order

        # 5. Assign indices and patch-size choices to the shuffled batches
        epoch_batches = []
        patch_size_probs = torch.tensor(self.patch_size_probs)
        idx_pos = 0
        scheduled_samples_count = 0
        for seq_len, bs in shuffled_schedule:
            # Ensure we don't try to grab more indices than available for the rank
            actual_bs = min(bs, effective_samples_this_rank - idx_pos)
            if actual_bs <= 0:
                 if scheduled_samples_count < effective_samples_this_rank:
                     # This indicates mismatch between schedule total and actual samples
                     warnings.warn(f"Rank {self.rank}: Ran out of samples ({idx_pos}/{effective_samples_this_rank}) before processing entire schedule. Check schedule generation.")
                 break # Stop if no more indices or batch size is zero

            batch_indices = indices_this_rank[idx_pos : idx_pos + actual_bs]
            patch_idx = 0
            if self.variable_patch_size:
                patch_idx = torch.multinomial(patch_size_probs, 1, generator=g).item()
            epoch_batches.append((seq_len, patch_idx, batch_indices))
            idx_pos += actual_bs
            scheduled_samples_count += actual_bs

        # Final check
        if scheduled_samples_count != effective_samples_this_rank:
             warnings.warn(
                f"Rank {self.rank}: Assigned {scheduled_samples_count} samples to batches, "
                f"but expected {effective_samples_this_rank} effective samples this epoch. "
                f"Indices remaining: {effective_samples_this_rank - scheduled_samples_count}."
             )
        return epoch_batches

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Resolve upstream schedule/indices mismatch (align dataset size, world_size, min_batch_size, batch_divisor so the schedule covers all samples)
2. If using persistent workers, ensure the shared epoch state is regenerated consistently every epoch
3. Upgrade timm if the schedule shuffle math was patched

Example fix

# before
NaFlexDataset(ds, ..., min_batch_size=8, world_size=7)  # uneven
# after
NaFlexDataset(ds, ..., min_batch_size=1, world_size=7)

sched_total = sum(bs for _, bs in schedule)\nassert sched_total == effective_samples, f'schedule {sched_total} != samples {effective_samples}'

When it happens

Trigger: Following warnings 267/268: schedule total < effective samples; shuffled schedule where a large batch consumed indices early leaving schedule entries with nothing to take; epoch regeneration with persistent workers reading stale shared state.

Common situations: Distributed NaFlex training with non-divisible sample counts; persistent DataLoader workers reusing a shared epoch state that drifted from a newly computed schedule.

Related errors

• Rank {self.rank}: Assigned {scheduled_samples_count} samples
• Rank {self.rank}: Canonical schedule accounts for {total_sch
• Rank {self.rank}: Number of indices for this rank ({len(indi
• Error processing sample index {idx}. Error: {e}. Skipping sa
• Calculated batch size <= 0 (seq_len={seq_len}, remaining={re

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/0725f114ae95bb7e. Report an issue: GitHub.