huggingface/pytorch-image-models · warning

Rank {self.rank}: Canonical schedule accounts for {total_sch

Error message

Rank {self.rank}: Canonical schedule accounts for {total_scheduled_samples} samples, but expected {num_samples_per_rank} samples per rank. This might happen if min_batch_size or batch_divisor constraints prevent utilizing all samples. Check parameters. Remaining samples: {remaining_samples}

What it means

After building the per-rank batch schedule, NaFlexDataset checks that total scheduled samples equals num_samples_per_rank. A warning means constraints (min_batch_size, batch_divisor, max_seq_len packing) made it impossible to schedule every sample — e.g. leftover samples fewer than min_batch_size that can't form a batch — so some data is dropped from the epoch.

Source

Thrown at timm/data/naflex_dataset.py:379

                max_size=remaining_samples,
                divisor=self.batch_divisor,
                rounding='floor',
            )
            # Ensure batch size is positive and doesn't exceed remaining samples
            batch_size = max(1, batch_size)
            batch_size = min(batch_size, remaining_samples)

            if batch_size <= 0:
                 warnings.warn(f"Calculated batch size <= 0 (seq_len={seq_len}, remaining={remaining_samples}). Stopping schedule generation early.")
                 break # Avoid infinite loop if something goes wrong

            current_schedule.append((seq_len, batch_size))
            remaining_samples -= batch_size
            total_scheduled_samples += batch_size

        # Sanity check: Ensure the schedule covers all samples for the rank
        if total_scheduled_samples != num_samples_per_rank:
            warnings.warn(
                f"Rank {self.rank}: Canonical schedule accounts for {total_scheduled_samples} samples, "
                f"but expected {num_samples_per_rank} samples per rank. "
                f"This might happen if min_batch_size or batch_divisor constraints prevent utilizing all samples. "
                f"Check parameters. Remaining samples: {remaining_samples}"
            )
            # Adjust if needed? Could add a final small batch, but might violate constraints.
            # Current behavior: some samples might be dropped if schedule logic fails.

        self._canonical_batch_schedule = current_schedule
        self._num_batches_per_rank = len(current_schedule)
        print(f"Rank {self.rank}: Created canonical schedule with {self._num_batches_per_rank} batches for {self._padded_samples_per_rank} samples/rank.")


    def _prepare_epoch_batches(self, epoch: int) -> List[Tuple[int, int, List[int]]]:
        """
        Prepares the batches for the current epoch by:
        1. Shuffling the full dataset indices (using epoch seed).
        2. Applying padding if in distributed mode.

View on GitHub (pinned to 9a5261e31b)

Solutions

  1. Accept it if only a handful of samples are dropped (expected behavior)
  2. Choose batch_divisor/min_batch_size so per-rank samples divide evenly, or pad the dataset length to a multiple
  3. Reduce min_batch_size or adjust max_seq_len so a final small batch is possible

Example fix

# before
ds = NaFlexDataset(ds, ..., min_batch_size=8)  # 1001 samples/rank -> 1 sample unscheduled
# after
ds = NaFlexDataset(ds, ..., min_batch_size=1)  # remainder can be scheduled
Defensive patterns

Strategy: validation

Validate before calling

per_rank = math.ceil(len(ds) / world_size)\nassert per_rank % batch_divisor == 0 or min_batch_size == 1, 'samples will be dropped from schedule'

Prevention

When it happens

Trigger: num_samples_per_rank not divisible by batch_divisor, leaving a remainder smaller than min_batch_size; very large min_batch_size relative to per-rank dataset size; max_seq_len forcing tiny batches that can't absorb the remainder.

Common situations: Small datasets or large world sizes where per-rank samples are few; odd dataset sizes like 1001 samples with min_batch_size=8. Impact is usually negligible (a few samples skipped per epoch).

Related errors

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/bfabf5f6f218761c. Report an issue: GitHub.