huggingface/pytorch-image-models · warning

IndexError encountered for index {idx} (possibly due to padd

Error message

IndexError encountered for index {idx} (possibly due to padding/repeated indices). Skipping sample.

What it means

While iterating, NaFlexDataset catches IndexError from base_dataset[idx] or the transform — typically when padding added indices beyond the real dataset length — warns and skips. It is the expected mechanism for tolerating padded/duplicate indices rather than crashing the loader.

Source

Thrown at timm/data/naflex_dataset.py:551

            batch_imgs = []
            batch_targets = []
            for idx in indices:
                try:
                    # Get original image and label from map-style dataset
                    img, label = self.base_dataset[idx]

                    # Apply transform if available
                    # Handle cases where transform might return None or fail
                    processed_img = transform(img) if transform else img
                    if processed_img is None:
                        warnings.warn(f"Transform returned None for index {idx}. Skipping sample.")
                        continue

                    batch_imgs.append(processed_img)
                    batch_targets.append(label)

                except IndexError:
                     warnings.warn(f"IndexError encountered for index {idx} (possibly due to padding/repeated indices). Skipping sample.")
                     continue
                except Exception as e:
                    # Log other potential errors during data loading/processing
                    warnings.warn(f"Error processing sample index {idx}. Error: {e}. Skipping sample.")
                    continue # Skip problematic sample

            if self.mixup_fn is not None:
                batch_imgs, batch_targets = self.mixup_fn(batch_imgs, batch_targets)

            batch_imgs = [batch_patchifier(img) for img in batch_imgs]
            batch_samples = list(zip(batch_imgs, batch_targets))
            if batch_samples: # Only yield if we successfully processed samples
                # Collate the processed samples into a batch
                yield self.collate_fns[seq_len](batch_samples)

            # If batch_samples is empty after processing 'indices', an empty batch is skipped.

View on GitHub (pinned to 9a5261e31b)

Solutions

1. If only pad indices trigger it, it's expected behavior — no action needed
2. If frequent, verify how indices are generated vs len(base_dataset) (e.g. Subset ranges)
3. Ensure base_dataset length is stable between schedule creation and iteration

Example fix

# before
idxs = list(range(len(parent_ds)))  # later wrapped in Subset(base, idxs) of smaller size
# after
sub = Subset(base, idxs)
idxs = list(range(len(sub)))  # schedule from the actual visible length

assert all(0 <= i < len(base_dataset) for i in batch_indices), 'index out of dataset range'

When it happens

Trigger: Distributed padding extends indices past len(base_dataset) (pad indices >= dataset size); an out-of-range idx computed from a mismatched schedule; a Subset/wrapper whose length shrank after caching indices.

Common situations: Distributed training with drop_last=False-style padding; mixing Subset views with indices computed from the parent dataset. Occasional skips are by design; many skips mean index computation is wrong.

Related errors

• Error processing sample index {idx}. Error: {e}. Skipping sa
• Calculated batch size <= 0 (seq_len={seq_len}, remaining={re
• Rank {self.rank}: Canonical schedule accounts for {total_sch
• Rank {self.rank}: Number of indices for this rank ({len(indi
• Rank {self.rank}: Ran out of samples ({idx_pos}/{effective_s

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/df5434c2c594046f. Report an issue: GitHub.