huggingface/pytorch-image-models · error · ValueError

Unsupported distill_type '{distill_type}'. Must be 'soft' or

Error message

Unsupported distill_type '{distill_type}'. Must be 'soft' or 'hard'.

What it means

TokenDistillation.__init__ validates its distill_type argument and only accepts 'soft' or 'hard'. Any other string (e.g. 'Soft', 'KD', 'logit') raises ValueError at construction time. This is a config-validation error for the token-level distillation wrapper in timm.task.token_distillation.

Source

Thrown at timm/task/token_distillation.py:234

                in_chans=in_chans,
                pretrained_path=teacher_pretrained_path,
                device=self.device,
                dtype=self.dtype,
            )
        else:
            raise TypeError(
                f"teacher_model must be a model name string, nn.Module, or TokenDistillationTeacher, "
                f"got {type(teacher_model).__name__}"
            )

        self.trainable_module = student_model
        self.teacher = teacher
        self.criterion = criterion if criterion is not None else nn.CrossEntropyLoss()
        self.distill_type = distill_type
        self.temperature = temperature

        if distill_type not in ('soft', 'hard'):
            raise ValueError(f"Unsupported distill_type '{distill_type}'. Must be 'soft' or 'hard'.")

        # Register student normalization values as non-persistent buffers
        student_mean = torch.tensor(
            student_unwrapped.pretrained_cfg['mean'],
            device=self.device,
            dtype=self.dtype,
        ).view(1, -1, 1, 1)
        student_std = torch.tensor(
            student_unwrapped.pretrained_cfg['std'],
            device=self.device,
            dtype=self.dtype,
        ).view(1, -1, 1, 1)
        self.register_buffer('student_mean', student_mean, persistent=False)
        self.register_buffer('student_std', student_std, persistent=False)

        # Determine weighting mode
        if distill_loss_weight is not None:
            # Mode 1: distill_weight specified - independent weights (task defaults to 1.0 if not set)

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Set distill_type to exactly 'soft' (KL-divergence over softened logits) or 'hard' (hard-label CE)
2. Check for typos/case in the config value feeding distill_type
3. Omit distill_type if you want the default (typically 'soft')

Example fix

# before
distiller = TokenDistillation(teacher, student, distill_type='feature')
# after
distiller = TokenDistillation(teacher, student, distill_type='soft')

assert distill_type in ('soft', 'hard'), f"bad distill_type: {distill_type}"

When it happens

Trigger: Constructing TokenDistillation(teacher, student, distill_type='feature') or passing a typo like distill_type='sotf' or a non-lowercase variant such as 'Soft'.

Common situations: Copy-pasted config from another distillation library with different type names; case mismatch; passing None or an empty string when the default was expected.

Related errors

• Preset '{value}' is empty or invalid
• Coefficient list cannot be empty
• Invalid learning rate: {}
• Invalid learning rate: {lr}
• Invalid epsilon value: {eps}

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/c122e30d90e01b15. Report an issue: GitHub.