huggingface/transformers · error · ValueError

Unknown modality for: {model_classname}

Error message

Unknown modality for: {model_classname}

What it means

ModelManager.get_model_modality classifies a loaded model by matching its class name against the auto-mappings MODEL_FOR_MULTIMODAL_LM / MODEL_FOR_IMAGE_TEXT_TO_TEXT / MODEL_FOR_CAUSAL_LM. If the model's class is registered in none of them (custom architecture, non-generative model, or a class not in the mappings' values), it raises ValueError 'Unknown modality'.

Source

Thrown at src/transformers/cli/serving/model_manager.py:444

        """
        if processor is not None and isinstance(processor, PreTrainedTokenizerBase):
            return Modality.LLM

        from transformers.models.auto.modeling_auto import (
            MODEL_FOR_CAUSAL_LM_MAPPING_NAMES,
            MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES,
            MODEL_FOR_MULTIMODAL_LM_MAPPING_NAMES,
        )

        model_classname = model.__class__.__name__
        if model_classname in MODEL_FOR_MULTIMODAL_LM_MAPPING_NAMES.values():
            return Modality.MULTIMODAL
        elif model_classname in MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES.values():
            return Modality.VLM
        elif model_classname in MODEL_FOR_CAUSAL_LM_MAPPING_NAMES.values():
            return Modality.LLM
        else:
            raise ValueError(f"Unknown modality for: {model_classname}")

    @staticmethod
    @lru_cache
    def get_gen_models(cache_dir: str | None = None) -> list[dict]:
        """List generative models (LLMs and VLMs) available in the HuggingFace cache.

        Args:
            cache_dir (`str`, *optional*): Path to the HuggingFace cache directory.
                Defaults to the standard cache location.

        Returns:
            `list[dict]`: OpenAI-compatible model list entries with ``id``, ``object``, etc.
        """
        from transformers.models.auto.modeling_auto import (
            MODEL_FOR_CAUSAL_LM_MAPPING_NAMES,
            MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES,
            MODEL_FOR_MULTIMODAL_LM_MAPPING_NAMES,
        )

View on GitHub (pinned to a597f97485)

Solutions

1. Serve a model class that is registered in the causal-LM / image-text-to-text / multimodal auto mappings
2. For custom classes, register them in the mapping or wrap/serv them with your own FastAPI app
3. Upgrade transformers so newer model classes appear in the mappings
4. Check membership first: any(model.__class__.__name__ in m.values() for m in (MODEL_FOR_CAUSAL_LM_MAPPING_NAMES, ...))

Example fix

# before: custom class
serve --model_id ./my-custom-arch  # ValueError: Unknown modality

# after: serve a mapped architecture
serve --model_id meta-llama/Llama-3.1-8B-Instruct

from transformers.models.auto.modeling_auto import (
    MODEL_FOR_CAUSAL_LM_MAPPING_NAMES,
    MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES,
    MODEL_FOR_MULTIMODAL_LM_MAPPING_NAMES,
)

classname = type(model).__name__
known = any(
    classname in m.values()
    for m in (MODEL_FOR_CAUSAL_LM_MAPPING_NAMES, MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES, MODEL_FOR_MULTIMODAL_LM_MAPPING_NAMES)
)
if not known:
    print("Model class not routable by serve; use a mapped architecture")

def is_servable_class(model) -> bool:
    name = model.__class__.__name__
    return any(
        name in m.values()
        for m in (
            MODEL_FOR_CAUSAL_LM_MAPPING_NAMES,
            MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES,
            MODEL_FOR_MULTIMODAL_LM_MAPPING_NAMES,
        )
    )

try:
    modality = manager.get_model_modality(model, processor=processor)
except ValueError as e:
    if "Unknown modality" in str(e):
        raise SystemExit("Serve a causal-LM/VLM/multimodal model or register your class") from e
    raise

When it happens

Trigger: Loading a custom PreTrainedModel subclass (e.g. your own architecture) and hitting any endpoint that needs modality routing; a generative model class missing from the three mappings (e.g. seq2seq or audio models); a model registered only under MODEL_FOR_SEQ2SEQ or MODEL_FOR_SPEECH mappings.

Common situations: Serving a custom fine-tuned architecture; serving encoder-decoder (seq2seq) models the serve CLI does not route; brand-new model classes before their mapping lands in your transformers version.

Related errors

• Unsupported dtype: '{dtype}'. Must be 'auto' or a valid torc
• Unsupported quantization method: '{self.quantization}'. Must
• Unsupported attention implementation: '{self.attn_implementa
• `axis_value` for `HQQ` backend has to be one of [`0`, `1`] b
• You can construct a Cache either from a list `layers` of all

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/4e3a728282f26b03. Report an issue: GitHub.