dotnet/machinelearning · error · ArgumentException

The model type ' ' is not supported.

Error message

The model type '{modelProto.TrainerSpec.ModelType}' is not supported.

What it means

The SentencePieceTokenizer constructor dispatches on ModelProto.TrainerSpec.ModelType and only supports Bpe and Unigram. Any other model type (e.g. WordPiece-derived or unspecified SentencePiece exports) falls to the default arm and throws ArgumentException naming modelProto.

Solutions

1. Retrain/re-export the model with --model_type=bpe or unigram.
2. Convert the model to a supported type, or load it via a different tokenizer (e.g. Hugging Face tokenizer.json with CreateFromTokenizerJson).
3. Inspect modelProto.TrainerSpec.ModelType with the protobuf API before loading to confirm support.
4. Check the file is genuinely a SentencePiece protobuf, not another format.

Example fix

# before: sentencepiece_trainer --input=corpus.txt --model_type=word --model_prefix=m
# after
sentencepiece_trainer --input=corpus.txt --model_type=unigram --model_prefix=m

var proto = ModelProto.Parser.ParseFrom(modelStream);
if (proto.TrainerSpec.ModelType != TrainerSpec.Types.ModelType.Bpe && proto.TrainerSpec.ModelType != TrainerSpec.Types.ModelType.Unigram)
    throw new NotSupportedException($"Unsupported SentencePiece model type: {proto.TrainerSpec.ModelType}");

try { var tok = SentencePieceTokenizer.Create(stream); } catch (ArgumentException ex) { // fall back to another loader or retrain path
    logger.LogError(ex, "Unsupported SentencePiece model type"); }

When it happens

Trigger: Loading a .model sentencepiece file whose trainer_spec.model_type is neither BPE nor UNIGRAM (e.g. WORD model, corrupt/unset trainer spec) and passing it to SentencePieceTokenizer.Create / the internal constructor.

Common situations: Exporting a SentencePiece model trained with --model_type=word or bpe-less tools; models produced by non-SentencePiece tools saved with the .sentencepiece extension; protobuf files whose trainer_spec was stripped.

Related errors

• A tokenizer.json normalizer entry must be a JSON object.
• ArgumentNullException
• ArgumentNullException
• Blob for normalization rule is broken.
• Expected model type 'Unigram' but found

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/960f94f8e70027bb. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:30

namespace Microsoft.ML.Tokenizers
{
    // SentencePieceBpe is implementing the BPE algorithm based on the SentencePiece https://github.com/google/sentencepiece.
    // SentencePiece is under the Apache License 2.0 https://github.com/google/sentencepiece/blob/master/LICENSE

    /// <summary>
    /// SentencePieceBpe is a tokenizer that splits the input into tokens using the SentencePiece Bpe model.
    /// </summary>
    public class SentencePieceTokenizer : Tokenizer
    {
        private readonly SentencePieceBaseModel _model;

        internal SentencePieceTokenizer(ModelProto modelProto, bool addBos, bool addEos, IReadOnlyDictionary<string, int>? specialTokens = null)
        {
            _model = modelProto.TrainerSpec.ModelType switch
            {
                TrainerSpec.Types.ModelType.Bpe => new SentencePieceBpeModel(modelProto, addBos, addEos, specialTokens),
                TrainerSpec.Types.ModelType.Unigram => new SentencePieceUnigramModel(modelProto, addBos, addEos, specialTokens),
                _ => throw new ArgumentException($"The model type '{modelProto.TrainerSpec.ModelType}' is not supported.", nameof(modelProto))
            };
        }

        private SentencePieceTokenizer(SentencePieceBaseModel model)
        {
            _model = model;
        }

        /// <summary>
        /// The special tokens.
        /// </summary>
        public IReadOnlyDictionary<string, int>? SpecialTokens => _model.SpecialTokens;

        /// <summary>
        /// Specifies whether the model will do a byte fallback when it encounters unknown tokens during the encoding process.
        /// </summary>
        public bool ByteFallback => _model.ByteFallback;

View on GitHub (pinned to 7b76e69cf9)