{"record":{"id":"960f94f8e70027bb","repo":"dotnet/machinelearning","slug":"the-model-type-modelproto-trainerspec-modeltype","errorCode":null,"errorMessage":"The model type '{modelProto.TrainerSpec.ModelType}' is not supported.","messagePattern":"The model type '(.+?)' is not supported\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":30,"sourceCode":"namespace Microsoft.ML.Tokenizers\n{\n    // SentencePieceBpe is implementing the BPE algorithm based on the SentencePiece https://github.com/google/sentencepiece.\n    // SentencePiece is under the Apache License 2.0 https://github.com/google/sentencepiece/blob/master/LICENSE\n\n    /// <summary>\n    /// SentencePieceBpe is a tokenizer that splits the input into tokens using the SentencePiece Bpe model.\n    /// </summary>\n    public class SentencePieceTokenizer : Tokenizer\n    {\n        private readonly SentencePieceBaseModel _model;\n\n        internal SentencePieceTokenizer(ModelProto modelProto, bool addBos, bool addEos, IReadOnlyDictionary<string, int>? specialTokens = null)\n        {\n            _model = modelProto.TrainerSpec.ModelType switch\n            {\n                TrainerSpec.Types.ModelType.Bpe => new SentencePieceBpeModel(modelProto, addBos, addEos, specialTokens),\n                TrainerSpec.Types.ModelType.Unigram => new SentencePieceUnigramModel(modelProto, addBos, addEos, specialTokens),\n                _ => throw new ArgumentException($\"The model type '{modelProto.TrainerSpec.ModelType}' is not supported.\", nameof(modelProto))\n            };\n        }\n\n        private SentencePieceTokenizer(SentencePieceBaseModel model)\n        {\n            _model = model;\n        }\n\n        /// <summary>\n        /// The special tokens.\n        /// </summary>\n        public IReadOnlyDictionary<string, int>? SpecialTokens => _model.SpecialTokens;\n\n        /// <summary>\n        /// Specifies whether the model will do a byte fallback when it encounters unknown tokens during the encoding process.\n        /// </summary>\n        public bool ByteFallback => _model.ByteFallback;\n","sourceCodeStart":12,"sourceCodeEnd":48,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L12-L48","documentation":"The SentencePieceTokenizer constructor dispatches on ModelProto.TrainerSpec.ModelType and only supports Bpe and Unigram. Any other model type (e.g. WordPiece-derived or unspecified SentencePiece exports) falls to the default arm and throws ArgumentException naming modelProto.","triggerScenarios":"Loading a .model sentencepiece file whose trainer_spec.model_type is neither BPE nor UNIGRAM (e.g. WORD model, corrupt/unset trainer spec) and passing it to SentencePieceTokenizer.Create / the internal constructor.","commonSituations":"Exporting a SentencePiece model trained with --model_type=word or bpe-less tools; models produced by non-SentencePiece tools saved with the .sentencepiece extension; protobuf files whose trainer_spec was stripped.","solutions":["Retrain/re-export the model with --model_type=bpe or unigram.","Convert the model to a supported type, or load it via a different tokenizer (e.g. Hugging Face tokenizer.json with CreateFromTokenizerJson).","Inspect modelProto.TrainerSpec.ModelType with the protobuf API before loading to confirm support.","Check the file is genuinely a SentencePiece protobuf, not another format."],"exampleFix":"# before: sentencepiece_trainer --input=corpus.txt --model_type=word --model_prefix=m\n# after\nsentencepiece_trainer --input=corpus.txt --model_type=unigram --model_prefix=m","handlingStrategy":"validation","validationCode":"var proto = ModelProto.Parser.ParseFrom(modelStream);\nif (proto.TrainerSpec.ModelType != TrainerSpec.Types.ModelType.Bpe && proto.TrainerSpec.ModelType != TrainerSpec.Types.ModelType.Unigram)\n    throw new NotSupportedException($\"Unsupported SentencePiece model type: {proto.TrainerSpec.ModelType}\");","typeGuard":null,"tryCatchPattern":"try { var tok = SentencePieceTokenizer.Create(stream); } catch (ArgumentException ex) { // fall back to another loader or retrain path\n    logger.LogError(ex, \"Unsupported SentencePiece model type\"); }","preventionTips":["Train SentencePiece models only with --model_type=bpe or unigram","Pre-validate model type with the protobuf API before constructing","Keep a record of how each .model file was trained"],"tags":["unsupported-model-type","sentencepiece","tokenization"],"backgroundTag":"unsupported-enum-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}