{"record":{"id":"54c1d44a30336137","repo":"dotnet/machinelearning","slug":"the-model-type-is-not-bpe","errorCode":null,"errorMessage":"The model type is not Bpe.","messagePattern":"The model type is not Bpe\\.","errorType":"exception","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/LlamaTokenizer.cs","lineNumber":49,"sourceCode":"        /// <remarks>\n        /// When creating the tokenizer, ensure that the vocabulary stream is sourced from a trusted provider.\n        /// </remarks>\n        public static new LlamaTokenizer Create(\n            Stream modelStream,\n            bool addBeginOfSentence = true,\n            bool addEndOfSentence = false,\n            IReadOnlyDictionary<string, int>? specialTokens = null)\n        {\n            ModelProto modelProto = ModelProto.Parser.ParseFrom(modelStream);\n\n            if (modelProto is null)\n            {\n                throw new ArgumentNullException(nameof(modelProto));\n            }\n\n            if (modelProto.TrainerSpec.ModelType != TrainerSpec.Types.ModelType.Bpe)\n            {\n                throw new ArgumentException(\"The model type is not Bpe.\", nameof(modelProto));\n            }\n\n            if (modelProto.NormalizerSpec.Name != \"identity\" && !string.IsNullOrEmpty(modelProto.NormalizerSpec.Name))\n            {\n                throw new ArgumentException($\"Normalization '{modelProto.NormalizerSpec.Name}' is not supported.\", nameof(modelProto));\n            }\n\n            return new LlamaTokenizer(modelProto, addBeginOfSentence, addEndOfSentence, specialTokens);\n        }\n    }\n}\n","sourceCodeStart":31,"sourceCodeEnd":61,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/LlamaTokenizer.cs#L31-L61","documentation":"LlamaTokenizer.Create requires a SentencePiece ModelProto whose TrainerSpec.ModelType is Bpe. Passing a model proto trained with a different SentencePiece algorithm (Unigram, Word, Char) throws this ArgumentException.","triggerScenarios":"Calling LlamaTokenizer.Create(modelProto, ...) with a ModelProto loaded from a .model file that was trained as Unigram (the common SentencePiece default) or Word/Char model.","commonSituations":"Downloading a generic SentencePiece .model from HuggingFace that is Unigram-based and passing it to LlamaTokenizer; mixing up LLaMA BPE models with T5-style Unigram models.","solutions":["Train or download a SentencePiece model with model_type=bpe and pass that proto","Inspect proto.TrainerSpec.ModelType before calling Create and route to the matching tokenizer class","Regenerate the .model file using sentencepiece with --model_type=bpe"],"exampleFix":"// before\nvar proto = ModelProto.Parser.ParseFrom(File.ReadAllBytes(\"unigram.model\")); // Unigram\nvar tok = LlamaTokenizer.Create(proto);\n// after\n// train with: spm_train --model_type=bpe ...\nvar proto = ModelProto.Parser.ParseFrom(File.ReadAllBytes(\"bpe.model\"));\nvar tok = LlamaTokenizer.Create(proto);","handlingStrategy":"validation","validationCode":"if (proto.TrainerSpec.ModelType != TrainerSpec.Types.ModelType.Bpe) throw new InvalidOperationException($\"Expected Bpe model, got {proto.TrainerSpec.ModelType}\");","typeGuard":null,"tryCatchPattern":"try { var tok = LlamaTokenizer.Create(proto); } catch (ArgumentException ex) { log.LogError(ex, \"SentencePiece model is not Bpe-trained\"); throw; }","preventionTips":["Check TrainerSpec.ModelType before constructing","Only use models trained with --model_type=bpe with LlamaTokenizer","Document the model provenance of your .model files"],"tags":["tokenizers","sentencepiece","model-type"],"backgroundTag":"invalid-argument-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}