{"record":{"id":"8ef71ac8934c26ab","repo":"dotnet/machinelearning","slug":"the-input-tokenizer-is-not-using-the-englishrobert","errorCode":null,"errorMessage":"The input tokenizer is not using the EnglishRoberta model.","messagePattern":"The input tokenizer is not using the EnglishRoberta model\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.TorchSharp/Extensions/TokenizerExtensions.cs","lineNumber":46,"sourceCode":"\n                _instance = EnglishRobertaTokenizer.Create(\n                                            assembly.GetManifestResourceStream(\"encoder.json\"),\n                                            assembly.GetManifestResourceStream(\"vocab.bpe\"),\n                                            assembly.GetManifestResourceStream(\"dict.txt\"),\n                                            RobertaPreTokenizer.Instance);\n                (_instance as EnglishRobertaTokenizer).AddMaskSymbol();\n            }\n\n            return _instance;\n        }\n\n        [MethodImpl(MethodImplOptions.AggressiveInlining)]\n        internal static EnglishRobertaTokenizer RobertaModel(this Tokenizer tokenizer)\n        {\n            EnglishRobertaTokenizer model = tokenizer as EnglishRobertaTokenizer;\n            if (model is null)\n            {\n                throw new InvalidOperationException($\"The input tokenizer is not using the EnglishRoberta model.\");\n            }\n\n            return model;\n        }\n\n        internal static IReadOnlyList<int> EncodeToConverted(this Tokenizer tokenizer, string sentence)\n        {\n            return tokenizer.RobertaModel().ConvertIdsToOccurrenceRanks(tokenizer.EncodeToIds(sentence));\n        }\n    }\n}","sourceCodeStart":28,"sourceCodeEnd":57,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.TorchSharp/Extensions/TokenizerExtensions.cs#L28-L57","documentation":"This extension method in TokenizerExtensions downcasts a generic Tokenizer to EnglishRobertaTokenizer via `as` and throws InvalidOperationException if the instance is not actually an EnglishRobertaTokenizer. Roberta-specific helpers (e.g., vocab mapping, EncodeToConverted) only make sense for the Roberta tokenizer, so the library refuses other tokenizer types (e.g., BERT WordPiece) rather than silently misbehaving. It is a type-mismatch guard at the point where Roberta-only behavior is required.","triggerScenarios":"Calling the internal RobertaModel() extension (or methods that route through it, such as Roberta token encoding used by TorchSharp text pipelines like TextClassification/QuestionAnswering trainers) with a tokenizer instance created from a non-Roberta model, e.g. `tokenizer as` result being null because a WordPieceTokenizer or another Tokenizer subclass was loaded.","commonSituations":"Loading a tokenizer file saved for a different model family (BERT vs RoBERTa), using a tokenizer produced by a different trainer entry point, or passing a null/other tokenizer into a pipeline component that requires EnglishRobertaTokenizer.","solutions":["Ensure the tokenizer is created as EnglishRobertaTokenizer (e.g., via EnglishRobertaTokenizer.Create with the model's vocab hash/paths) before calling Roberta-specific APIs.","Check the tokenizer type at runtime with `if (tokenizer is EnglishRobertaTokenizer roberta)` before invoking the extension.","Verify you are pointing at the RoBERTa vocabulary files (vocab.json/merges.txt) and not a BERT vocab.","Wrap the call in try-catch on InvalidOperationException to surface a clearer message to users when the wrong tokenizer type is supplied."],"exampleFix":"// before\nvar tokenizer = Tokenizer.Create(vocabPath); // may not be Roberta\nvar roberta = tokenizer.RobertaModel(); // throws\n\n// after\nif (tokenizer is not EnglishRobertaTokenizer)\n    tokenizer = EnglishRobertaTokenizer.Create(vocabPath, mergesPath);\nvar roberta = tokenizer.RobertaModel(); // OK","handlingStrategy":"type-guard","validationCode":"if (tokenizer is not EnglishRobertaTokenizer)\n    throw new ArgumentException(\"A tokenizer backed by EnglishRobertaTokenizer is required.\", nameof(tokenizer));","typeGuard":"bool IsRobertaTokenizer(Tokenizer t) => t is EnglishRobertaTokenizer;","tryCatchPattern":"try { var roberta = tokenizer.RobertaModel(); ... }\ncatch (InvalidOperationException ex) { log.LogError(ex, \"Tokenizer must be EnglishRobertaTokenizer\"); throw new InvalidModelException(...); }","preventionTips":["Create tokenizers with EnglishRobertaTokenizer.Create when targeting RoBERTa pipelines.","Never re-use a tokenizer instance across different model families.","Assert tokenizer type in pipeline setup tests."],"tags":["csharp","tokenizer","type-mismatch","invalid-operation"],"backgroundTag":"type-mismatch","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}