dotnet/machinelearning · error · InvalidOperationException

The input tokenizer is not using the EnglishRoberta model.

Error message

The input tokenizer is not using the EnglishRoberta model.

What it means

This extension method in TokenizerExtensions downcasts a generic Tokenizer to EnglishRobertaTokenizer via `as` and throws InvalidOperationException if the instance is not actually an EnglishRobertaTokenizer. Roberta-specific helpers (e.g., vocab mapping, EncodeToConverted) only make sense for the Roberta tokenizer, so the library refuses other tokenizer types (e.g., BERT WordPiece) rather than silently misbehaving. It is a type-mismatch guard at the point where Roberta-only behavior is required.

Source

Thrown at src/Microsoft.ML.TorchSharp/Extensions/TokenizerExtensions.cs:46

                _instance = EnglishRobertaTokenizer.Create(
                                            assembly.GetManifestResourceStream("encoder.json"),
                                            assembly.GetManifestResourceStream("vocab.bpe"),
                                            assembly.GetManifestResourceStream("dict.txt"),
                                            RobertaPreTokenizer.Instance);
                (_instance as EnglishRobertaTokenizer).AddMaskSymbol();
            }

            return _instance;
        }

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        internal static EnglishRobertaTokenizer RobertaModel(this Tokenizer tokenizer)
        {
            EnglishRobertaTokenizer model = tokenizer as EnglishRobertaTokenizer;
            if (model is null)
            {
                throw new InvalidOperationException($"The input tokenizer is not using the EnglishRoberta model.");
            }

            return model;
        }

        internal static IReadOnlyList<int> EncodeToConverted(this Tokenizer tokenizer, string sentence)
        {
            return tokenizer.RobertaModel().ConvertIdsToOccurrenceRanks(tokenizer.EncodeToIds(sentence));
        }
    }
}

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Ensure the tokenizer is created as EnglishRobertaTokenizer (e.g., via EnglishRobertaTokenizer.Create with the model's vocab hash/paths) before calling Roberta-specific APIs.
2. Check the tokenizer type at runtime with `if (tokenizer is EnglishRobertaTokenizer roberta)` before invoking the extension.
3. Verify you are pointing at the RoBERTa vocabulary files (vocab.json/merges.txt) and not a BERT vocab.
4. Wrap the call in try-catch on InvalidOperationException to surface a clearer message to users when the wrong tokenizer type is supplied.

Example fix

// before
var tokenizer = Tokenizer.Create(vocabPath); // may not be Roberta
var roberta = tokenizer.RobertaModel(); // throws

// after
if (tokenizer is not EnglishRobertaTokenizer)
    tokenizer = EnglishRobertaTokenizer.Create(vocabPath, mergesPath);
var roberta = tokenizer.RobertaModel(); // OK

if (tokenizer is not EnglishRobertaTokenizer)
    throw new ArgumentException("A tokenizer backed by EnglishRobertaTokenizer is required.", nameof(tokenizer));

bool IsRobertaTokenizer(Tokenizer t) => t is EnglishRobertaTokenizer;

try { var roberta = tokenizer.RobertaModel(); ... }
catch (InvalidOperationException ex) { log.LogError(ex, "Tokenizer must be EnglishRobertaTokenizer"); throw new InvalidModelException(...); }

When it happens

Trigger: Calling the internal RobertaModel() extension (or methods that route through it, such as Roberta token encoding used by TorchSharp text pipelines like TextClassification/QuestionAnswering trainers) with a tokenizer instance created from a non-Roberta model, e.g. `tokenizer as` result being null because a WordPieceTokenizer or another Tokenizer subclass was loaded.

Common situations: Loading a tokenizer file saved for a different model family (BERT vs RoBERTa), using a tokenizer produced by a different trainer entry point, or passing a null/other tokenizer into a pipeline component that requires EnglishRobertaTokenizer.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• getter must be of type '{typeof(ValueGetter<TValue>).FullNam
• Not implemented type {typeof(T)}
• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabFilePath));
• throw new ArgumentNullException(nameof(vocabStream));

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/8ef71ac8934c26ab. Report an issue: GitHub.