dotnet/machinelearning · error · ArgumentNullException

ArgumentNullException

Error message

ArgumentNullException

What it means

Create(stream) parses the model protobuf and then checks whether the result is null, throwing ArgumentNullException for modelProto. In practice ModelProto.Parser.ParseFrom throws on bad input before returning, so this guard mainly documents the contract: a null/unusable model cannot produce a tokenizer.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs:461

        /// </summary>
        /// <param name="modelStream">The stream containing the SentencePiece Bpe or Unigram model.</param>
        /// <param name="addBeginningOfSentence">Indicate emitting the beginning of sentence token during the encoding.</param>
        /// <param name="addEndOfSentence">Indicate emitting the end of sentence token during the encoding.</param>
        /// <param name="specialTokens">The additional tokens to add to the vocabulary.</param>
        /// <remarks>
        /// When creating the tokenizer, ensure that the vocabulary stream is sourced from a trusted provider.
        /// </remarks>
        public static SentencePieceTokenizer Create(
            Stream modelStream,
            bool addBeginningOfSentence = true,
            bool addEndOfSentence = false,
            IReadOnlyDictionary<string, int>? specialTokens = null)
        {
            ModelProto modelProto = ModelProto.Parser.ParseFrom(modelStream);

            if (modelProto is null)
            {
                throw new ArgumentNullException(nameof(modelProto));
            }

            return new SentencePieceTokenizer(modelProto, addBeginningOfSentence, addEndOfSentence, specialTokens);
        }

        /// <summary>
        /// Creates a Unigram <see cref="SentencePieceTokenizer"/> from an in-memory vocabulary of (piece, score) pairs.
        /// </summary>
        /// <param name="vocab">
        /// The vocabulary as an ordered sequence of (piece, score) pairs. The position of each pair
        /// in the sequence determines its token ID.
        /// </param>
        /// <param name="unkId">The index (token ID) of the unknown token in <paramref name="vocab"/>.</param>
        /// <param name="addBeginningOfSentence">Whether to emit the beginning-of-sentence token during encoding.</param>
        /// <param name="addEndOfSentence">Whether to emit the end-of-sentence token during encoding.</param>
        /// <param name="precompiledCharsMap">
        /// Optional precompiled character normalization map (as found in the SentencePiece <c>normalizer_spec.precompiled_charsmap</c>
        /// field or in the Hugging Face <c>tokenizer.json</c> <c>normalizer.precompiled_charsmap</c> property).

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Verify the stream is a valid sentencepiece .model protobuf and fully written before calling Create.
2. Check stream length > 0 and that the stream was read from the correct file/resource.
3. Guard with stream null/length checks in your code; catch ArgumentException/ArgumentNullException around Create for bad files.
4. Regenerate or re-download the model file.

Example fix

// before
using var fs = File.OpenRead(path);
var tok = SentencePieceTokenizer.Create(fs);
// after
using var fs = File.OpenRead(path);
if (fs.Length == 0) throw new InvalidOperationException($"Model file '{path}' is empty.");
var tok = SentencePieceTokenizer.Create(fs);

if (modelStream is null || !modelStream.CanRead || modelStream.Length == 0)
    throw new IOException("SentencePiece model stream is empty or unreadable.");

static bool IsUsableStream(Stream? s) => s is { CanRead: true } && (s.Length == 0 || s.Position < s.Length || s.CanSeek == false);

try { var tok = SentencePieceTokenizer.Create(stream); } catch (ArgumentNullException ex) { throw new InvalidOperationException("Model file is missing or empty.", ex); }

When it happens

Trigger: Calling SentencePieceTokenizer.Create with a stream that yields no usable ModelProto — e.g. an empty, truncated, or wrong-format stream — or a code path where the parsed object is null.

Common situations: Passing an empty MemoryStream, a stream pointing at a text/JSON file instead of a sentencepiece .model protobuf, or a partially downloaded model file.

Related errors

• The BOS, EOS, or UNK token is not present in the vocabulary.
• The tokenizer.json model enables byte_fallback but does not
• vocab
• The vocabulary does not contain the required special token.
• Value cannot be null. (Parameter 'idColumns')

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/80389529f17d12be. Report an issue: GitHub.