dotnet/machinelearning · error · ArgumentNullException

ArgumentNullException

Error message

ArgumentNullException

What it means

The SentencePieceBaseModel constructor requires a non-null SentencePiece ModelProto; this guard throws ArgumentNullException when modelProto is null. It is the base initializer for Llama/Phi SentencePiece tokenizers.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/SentencePieceBaseModel.cs:22

using Sentencepiece;
using System;
using System.Buffers;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;

namespace Microsoft.ML.Tokenizers
{
    internal abstract class SentencePieceBaseModel
    {
        internal SentencePieceBaseModel(ModelProto modelProto, bool addBos = false, bool addEos = false, IReadOnlyDictionary<string, int>? specialTokens = null)
        {
            if (modelProto is null)
            {
                throw new ArgumentNullException(nameof(modelProto));
            }

            AddBeginningOfSentence = addBos;
            AddEndOfSentence = addEos;
            BeginningOfSentenceToken = modelProto.TrainerSpec.BosPiece ?? "<s>";
            BeginningOfSentenceId = Math.Max(0, modelProto.TrainerSpec.BosId);
            EndOfSentenceToken = modelProto.TrainerSpec.EosPiece ?? "</s>";
            EndOfSentenceId = Math.Max(0, modelProto.TrainerSpec.EosId);
            UnknownToken = modelProto.TrainerSpec.UnkPiece ?? "<unk>";
            UnknownId = Math.Max(0, modelProto.TrainerSpec.UnkId);
            AddDummyPrefix = modelProto.NormalizerSpec.AddDummyPrefix;
            EscapeWhiteSpaces = modelProto.NormalizerSpec.EscapeWhitespaces;
            TreatWhitespaceAsSuffix = modelProto.TrainerSpec.TreatWhitespaceAsSuffix;
            ByteFallback = modelProto.TrainerSpec.ByteFallback;
            SpecialTokens = specialTokens;

            if (specialTokens is not null && specialTokens.Count > 0)
            {

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Ensure the .model file was read and parsed successfully before constructing the tokenizer
2. Null-check the parsed ModelProto before passing it in
3. Throw or log a clear error at load time instead of propagating null

Example fix

// before
ModelProto? proto = TryLoadModel(path); // may be null
var tok = LlamaTokenizer.Create(proto);
// after
var proto = ModelProto.Parser.ParseFrom(File.ReadAllBytes(path));
var tok = LlamaTokenizer.Create(proto);

if (modelProto is null) throw new InvalidOperationException("ModelProto must be parsed before constructing the tokenizer.");

try { var tok = LlamaTokenizer.Create(modelProto); } catch (ArgumentNullException ex) { log.LogError(ex, "modelProto was null"); throw; }

When it happens

Trigger: Constructing a derived SentencePiece model (e.g. via LlamaTokenizer.Create) with a ModelProto variable that is null because parsing the .model file failed or returned null.

Common situations: ModelProto.Parser.ParseFrom silently handling empty byte arrays downstream; a nullable proto passed along after a failed load; refactored caching returning null.

Related errors

• The model type is not Bpe.
• Normalization '{modelProto.NormalizerSpec.Name}' is not supp
• ArgumentNullException
• The pre_tokenizer 'add_prefix_space' must be a boolean.
• The pre_tokenizer 'replacement' must be a string.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/9397fb8149f41706. Report an issue: GitHub.