{"record":{"id":"80389529f17d12be","repo":"dotnet/machinelearning","slug":"argumentnullexception-803895","errorCode":null,"errorMessage":"ArgumentNullException","messagePattern":"ArgumentNullException","errorType":"exception","errorClass":"ArgumentNullException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs","lineNumber":461,"sourceCode":"        /// </summary>\n        /// <param name=\"modelStream\">The stream containing the SentencePiece Bpe or Unigram model.</param>\n        /// <param name=\"addBeginningOfSentence\">Indicate emitting the beginning of sentence token during the encoding.</param>\n        /// <param name=\"addEndOfSentence\">Indicate emitting the end of sentence token during the encoding.</param>\n        /// <param name=\"specialTokens\">The additional tokens to add to the vocabulary.</param>\n        /// <remarks>\n        /// When creating the tokenizer, ensure that the vocabulary stream is sourced from a trusted provider.\n        /// </remarks>\n        public static SentencePieceTokenizer Create(\n            Stream modelStream,\n            bool addBeginningOfSentence = true,\n            bool addEndOfSentence = false,\n            IReadOnlyDictionary<string, int>? specialTokens = null)\n        {\n            ModelProto modelProto = ModelProto.Parser.ParseFrom(modelStream);\n\n            if (modelProto is null)\n            {\n                throw new ArgumentNullException(nameof(modelProto));\n            }\n\n            return new SentencePieceTokenizer(modelProto, addBeginningOfSentence, addEndOfSentence, specialTokens);\n        }\n\n        /// <summary>\n        /// Creates a Unigram <see cref=\"SentencePieceTokenizer\"/> from an in-memory vocabulary of (piece, score) pairs.\n        /// </summary>\n        /// <param name=\"vocab\">\n        /// The vocabulary as an ordered sequence of (piece, score) pairs. The position of each pair\n        /// in the sequence determines its token ID.\n        /// </param>\n        /// <param name=\"unkId\">The index (token ID) of the unknown token in <paramref name=\"vocab\"/>.</param>\n        /// <param name=\"addBeginningOfSentence\">Whether to emit the beginning-of-sentence token during encoding.</param>\n        /// <param name=\"addEndOfSentence\">Whether to emit the end-of-sentence token during encoding.</param>\n        /// <param name=\"precompiledCharsMap\">\n        /// Optional precompiled character normalization map (as found in the SentencePiece <c>normalizer_spec.precompiled_charsmap</c>\n        /// field or in the Hugging Face <c>tokenizer.json</c> <c>normalizer.precompiled_charsmap</c> property).","sourceCodeStart":443,"sourceCodeEnd":479,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.cs#L443-L479","documentation":"Create(stream) parses the model protobuf and then checks whether the result is null, throwing ArgumentNullException for modelProto. In practice ModelProto.Parser.ParseFrom throws on bad input before returning, so this guard mainly documents the contract: a null/unusable model cannot produce a tokenizer.","triggerScenarios":"Calling SentencePieceTokenizer.Create with a stream that yields no usable ModelProto — e.g. an empty, truncated, or wrong-format stream — or a code path where the parsed object is null.","commonSituations":"Passing an empty MemoryStream, a stream pointing at a text/JSON file instead of a sentencepiece .model protobuf, or a partially downloaded model file.","solutions":["Verify the stream is a valid sentencepiece .model protobuf and fully written before calling Create.","Check stream length > 0 and that the stream was read from the correct file/resource.","Guard with stream null/length checks in your code; catch ArgumentException/ArgumentNullException around Create for bad files.","Regenerate or re-download the model file."],"exampleFix":"// before\nusing var fs = File.OpenRead(path);\nvar tok = SentencePieceTokenizer.Create(fs);\n// after\nusing var fs = File.OpenRead(path);\nif (fs.Length == 0) throw new InvalidOperationException($\"Model file '{path}' is empty.\");\nvar tok = SentencePieceTokenizer.Create(fs);","handlingStrategy":"validation","validationCode":"if (modelStream is null || !modelStream.CanRead || modelStream.Length == 0)\n    throw new IOException(\"SentencePiece model stream is empty or unreadable.\");","typeGuard":"static bool IsUsableStream(Stream? s) => s is { CanRead: true } && (s.Length == 0 || s.Position < s.Length || s.CanSeek == false);","tryCatchPattern":"try { var tok = SentencePieceTokenizer.Create(stream); } catch (ArgumentNullException ex) { throw new InvalidOperationException(\"Model file is missing or empty.\", ex); }","preventionTips":["Check file existence and length before opening","Catch protobuf parse exceptions around Create to give better errors","Verify downloaded model checksums"],"tags":["null-argument","sentencepiece","model-loading"],"backgroundTag":"null-argument","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}