{"record":{"id":"00b02cdcfc2d1732","repo":"dotnet/machinelearning","slug":"the-merge-entries-cannot-be-null","errorCode":null,"errorMessage":"The merge entries cannot be null.","messagePattern":"The merge entries cannot be null\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":173,"sourceCode":"            {\n                vocab.Add(new StringSpanOrdinalKey(kvp.Key), kvp.Value);\n            }\n\n            if (vocab.Count == 0)\n            {\n                throw new InvalidOperationException(\"The vocabulary cannot be empty.\");\n            }\n\n            Vec<(string, string)> merges = default;\n            if (options.Merges is not null)\n            {\n                merges = new Vec<(string, string)>(1000);\n\n                foreach (string merge in options.Merges)\n                {\n                    if (merge is null)\n                    {\n                        throw new InvalidOperationException(\"The merge entries cannot be null.\");\n                    }\n\n                    int index = merge.IndexOf(' ');\n                    if (index < 0 || index == merge.Length - 1 || merge.IndexOf(' ', index + 1) >= 0)\n                    {\n                        throw new InvalidOperationException($\"Invalid merger file format\");\n                    }\n\n                    merges.Push((merge.Substring(0, index), merge.Substring(index + 1)));\n                }\n            }\n\n            return new BpeTokenizer(\n                            vocab, merges,\n                            options.PreTokenizer,\n                            options.Normalizer,\n                            options.SpecialTokens,\n                            options.UnknownToken,","sourceCodeStart":155,"sourceCodeEnd":191,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L155-L191","documentation":"When BpeOptions.Merges is supplied, each merge entry must be a non-null string; a null entry throws InvalidOperationException('The merge entries cannot be null.'). Merges define the BPE pair-merge table and must all be valid.","triggerScenarios":"Passing a List<string> (or string[]) to BpeOptions.Merges that contains at least one null element — commonly from array initialization with unfilled slots, deserialization of sparse lists, or Select() producing nulls.","commonSituations":"Reading merge lines from a file that has trailing/blank lines mapped to null; constructing merges = new string[n] without filling all slots; LINQ projection returning null for malformed rows.","solutions":["Filter out null (and empty) entries before Create: merges.Where(m => m != null).","Fix the file/loader so blank or malformed lines are skipped rather than converted to null.","Validate with Array.FindIndex(merges, m => m == null) to find the offending index before calling Create.","If deserializing JSON, use a deserializer setting that ignores null items or sanitize afterward."],"exampleFix":"// before\noptions.Merges = rawLines; // rawLines contains nulls from blank lines\n// after\noptions.Merges = rawLines.Where(m => !string.IsNullOrEmpty(m)).ToArray();","handlingStrategy":"validation","validationCode":"if (merges?.Any(m => m is null) == true)\n    throw new InvalidOperationException(\"Merges contains null entries.\");","typeGuard":"static bool AllMergesNonNull(string[]? m) => m is not null && Array.TrueForAll(m, x => x is not null);","tryCatchPattern":"try { var t = BpeTokenizer.Create(options); }\ncatch (InvalidOperationException ex) when (ex.Message.Contains(\"merge entries cannot be null\"))\n{ options.Merges = options.Merges.Where(x => x != null).ToArray(); var t = BpeTokenizer.Create(options); }","preventionTips":["Sanitize merges with Where(m => m != null) after any file/JSON load.","Avoid pre-sized arrays (new string[n]) that can leave null slots.","Reject blank lines at parse time instead of converting them to null."],"tags":["dotnet","tokenizers","bpe","merges"],"backgroundTag":"invalid-argument-value","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}