{"record":{"id":"8695b687f854aad6","repo":"dotnet/machinelearning","slug":"invalid-merger-file-format","errorCode":null,"errorMessage":"Invalid merger file format","messagePattern":"Invalid merger file format","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":179,"sourceCode":"                throw new InvalidOperationException(\"The vocabulary cannot be empty.\");\n            }\n\n            Vec<(string, string)> merges = default;\n            if (options.Merges is not null)\n            {\n                merges = new Vec<(string, string)>(1000);\n\n                foreach (string merge in options.Merges)\n                {\n                    if (merge is null)\n                    {\n                        throw new InvalidOperationException(\"The merge entries cannot be null.\");\n                    }\n\n                    int index = merge.IndexOf(' ');\n                    if (index < 0 || index == merge.Length - 1 || merge.IndexOf(' ', index + 1) >= 0)\n                    {\n                        throw new InvalidOperationException($\"Invalid merger file format\");\n                    }\n\n                    merges.Push((merge.Substring(0, index), merge.Substring(index + 1)));\n                }\n            }\n\n            return new BpeTokenizer(\n                            vocab, merges,\n                            options.PreTokenizer,\n                            options.Normalizer,\n                            options.SpecialTokens,\n                            options.UnknownToken,\n                            options.ContinuingSubwordPrefix,\n                            options.EndOfWordSuffix,\n                            options.FuseUnknownTokens,\n                            options.ByteLevel,\n                            options.BeginningOfSentenceToken,\n                            options.EndOfSentenceToken);","sourceCodeStart":161,"sourceCodeEnd":197,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L161-L197","documentation":"Each entry in BpeOptions.Merges must contain exactly one space separating the two parts of the merge pair ('left right'). Entries with no space, a trailing space (empty second part), or multiple spaces throw InvalidOperationException('Invalid merger file format'). This mirrors the merges.txt two-column format.","triggerScenarios":"Passing merge strings like 'ab' (no space), 'ab ' (trailing space => empty second token), or 'a b c' (two spaces) in BpeOptions.Merges — e.g. from hand-edited merges files or lines with tab separators instead of spaces.","commonSituations":"Merges file saved with tabs/CSV formatting; version mismatch where merges use a different delimiter; copy-pasted merges losing or duplicating spaces; Windows line endings leaving stray characters (may also trip this depending on parsing).","solutions":["Normalize each merge line to exactly 'token1 token2' with a single ASCII space before Create.","Fix the source merges.txt to the standard BPE two-token-per-line format.","If your delimiter is a tab, convert: line.Replace('\\t', ' ') and re-validate.","Pre-validate entries with a regex like ^\\S+ \\S+$ and reject/log bad lines before constructing the tokenizer."],"exampleFix":"// before\noptions.Merges = File.ReadAllLines(\"merges.tsv\"); // tab-separated -> invalid format\n// after\noptions.Merges = File.ReadAllLines(\"merges.tsv\")\n    .Where(l => !string.IsNullOrWhiteSpace(l))\n    .Select(l => l.Replace('\\t', ' '))\n    .Where(l => System.Text.RegularExpressions.Regex.IsMatch(l, @\"^\\S+ \\S+$\"))\n    .ToArray();","handlingStrategy":"validation","validationCode":"var bad = merges?.Where(m => m is null || !System.Text.RegularExpressions.Regex.IsMatch(m, @\"^\\S+ \\S+$\")).ToList();\nif (bad?.Count > 0) throw new FormatException($\"Invalid merge entries: {string.Join('|', bad)}\");","typeGuard":"static bool IsValidMerge(string? m) => m is not null && System.Text.RegularExpressions.Regex.IsMatch(m, @\"^\\S+ \\S+$\");","tryCatchPattern":"try { var t = BpeTokenizer.Create(options); }\ncatch (InvalidOperationException ex) when (ex.Message.Contains(\"Invalid merger file format\"))\n{ /* log offending lines, repair delimiter, retry */ }","preventionTips":["Ensure merges files use the standard 'token1 token2' space-separated format.","Pre-validate with ^\\S+ \\S+$ regex before Create and log offenders.","Convert alternate delimiters (tabs/commas) to a single space at load time."],"tags":["dotnet","tokenizers","bpe","format-validation"],"backgroundTag":"invalid-argument-format","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}