{"record":{"id":"b4d501758c0ed38e","repo":"dotnet/machinelearning","slug":"invalid-merge-file-format-at-line-linenumber","errorCode":null,"errorMessage":"Invalid merge file format at line: {lineNumber}","messagePattern":"Invalid merge file format at line: (.+?)","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs","lineNumber":86,"sourceCode":"\n                List<string> merges = new();\n\n                int lineNumber = 0;\n                string? line;\n\n                while ((line = reader.ReadLine()) is not null)\n                {\n                    lineNumber++;\n                    if (line.StartsWith(\"#version\", StringComparison.Ordinal) || line.Length == 0)\n                    {\n                        continue;\n                    }\n\n                    // validate the merges format\n                    int index = line.IndexOf(' ');\n                    if (index < 0 || index == line.Length - 1 || line.IndexOf(' ', index + 1) >= 0)\n                    {\n                        throw new InvalidOperationException($\"Invalid merge file format at line: {lineNumber}\");\n                    }\n\n                    merges.Add(line);\n                }\n\n                Merges = merges;\n            }\n        }\n\n        /// <summary>\n        /// Gets or sets the vocabulary to use.\n        /// </summary>\n        public IEnumerable<KeyValuePair<string, int>> Vocabulary { get; }\n\n        /// <summary>\n        /// Gets or sets the list of the merge strings used to merge tokens during encoding.\n        /// </summary>\n        public IEnumerable<string>? Merges { get; set; }","sourceCodeStart":68,"sourceCodeEnd":104,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs#L68-L104","documentation":"BpeOptions validates each non-comment line of the merges file must contain exactly one space separating two tokens, with a non-empty second token and no additional spaces. Any line violating this (zero spaces, trailing space, or two+ spaces) throws InvalidOperationException with the offending line number. This guards against a corrupt or wrong-format merges file.","triggerScenarios":"merges.txt contains a malformed line such as 'a  b' (double space), 'ab' (no space), 'a ' (trailing space), or CRLF/encoding artifacts; passing a non-merges text file (e.g. a plain word list) as mergesFile.","commonSituations":"Manual edits to the merges file introduced stray whitespace; file re-encoded with BOM or converted line endings; using merges from an incompatible tokenizer version.","solutions":["Open merges.txt at the reported line number and fix the line to exactly 'token1 token2' with a single space.","Re-download the merges file from the original model repository instead of a hand-edited copy.","Strip BOM and normalize line endings (LF) if the file was re-encoded.","Pre-validate all lines with a regex like ^\\S+ \\S+$ before constructing BpeOptions."],"exampleFix":"// before (merges.txt line 42: 'l  o')\nvar options = new BpeOptions(vocabPath, mergesPath);\n// after: validate lines first\nvar bad = File.ReadAllLines(mergesPath)\n    .Select((line, i) => (line, i))\n    .Where(x => !string.IsNullOrWhiteSpace(x.line) && !x.line.StartsWith(\"#\") && !System.Text.RegularExpressions.Regex.IsMatch(x.line, @\"^\\S+ \\S+$\"))\n    .ToList();\nif (bad.Count > 0) throw new InvalidDataException($\"Bad merges line {bad[0].i + 1}\");\nvar options = new BpeOptions(vocabPath, mergesPath);","handlingStrategy":"validation","validationCode":"var bad = File.ReadAllLines(mergesPath)\n    .Select((line, i) => (line, i))\n    .Where(x => !string.IsNullOrWhiteSpace(x.line) && !x.line.StartsWith(\"#\") && !System.Text.RegularExpressions.Regex.IsMatch(x.line, @\"^\\S+ \\S+$\"))\n    .ToList();\nif (bad.Count > 0) throw new InvalidDataException($\"Malformed merges line(s): {string.Join(\",\", bad.Select(b => b.i + 1))}\");","typeGuard":null,"tryCatchPattern":"try { var options = new BpeOptions(vocabPath, mergesPath); } catch (InvalidOperationException ex) { logger.LogError(ex, \"Malformed merges file {Path}\", mergesPath); }","preventionTips":["Never hand-edit merges files; re-download instead","Preserve original encoding and line endings of model assets","Pre-validate merge line format with ^\\S+ \\S+$ when ingesting third-party merges"],"tags":["csharp","file-format","tokenizer"],"backgroundTag":"invalid-argument-format","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}