{"record":{"id":"7a6efd5036967a7b","repo":"dotnet/machinelearning","slug":"invalid-merger-file-format-at-line-linenumber","errorCode":null,"errorMessage":"Invalid merger file format at line: {lineNumber}","messagePattern":"Invalid merger file format at line: (.+?)","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":1167,"sourceCode":"                string? line = useAsync ?\n                    await Helpers.ReadLineAsync(reader, cancellationToken).ConfigureAwait(false) :\n                    reader.ReadLine();\n\n                if (line is null)\n                {\n                    break;\n                }\n\n                lineNumber++;\n                if (line.StartsWith(\"#version\", StringComparison.Ordinal) || line.Length == 0)\n                {\n                    continue;\n                }\n\n                int index = line.IndexOf(' ');\n                if (index < 0 || index == line.Length - 1 || line.IndexOf(' ', index + 1) >= 0)\n                {\n                    throw new InvalidOperationException($\"Invalid merger file format at line: {lineNumber}\");\n                }\n                merges.Push((line.Substring(0, index), line.Substring(index + 1)));\n            }\n\n            return merges;\n        }\n\n        private readonly Dictionary<char, string> _charToString = new Dictionary<char, string>();\n\n        [MethodImpl(MethodImplOptions.AggressiveInlining)]\n        internal string CharToString(char c)\n        {\n            if (_charToString.TryGetValue(c, out string? v))\n            {\n                return v;\n            }\n\n            string s = c.ToString();","sourceCodeStart":1149,"sourceCodeEnd":1185,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L1149-L1185","documentation":"When loading a BPE merges file, each non-header line must contain exactly one space separating the two parts of the merge pair, with non-empty content on both sides. A line with no space, a trailing-only space, or more than one space is a malformed merges file and throws InvalidOperationException with the offending line number.","triggerScenarios":"Loading a merges file where a line lacks the required single-space separator: a JSON merges array exported as text, a CRLF/encoding-corrupted file, or a merges file from a different tokenizer format.","commonSituations":"Pointing BPETokenizer at a vocab.json with the wrong merges.txt, hand-editing merges.txt, or using merges files saved by tools that write double-space entries.","solutions":["Verify the merges file is the plain-text HuggingFace merges.txt format: exactly two space-separated tokens per line.","Re-export or re-download the merges file for the specific model.","Check file encoding/line endings (re-save as UTF-8, LF) if the file was transferred across platforms."],"exampleFix":"// before\nvar tokenizer = BpeTokenizer.Create(vocabPath, mergesPath); // merges.txt is actually a JSON array\n// after\n// ensure merges.txt lines look like: \"t h\" per line\nvar tokenizer = BpeTokenizer.Create(vocabPath, correctMergesPath);","handlingStrategy":"validation","validationCode":"foreach (var (line, i) in File.ReadLines(mergesPath).Select((l, i) => (l, i)))\n{\n    int sp = line.IndexOf(' ');\n    if (sp < 0 || sp == line.Length - 1 || line.IndexOf(' ', sp + 1) >= 0)\n        throw new InvalidDataException($\"Bad merges line {i + 1}: '{line}'\");\n}","typeGuard":null,"tryCatchPattern":"try { var tok = BpeTokenizer.Create(vocabPath, mergesPath); }\ncatch (InvalidOperationException ex) when (ex.Message.StartsWith(\"Invalid merger file format\")) { /* re-export or fix merges file */ }","preventionTips":["Always use the merges.txt shipped with the model, not hand-edited copies.","Verify file encoding (UTF-8) and line endings after transfers.","Spot-check the first merges lines: each must contain exactly one space."],"tags":["csharp","tokenizers","file-format"],"backgroundTag":"invalid-argument-format","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}