{"record":{"id":"21b9b86489bf856c","repo":"dotnet/machinelearning","slug":"invalid-format-of-merge-file-at-line-line","errorCode":null,"errorMessage":"Invalid format of merge file at line: \"{line}\"","messagePattern":"Invalid format of merge file at line: \"(.+?)\"","errorType":"exception","errorClass":"FormatException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs","lineNumber":1787,"sourceCode":"            var mergeRanks = new Dictionary<StringSpanOrdinalKeyPair, int>();\n            try\n            {\n                using StreamReader reader = new StreamReader(mergeStream);\n\n                // We ignore the first and last line in the file\n                if (reader.Peek() >= 0)\n                {\n                    string ignored = reader.ReadLine()!;\n                }\n\n                int rank = 1;\n                while (reader.Peek() >= 0)\n                {\n                    string line = reader.ReadLine()!;\n                    int index = line.IndexOf(' ');\n                    if (index < 1 || index == line.Length - 1 || line.IndexOf(' ', index + 1) != -1)\n                    {\n                        throw new FormatException($\"Invalid format of merge file at line: \\\"{line}\\\"\");\n                    }\n\n                    mergeRanks.Add(new StringSpanOrdinalKeyPair(line.Substring(0, index), line.Substring(index + 1)), rank++);\n                }\n            }\n            catch (Exception e)\n            {\n                // Report any issues encountered while consuming a data file as IOExceptions.\n                throw new IOException($\"Cannot read the file Merge file.{Environment.NewLine}Error message: {e.Message}\", e);\n            }\n\n            return mergeRanks;\n        }\n\n        private struct SymbolPair : IEquatable<SymbolPair>, IComparable<SymbolPair>\n        {\n            public int Left { get; set; }\n            public int Right { get; set; }","sourceCodeStart":1769,"sourceCodeEnd":1805,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs#L1769-L1805","documentation":"GetMergeRanks parses the BPE merges file line by line, requiring each line to contain exactly one space separating a non-empty pair of parts. Any line violating this (no space, space at start/end, or multiple spaces) throws FormatException identifying the offending line.","triggerScenarios":"Supplying a merges file whose lines are not of the form 'tokenA tokenB' — e.g. the vocab JSON passed as merges, an empty first header line retained, blank lines, CRLF remnants, or a merges file from an incompatible tokenizer version.","commonSituations":"Swapped vocab/merges files, merges.txt with the '#version' header handling differences, files downloaded with wrong line endings, or hand-edited merges files.","solutions":["Verify you passed merges.txt (line-based merge pairs), not vocab.json, as the merge stream.","Open merges.txt and check each line has exactly two non-empty parts separated by a single space; fix or remove malformed lines.","Re-download the merges file for the exact model checkpoint (matching the vocabulary).","Preprocess the stream to strip blank lines/BOM before passing it to the tokenizer factory."],"exampleFix":"// before\nvar tok = CodeGenTokenizer.Create(vocabStream, vocabStream); // vocab parsed as merges -> 'Invalid format of merge file'\n// after\nvar tok = CodeGenTokenizer.Create(vocabStream, mergesStream);","handlingStrategy":"validation","validationCode":"foreach (var line in File.ReadLines(mergesPath))\n{\n    int i = line.IndexOf(' ');\n    if (i < 1 || i == line.Length - 1 || line.IndexOf(' ', i + 1) != -1)\n        throw new InvalidDataException($\"Bad merge line: {line}\");\n}","typeGuard":"bool IsValidMergeLine(string line) { int i = line.IndexOf(' '); return i > 0 && i < line.Length - 1 && line.IndexOf(' ', i + 1) == -1; }","tryCatchPattern":"try { var tok = CodeGenTokenizer.Create(vocabStream, mergesStream); }\ncatch (FormatException ex) when (ex.Message.StartsWith(\"Invalid format of merge file\")) { /* repair or re-download merges.txt */ }","preventionTips":["Never pass vocab.json as the merges stream (and vice versa)","Strip BOM/blank lines and normalize line endings before loading","Keep vocab.json and merges.txt from the same model release together"],"tags":["tokenizer","bpe","file-format","parsing","dotnet"],"backgroundTag":"invalid-argument-format","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}