{"record":{"id":"4bd44011edd8e945","repo":"dotnet/machinelearning","slug":"invalid-format-of-merge-file-at-line-line-4bd440","errorCode":null,"errorMessage":"Invalid format of merge file at line: \"{line}\"","messagePattern":"Invalid format of merge file at line: \"(.+?)\"","errorType":"exception","errorClass":"FormatException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs","lineNumber":219,"sourceCode":"            var mergeRanks = new Cache<(string, string), int>(60_000);\n            try\n            {\n                using StreamReader reader = new StreamReader(mergeStream);\n\n                // We ignore the first and last line in the file\n                if (reader.Peek() >= 0)\n                {\n                    string ignored = reader.ReadLine()!;\n                }\n\n                int rank = 1;\n                while (reader.Peek() >= 0)\n                {\n                    string line = reader.ReadLine()!;\n                    int index = line.IndexOf(' ');\n                    if (index < 1 || index == line.Length - 1 || line.IndexOf(' ', index + 1) != -1)\n                    {\n                        throw new FormatException($\"Invalid format of merge file at line: \\\"{line}\\\"\");\n                    }\n\n                    mergeRanks.Set((line.Substring(0, index), line.Substring(index + 1)), rank++);\n                }\n            }\n            catch (Exception e)\n            {\n                // Report any issues encountered while consuming a data file as IOExceptions.\n                throw new IOException($\"Cannot read the file Merge file.{Environment.NewLine}Error message: {e.Message}\", e);\n            }\n\n            return mergeRanks;\n        }\n\n        private Dictionary<string, int> GetVocab()\n        {\n            Dictionary<string, int>? publicVocab = Volatile.Read(ref _vocabOriginal);\n            if (publicVocab is null)","sourceCodeStart":201,"sourceCodeEnd":237,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs#L201-L237","documentation":"Thrown when a line of the BPE merge file does not have the required format 'token1 token2'. A valid line must contain exactly one space, not at the start, not at the end. The library throws FormatException for this malformed line.","triggerScenarios":"A merge file line with no space, with a leading space, with a trailing space, or with two or more spaces (three or more tokens) is read by GetMergeRanks during EnglishRobertaTokenizer construction.","commonSituations":"Hand-edited merges.txt; a file converted between encodings that altered whitespace; using a merges file from an incompatible tokenizer version with different line format; CRLF/BOM corruption from copy-paste.","solutions":["Open the merge file and inspect the reported line; fix it to contain exactly two space-separated tokens with no leading/trailing space.","Re-download the canonical merges.txt for the model instead of hand-editing.","Pre-validate the file: reject lines where IndexOf(' ') < 1, is last char, or a second space exists.","Check for encoding issues (BOM, non-breaking spaces) and re-save as UTF-8 without BOM."],"exampleFix":"// before (merges.txt line)\n\"  Ġ t\"\n// after\n\"Ġ t\"","handlingStrategy":"validation","validationCode":"foreach (var (line, i) in File.ReadLines(mergesPath).Select((l, i) => (l, i)))\n{\n    int sp = line.IndexOf(' ');\n    bool ok = sp >= 1 && sp < line.Length - 1 && line.IndexOf(' ', sp + 1) == -1;\n    if (!ok && line.Length > 0) throw new InvalidDataException($\"Bad merge line {i + 1}: '{line}'\");\n}","typeGuard":null,"tryCatchPattern":"try { var t = new EnglishRobertaTokenizer(vocabStream, mergesStream); }\ncatch (FormatException ex) { throw new InvalidDataException($\"merges.txt malformed: {ex.Message}\", ex); }","preventionTips":["Never hand-edit merges.txt; always use the canonical model file.","Save as UTF-8 without BOM; avoid editors that convert spaces/tabs.","Pre-validate merge line format when ingesting third-party model files.","Keep the reported line from the exception in diagnostics."],"tags":["format","parse","tokenizer","csharp"],"backgroundTag":"invalid-argument-format","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}