{"record":{"id":"a41a202d26b71f89","repo":"dotnet/machinelearning","slug":"cannot-read-the-file-merge-file-environment-newli","errorCode":null,"errorMessage":"Cannot read the file Merge file.{Environment.NewLine}Error message: {e.Message}","messagePattern":"Cannot read the file Merge file\\.(.+?)Error message: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs","lineNumber":1796,"sourceCode":"                }\n\n                int rank = 1;\n                while (reader.Peek() >= 0)\n                {\n                    string line = reader.ReadLine()!;\n                    int index = line.IndexOf(' ');\n                    if (index < 1 || index == line.Length - 1 || line.IndexOf(' ', index + 1) != -1)\n                    {\n                        throw new FormatException($\"Invalid format of merge file at line: \\\"{line}\\\"\");\n                    }\n\n                    mergeRanks.Add(new StringSpanOrdinalKeyPair(line.Substring(0, index), line.Substring(index + 1)), rank++);\n                }\n            }\n            catch (Exception e)\n            {\n                // Report any issues encountered while consuming a data file as IOExceptions.\n                throw new IOException($\"Cannot read the file Merge file.{Environment.NewLine}Error message: {e.Message}\", e);\n            }\n\n            return mergeRanks;\n        }\n\n        private struct SymbolPair : IEquatable<SymbolPair>, IComparable<SymbolPair>\n        {\n            public int Left { get; set; }\n            public int Right { get; set; }\n            public int Length { get; set; }\n            public int Score { get; set; }\n\n            public SymbolPair(int left, int right, int score, int length)\n            {\n                Left = left;\n                Right = right;\n                Score = score;\n                Length = length;","sourceCodeStart":1778,"sourceCodeEnd":1814,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs#L1778-L1814","documentation":"Thrown when any exception occurs while reading and parsing the BPE merges.txt data file during CodeGenTokenizer construction. The library wraps all underlying failures (I/O, format, encoding) as IOException with this message so callers get a consistent exception type, preserving the original exception as InnerException.","triggerScenarios":"Constructing a CodeGenTokenizer whose merge file stream is unreadable, a closed/disposed stream, a stream in the wrong encoding, or a stream whose content cannot be parsed line-by-line into 'token token' merge pairs.","commonSituations":"Passing a FileStream to a missing/locked file via a stream already consumed elsewhere; shipping a truncated or corrupted merges.txt with an app; opening the merge file with the wrong text encoding; a deployment step that did not copy the vocab data files.","solutions":["Verify the merge file exists at the expected path and the stream is open, readable, and not already consumed (seek to 0 if reusable).","Check the InnerException to identify the root cause (FileNotFound, UnauthorizedAccess, decoder failure, etc.).","Validate the merge file content: each non-empty line must contain exactly one space separating two tokens with no leading space.","Copy vocab/merge data files with the application build (ensure they are included as content and deployed)."],"exampleFix":"// before\nvar tokenizer = new CodeGenTokenizer(vocabStream, mergeStream);\n// after\nif (!mergeStream.CanRead) throw new InvalidOperationException(\"Merge stream is not readable\");\nmergeStream.Seek(0, SeekOrigin.Begin);\ntry { var tokenizer = new CodeGenTokenizer(vocabStream, mergeStream); }\ncatch (IOException ex) { Console.WriteLine($\"Merge file issue: {ex.InnerException?.Message}\"); throw; }","handlingStrategy":"try-catch","validationCode":"if (!mergeStream.CanRead) throw new InvalidOperationException(\"Merge stream must be readable\");\nif (mergeStream.CanSeek) mergeStream.Seek(0, SeekOrigin.Begin);","typeGuard":null,"tryCatchPattern":"try { var t = new CodeGenTokenizer(vocabStream, mergeStream); }\ncatch (IOException ex) { log(ex.InnerException ?? ex); throw new ApplicationException(\"Invalid or unreadable merge file\", ex); }","preventionTips":["Always open a fresh, positioned stream per tokenizer construction.","Deploy merges.txt alongside the app and verify presence at startup.","Wrap tokenizer construction in startup init so failures surface immediately.","Inspect InnerException before handling the outer IOException."],"tags":["io","file-read","tokenizer","csharp"],"backgroundTag":"file-read-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}