{"record":{"id":"5a989e02fd015a4a","repo":"dotnet/machinelearning","slug":"cannot-read-the-file-merge-file-environment-newli-5a989e","errorCode":null,"errorMessage":"Cannot read the file Merge file.{Environment.NewLine}Error message: {e.Message}","messagePattern":"Cannot read the file Merge file\\.(.+?)Error message: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs","lineNumber":228,"sourceCode":"                }\n\n                int rank = 1;\n                while (reader.Peek() >= 0)\n                {\n                    string line = reader.ReadLine()!;\n                    int index = line.IndexOf(' ');\n                    if (index < 1 || index == line.Length - 1 || line.IndexOf(' ', index + 1) != -1)\n                    {\n                        throw new FormatException($\"Invalid format of merge file at line: \\\"{line}\\\"\");\n                    }\n\n                    mergeRanks.Set((line.Substring(0, index), line.Substring(index + 1)), rank++);\n                }\n            }\n            catch (Exception e)\n            {\n                // Report any issues encountered while consuming a data file as IOExceptions.\n                throw new IOException($\"Cannot read the file Merge file.{Environment.NewLine}Error message: {e.Message}\", e);\n            }\n\n            return mergeRanks;\n        }\n\n        private Dictionary<string, int> GetVocab()\n        {\n            Dictionary<string, int>? publicVocab = Volatile.Read(ref _vocabOriginal);\n            if (publicVocab is null)\n            {\n                var vocab = new Dictionary<string, int>();\n                foreach (var item in _vocab)\n                {\n                    vocab.Add(item.Key.ToString(), item.Value);\n                }\n\n                Interlocked.CompareExchange(ref _vocabOriginal, vocab, null);\n                publicVocab = _vocabOriginal;","sourceCodeStart":210,"sourceCodeEnd":246,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs#L210-L246","documentation":"Thrown when any exception occurs while reading the BPE merge file stream in EnglishRobertaTokenizer.GetMergeRanks. All underlying failures are reported as IOException with this message and the original exception attached as InnerException, mirroring the CodeGenTokenizer behavior.","triggerScenarios":"The merge stream is closed/disposed, unreadable, already fully consumed, or an I/O/decoder error occurs mid-read while parsing merge lines.","commonSituations":"Reusing a stream already read by another tokenizer instance; file deleted or locked between open and read; network stream interrupted when reading embedded resources over a stream; wrong encoding causing a decoder exception.","solutions":["Check InnerException to find the root cause and address that specific failure.","Ensure the merge stream is open, readable, and positioned at 0 before constructing the tokenizer.","Open a fresh stream per tokenizer construction instead of sharing one across instances.","Verify the file is present, accessible, and its encoding is UTF-8 compatible."],"exampleFix":"// before\nvar shared = File.OpenRead(\"merges.txt\");\nvar t1 = new EnglishRobertaTokenizer(vocab, shared);\nvar t2 = new EnglishRobertaTokenizer(vocab, shared); // stream consumed\n// after\nvar t1 = new EnglishRobertaTokenizer(vocab, File.OpenRead(\"merges.txt\"));\nvar t2 = new EnglishRobertaTokenizer(vocab, File.OpenRead(\"merges.txt\"));","handlingStrategy":"try-catch","validationCode":"if (!mergesStream.CanRead) throw new InvalidOperationException(\"Merges stream must be readable\");\nif (mergesStream.CanSeek) mergesStream.Seek(0, SeekOrigin.Begin);","typeGuard":null,"tryCatchPattern":"try { var t = new EnglishRobertaTokenizer(vocabStream, mergesStream); }\ncatch (IOException ex) { log(ex.InnerException ?? ex); throw new InvalidDataException(\"Could not read merges file\", ex); }","preventionTips":["Use File.OpenRead (or equivalent) fresh per construction; never share streams between tokenizers.","Verify file access permissions and encoding before loading.","Load tokenizer models during startup, not per-request.","Diagnose via InnerException before handling generically."],"tags":["io","file-read","tokenizer","csharp"],"backgroundTag":"file-read-failed","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}