dotnet/machinelearning · error · IOException

Cannot read the file Merge file.{Environment.NewLine}Error m

Error message

Cannot read the file Merge file.{Environment.NewLine}Error message: {e.Message}

What it means

Thrown when any exception occurs while reading the BPE merge file stream in EnglishRobertaTokenizer.GetMergeRanks. All underlying failures are reported as IOException with this message and the original exception attached as InnerException, mirroring the CodeGenTokenizer behavior.

Source

Thrown at src/Microsoft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs:228

                }

                int rank = 1;
                while (reader.Peek() >= 0)
                {
                    string line = reader.ReadLine()!;
                    int index = line.IndexOf(' ');
                    if (index < 1 || index == line.Length - 1 || line.IndexOf(' ', index + 1) != -1)
                    {
                        throw new FormatException($"Invalid format of merge file at line: \"{line}\"");
                    }

                    mergeRanks.Set((line.Substring(0, index), line.Substring(index + 1)), rank++);
                }
            }
            catch (Exception e)
            {
                // Report any issues encountered while consuming a data file as IOExceptions.
                throw new IOException($"Cannot read the file Merge file.{Environment.NewLine}Error message: {e.Message}", e);
            }

            return mergeRanks;
        }

        private Dictionary<string, int> GetVocab()
        {
            Dictionary<string, int>? publicVocab = Volatile.Read(ref _vocabOriginal);
            if (publicVocab is null)
            {
                var vocab = new Dictionary<string, int>();
                foreach (var item in _vocab)
                {
                    vocab.Add(item.Key.ToString(), item.Value);
                }

                Interlocked.CompareExchange(ref _vocabOriginal, vocab, null);
                publicVocab = _vocabOriginal;

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Check InnerException to find the root cause and address that specific failure.
2. Ensure the merge stream is open, readable, and positioned at 0 before constructing the tokenizer.
3. Open a fresh stream per tokenizer construction instead of sharing one across instances.
4. Verify the file is present, accessible, and its encoding is UTF-8 compatible.

Example fix

// before
var shared = File.OpenRead("merges.txt");
var t1 = new EnglishRobertaTokenizer(vocab, shared);
var t2 = new EnglishRobertaTokenizer(vocab, shared); // stream consumed
// after
var t1 = new EnglishRobertaTokenizer(vocab, File.OpenRead("merges.txt"));
var t2 = new EnglishRobertaTokenizer(vocab, File.OpenRead("merges.txt"));

if (!mergesStream.CanRead) throw new InvalidOperationException("Merges stream must be readable");
if (mergesStream.CanSeek) mergesStream.Seek(0, SeekOrigin.Begin);

try { var t = new EnglishRobertaTokenizer(vocabStream, mergesStream); }
catch (IOException ex) { log(ex.InnerException ?? ex); throw new InvalidDataException("Could not read merges file", ex); }

When it happens

Trigger: The merge stream is closed/disposed, unreadable, already fully consumed, or an I/O/decoder error occurs mid-read while parsing merge lines.

Common situations: Reusing a stream already read by another tokenizer instance; file deleted or locked between open and read; network stream interrupted when reading embedded resources over a stream; wrong encoding causing a decoder exception.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Cannot read the file Merge file.{Environment.NewLine}Error m
• throw new ArgumentNullException(nameof(vocabStream));
• throw new ArgumentNullException(nameof(vocabFilePath));
• throw new ArgumentNullException(nameof(vocabStream));
• The special token '{kvp.Key}' is not in the vocabulary or as

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/5a989e02fd015a4a. Report an issue: GitHub.