{"record":{"id":"cedd01259d612fc3","repo":"dotnet/machinelearning","slug":"the-merge-value-mergevalues-b-is-too-short-to","errorCode":null,"errorMessage":"The merge value '{mergeValues.b}' is too short to be merged with a prefix of length {prefixLen}. This implies that the merge file is either damaged or missing the prefix in its entries.","messagePattern":"The merge value '(.+?)' is too short to be merged with a prefix of length (.+?)\\. This implies that the merge file is either damaged or missing the prefix in its entries\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs","lineNumber":380,"sourceCode":"\n            Merges = new();\n            for (int i = 0; i < merges.Count; i++)\n            {\n                (string a, string b) mergeValues = merges[i];\n\n                if (!_vocab.TryGetValue(mergeValues.a, out int aId))\n                {\n                    throw new InvalidOperationException($\"Trying to merge a token '{mergeValues.a}' which not exist in the vocabulary.\");\n                }\n\n                if (!_vocab.TryGetValue(mergeValues.b, out int bId))\n                {\n                    throw new InvalidOperationException($\"Trying to merge a token '{mergeValues.b}' which not exist in the vocabulary.\");\n                }\n\n                if (mergeValues.b.Length <= prefixLen)\n                {\n                    throw new InvalidOperationException($\"The merge value '{mergeValues.b}' is too short to be merged with a prefix of length {prefixLen}. This implies that the merge file is either damaged or missing the prefix in its entries.\");\n                }\n\n                string newToken = $\"{mergeValues.a}{mergeValues.b.Substring(prefixLen)}\";\n                if (!_vocab.TryGetValue(newToken, out int newId))\n                {\n                    throw new InvalidOperationException($\"Trying to merge a token '{newToken}' which not exist in the vocabulary.\");\n                }\n\n                Merges.Add(new Pair<int>(aId, bId), (i, newId));\n            }\n        }\n\n        /// <summary>\n        /// Gets a value indicating whether to handle the input text in byte level.\n        /// if true, the input text will be converted to UTF-8 bytes before encoding it.\n        /// Additionally, some ASCII characters will be transformed to another characters (e.g Space character will be transformed to 'Ġ' character).\n        /// </summary>\n        public bool ByteLevel { get; }","sourceCodeStart":362,"sourceCodeEnd":398,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs#L362-L398","documentation":"When processing a merge whose right token should be split against a continuing-subword prefix length, the right token is shorter than (or equal to) the expected prefix length, so the split is impossible. The BpeTokenizer constructor throws InvalidOperationException, treating this as evidence that the merges file is damaged or inconsistent with the configured continuingSubwordPrefix.","triggerScenarios":"Constructing BpeTokenizer with a non-empty continuingSubwordPrefix (e.g. \"Ġ\") while the merges file contains right-hand tokens shorter than the prefix length, i.e. token strings that do not start with the expected prefix.","commonSituations":"Using merges.txt without the byte-level space marker while configuring continuingSubwordPrefix: \"Ġ\"; mixing models that use different space-prefix conventions (SentencePiece ▁ vs GPT-2 Ġ); corrupted merge lines.","solutions":["Pass continuingSubwordPrefix: null (or the correct value) so it matches the actual prefix convention used in merges.txt.","Use the merges file belonging to the same model as the vocab — verify checksums.","Inspect the failing merge line and fix or remove the malformed entry.","Convert the merges file to the expected convention if migrating between tokenizers."],"exampleFix":"// before\nvar tok = new BpeTokenizer(vocab, merges, continuingSubwordPrefix: \"Ġ\"); // merges lack 'Ġ' prefixes\n// after\nvar tok = new BpeTokenizer(vocab, merges, continuingSubwordPrefix: null); // or use GPT-2-style merges","handlingStrategy":"validation","validationCode":"if (!string.IsNullOrEmpty(prefix))\n    foreach (var (_, b) in merges)\n        if (b.Length <= prefix.Length) throw new InvalidDataException($\"Merge token '{b}' is shorter than prefix '{prefix}'; merges file does not match prefix convention\");","typeGuard":"null","tryCatchPattern":"try { var tok = new BpeTokenizer(vocab, merges, continuingSubwordPrefix: prefix); }\ncatch (InvalidOperationException ex) when (ex.Message.Contains(\"too short to be merged\")) { throw new InvalidDataException(\"continuingSubwordPrefix does not match merges.txt convention\", ex); }","preventionTips":["Match continuingSubwordPrefix to the marker actually used in merges.txt (Ġ vs none vs ▁).","Inspect a few merge lines before choosing the prefix value.","Do not copy constructor options between models with different space-prefix conventions."],"tags":["merge-file","prefix","bpe","corrupt-data"],"backgroundTag":"internal-invariant-violation","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}