{"record":{"id":"51b0797f6df3508b","repo":"dotnet/machinelearning","slug":"the-provided-token-ids-could-not-be-decoded","errorCode":null,"errorMessage":"The provided token IDs could not be decoded.","messagePattern":"The provided token IDs could not be decoded\\.","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.Tokenizers/Tokenizer.cs","lineNumber":412,"sourceCode":"                        return result;\n\n                    case OperationStatus.DestinationTooSmall:\n                        long newSize = (long)destination.Length * 2;\n                        if (newSize > int.MaxValue)\n                        {\n                            newSize = (long)destination.Length + 1;\n                            if (newSize > int.MaxValue)\n                            {\n                                throw new OutOfMemoryException();\n                            }\n                        }\n\n                        ArrayPool<char>.Shared.Return(destination);\n                        destination = ArrayPool<char>.Shared.Rent((int)newSize);\n                        break;\n\n                    default:\n                        throw new InvalidOperationException(\"The provided token IDs could not be decoded.\");\n                }\n            }\n        }\n\n        /// <summary>\n        /// Decode the given ids back to text and store the result in the <paramref name=\"destination\"/> span.\n        /// </summary>\n        /// <param name=\"ids\">The list of ids that we want to decode.</param>\n        /// <param name=\"destination\">The span to store the decoded text.</param>\n        /// <param name=\"idsConsumed\">The number of ids consumed during the decoding.</param>\n        /// <param name=\"charsWritten\">The number of characters written to the destination span.</param>\n        /// <returns>The operation status indicates whether all IDs were successfully decoded or if the <paramref name=\"destination\"/> is too small to contain the entire decoded result.</returns>\n        public abstract OperationStatus Decode(IEnumerable<int> ids, Span<char> destination, out int idsConsumed, out int charsWritten);\n\n        internal static IEnumerable<(int Offset, int Length)>? InitializeForEncoding(\n                                                string? text,\n                                                ReadOnlySpan<char> textSpan,\n                                                bool considerPreTokenization,","sourceCodeStart":394,"sourceCodeEnd":430,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.Tokenizers/Tokenizer.cs#L394-L430","documentation":"This InvalidOperationException is thrown inside Tokenizer.Decode's internal decoding loop when a decoder state machine reaches an unknown state while turning token IDs back into text. It effectively means the decoder could not produce output for the supplied IDs — the internal decode loop's switch exhausted all expected cases without terminating. Callers see it as 'these IDs cannot be decoded by this tokenizer'.","triggerScenarios":"Calling Tokenizer.Decode / DecodeToString with token IDs that were not produced by the same tokenizer (foreign vocabulary IDs), or with IDs that drive the byte-level decoder into an unrecoverable state (e.g. an ID mapping to a partial multi-byte UTF-8 sequence with no continuation).","commonSituations":"Mixing tokenizers (encoding with GPT-2 BPE but decoding with a Llama tokenizer), passing hand-crafted or persisted IDs from an older model version, or decoding IDs read from storage after the vocab/model was updated.","solutions":["Ensure the IDs were produced by the same tokenizer model/vocab used for decoding — encode and decode with the same Tokenizer instance.","Verify the vocabulary/model file was not changed or upgraded between encoding and decoding; re-encode the original text.","Wrap Decode in try/catch for InvalidOperationException and fall back to reconstructing text from raw token strings via the vocab.","Validate IDs are within range [0, vocab.Count) before decoding."],"exampleFix":"// before: decoding IDs from a different tokenizer\nstring text = llamaTokenizer.DecodeToString(gpt2Ids);\n// after: use matching encoder/decoder\nIReadOnlyList<int> ids = llamaTokenizer.EncodeToIds(text);\nstring text = llamaTokenizer.DecodeToString(ids);","handlingStrategy":"try-catch","validationCode":"bool idsInVocab = ids.All(id => id >= 0 && id < tokenizer.VocabLength); // adjust to tokenizer's vocab size property","typeGuard":"static bool AreDecodableIds(IReadOnlyList<int> ids, int vocabSize) => ids is not null && ids.All(id => (uint)id < (uint)vocabSize);","tryCatchPattern":"try { text = tokenizer.DecodeToString(ids); }\ncatch (InvalidOperationException ex) when (ex.Message.Contains(\"could not be decoded\"))\n{ text = fallbackRawDecode(ids); }","preventionTips":["Always pair encode/decode with the same Tokenizer instance","Persist the model version alongside stored token IDs","Range-check IDs against vocab size before decoding"],"tags":["tokenizer","decoding","invalid-state"],"backgroundTag":"invalid-state-transition","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}