dotnet/machinelearning · error · InvalidOperationException

The provided token IDs could not be decoded.

Error message

The provided token IDs could not be decoded.

What it means

This InvalidOperationException is thrown inside Tokenizer.Decode's internal decoding loop when a decoder state machine reaches an unknown state while turning token IDs back into text. It effectively means the decoder could not produce output for the supplied IDs — the internal decode loop's switch exhausted all expected cases without terminating. Callers see it as 'these IDs cannot be decoded by this tokenizer'.

Source

Thrown at src/Microsoft.ML.Tokenizers/Tokenizer.cs:412

                        return result;

                    case OperationStatus.DestinationTooSmall:
                        long newSize = (long)destination.Length * 2;
                        if (newSize > int.MaxValue)
                        {
                            newSize = (long)destination.Length + 1;
                            if (newSize > int.MaxValue)
                            {
                                throw new OutOfMemoryException();
                            }
                        }

                        ArrayPool<char>.Shared.Return(destination);
                        destination = ArrayPool<char>.Shared.Rent((int)newSize);
                        break;

                    default:
                        throw new InvalidOperationException("The provided token IDs could not be decoded.");
                }
            }
        }

        /// <summary>
        /// Decode the given ids back to text and store the result in the <paramref name="destination"/> span.
        /// </summary>
        /// <param name="ids">The list of ids that we want to decode.</param>
        /// <param name="destination">The span to store the decoded text.</param>
        /// <param name="idsConsumed">The number of ids consumed during the decoding.</param>
        /// <param name="charsWritten">The number of characters written to the destination span.</param>
        /// <returns>The operation status indicates whether all IDs were successfully decoded or if the <paramref name="destination"/> is too small to contain the entire decoded result.</returns>
        public abstract OperationStatus Decode(IEnumerable<int> ids, Span<char> destination, out int idsConsumed, out int charsWritten);

        internal static IEnumerable<(int Offset, int Length)>? InitializeForEncoding(
                                                string? text,
                                                ReadOnlySpan<char> textSpan,
                                                bool considerPreTokenization,

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Ensure the IDs were produced by the same tokenizer model/vocab used for decoding — encode and decode with the same Tokenizer instance.
2. Verify the vocabulary/model file was not changed or upgraded between encoding and decoding; re-encode the original text.
3. Wrap Decode in try/catch for InvalidOperationException and fall back to reconstructing text from raw token strings via the vocab.
4. Validate IDs are within range [0, vocab.Count) before decoding.

Example fix

// before: decoding IDs from a different tokenizer
string text = llamaTokenizer.DecodeToString(gpt2Ids);
// after: use matching encoder/decoder
IReadOnlyList<int> ids = llamaTokenizer.EncodeToIds(text);
string text = llamaTokenizer.DecodeToString(ids);

bool idsInVocab = ids.All(id => id >= 0 && id < tokenizer.VocabLength); // adjust to tokenizer's vocab size property

static bool AreDecodableIds(IReadOnlyList<int> ids, int vocabSize) => ids is not null && ids.All(id => (uint)id < (uint)vocabSize);

try { text = tokenizer.DecodeToString(ids); }
catch (InvalidOperationException ex) when (ex.Message.Contains("could not be decoded"))
{ text = fallbackRawDecode(ids); }

When it happens

Trigger: Calling Tokenizer.Decode / DecodeToString with token IDs that were not produced by the same tokenizer (foreign vocabulary IDs), or with IDs that drive the byte-level decoder into an unrecoverable state (e.g. an ID mapping to a partial multi-byte UTF-8 sequence with no continuation).

Common situations: Mixing tokenizers (encoding with GPT-2 BPE but decoding with a Llama tokenizer), passing hand-crafted or persisted IDs from an older model version, or decoding IDs read from storage after the vocab/model was updated.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• The beginning of sentence token '{beginningOfSentenceToken}'
• The end of sentence token '{endOfSentenceToken}' was not pre
• There are no columns in the DataFrame to use as value column
• Start must be called on a ModelLoader before it can be used.
• Current estimator chain has no estimator, can't append cache

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/51b0797f6df3508b. Report an issue: GitHub.