What causes "Failed to decode token ids" in dotnet/machinelearning?

Verify the tokenizer model (sentencepiece .model) matches the checkpoint's tokenizer

dotnet/machinelearning · error · InvalidOperationException

Failed to decode token ids

Error message

Failed to decode token ids

What it means

After generating tokens, Generate (streaming wrapper) decodes token ids to text; the tokenizer's Decode returned null for the duplicated-token string, so the pipeline throws InvalidOperationException. This is the SentencePiece branch decoding tokenIds.Concat(tokenIds) with special tokens considered.

Source

Thrown at src/Microsoft.ML.GenAI.Core/Pipeline/CausalLMPipeline.cs:275

            {
                var tokens = this.Tokenizer.EncodeToTokens(x, out var _, false, false);

                return tokens
                // Skip the first _ token automatically added by tokenizer
                .Where(t => !t.Offset.Equals(new Range(0, 0)))
                .Select(t => t.Id)
                .ToArray();
            }));
        }

        stopTokenIds = stopTokenIds.Where(ids => ids.Count() > 0).ToList();

        foreach (var (token, _) in this.GenerateStreaming(inputTensor, attentionMask, stopTokenIds.ToArray(), temperature: temperature, maxLen: maxLen))
        {
            var tokenIds = token[0].to_type(ScalarType.Int32).data<int>().ToArray();
            var duplicateTokenString = this.Tokenizer switch
            {
                SentencePieceTokenizer bpeTokenizer => bpeTokenizer.Decode(tokenIds.Concat(tokenIds), considerSpecialTokens: true) ?? throw new InvalidOperationException("Failed to decode token ids"),
                _ => this.Tokenizer.Decode(tokenIds.Concat(tokenIds)) ?? throw new InvalidOperationException("Failed to decode token ids"),
            };

            var tokenString = this.Tokenizer switch
            {
                SentencePieceTokenizer bpeTokenizer => bpeTokenizer.Decode(tokenIds, considerSpecialTokens: true) ?? throw new InvalidOperationException("Failed to decode token ids"),
                _ => this.Tokenizer.Decode(tokenIds) ?? throw new InvalidOperationException("Failed to decode token ids"),
            };

            // replace the first occurrence of the token with the duplicate token
            tokenString = duplicateTokenString.Substring(tokenString.Length);

            yield return tokenString;
        }
    }

    protected torch.Tensor SampleTopP(torch.Tensor logits, float topP)
    {

View on GitHub (pinned to 7b76e69cf9)

Solutions

Verify the tokenizer model (sentencepiece .model) matches the checkpoint's tokenizer
Check that generated token ids are within the tokenizer's vocabulary size
Handle the null decode result explicitly and fall back to per-token decoding

Example fix

// before
var text = bpeTokenizer.Decode(tokenIds, considerSpecialTokens: true) ?? throw new InvalidOperationException("Failed to decode token ids");
// after
var text = bpeTokenizer.Decode(tokenIds, considerSpecialTokens: true) ?? string.Join(" ", tokenIds.Select(t => $"[id {t}]"));

Defensive patterns

Strategy: try-catch

Validate before calling

var probe = tokenizer.Decode(new[] { 1, 2, 3 }, considerSpecialTokens: true);
if (probe is null) throw new InvalidOperationException("Tokenizer cannot decode sample ids; vocabulary mismatch suspected");

Type guard

bool CanDecode(ITokenizer t, int[] ids) => t.Decode(ids) is not null;

Try / catch

try { text = pipeline.Generate(prompt); } catch (InvalidOperationException ex) when (ex.Message == "Failed to decode token ids") { // verify tokenizer/model pairing }

Prevention

Pair tokenizer files with the exact checkpoint
Validate generated ids fall within vocab size
Test decode round-trip on sample ids before generation

When it happens

Trigger: The SentencePiece tokenizer's Decode(ids.Concat(ids), considerSpecialTokens:true) returns null for a batch of generated token ids.

Common situations: Tokenizer vocabulary/model file mismatched with the model's output ids (e.g. ids out of vocabulary range); special-token ids appearing where the piece model cannot assemble text.

Understand the failure class

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/0d2b54adc587737e. Report an issue: GitHub.