{"record":{"id":"0d2b54adc587737e","repo":"dotnet/machinelearning","slug":"failed-to-decode-token-ids","errorCode":null,"errorMessage":"Failed to decode token ids","messagePattern":"Failed to decode token ids","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Microsoft.ML.GenAI.Core/Pipeline/CausalLMPipeline.cs","lineNumber":275,"sourceCode":"            {\n                var tokens = this.Tokenizer.EncodeToTokens(x, out var _, false, false);\n\n                return tokens\n                // Skip the first _ token automatically added by tokenizer\n                .Where(t => !t.Offset.Equals(new Range(0, 0)))\n                .Select(t => t.Id)\n                .ToArray();\n            }));\n        }\n\n        stopTokenIds = stopTokenIds.Where(ids => ids.Count() > 0).ToList();\n\n        foreach (var (token, _) in this.GenerateStreaming(inputTensor, attentionMask, stopTokenIds.ToArray(), temperature: temperature, maxLen: maxLen))\n        {\n            var tokenIds = token[0].to_type(ScalarType.Int32).data<int>().ToArray();\n            var duplicateTokenString = this.Tokenizer switch\n            {\n                SentencePieceTokenizer bpeTokenizer => bpeTokenizer.Decode(tokenIds.Concat(tokenIds), considerSpecialTokens: true) ?? throw new InvalidOperationException(\"Failed to decode token ids\"),\n                _ => this.Tokenizer.Decode(tokenIds.Concat(tokenIds)) ?? throw new InvalidOperationException(\"Failed to decode token ids\"),\n            };\n\n            var tokenString = this.Tokenizer switch\n            {\n                SentencePieceTokenizer bpeTokenizer => bpeTokenizer.Decode(tokenIds, considerSpecialTokens: true) ?? throw new InvalidOperationException(\"Failed to decode token ids\"),\n                _ => this.Tokenizer.Decode(tokenIds) ?? throw new InvalidOperationException(\"Failed to decode token ids\"),\n            };\n\n            // replace the first occurrence of the token with the duplicate token\n            tokenString = duplicateTokenString.Substring(tokenString.Length);\n\n            yield return tokenString;\n        }\n    }\n\n    protected torch.Tensor SampleTopP(torch.Tensor logits, float topP)\n    {","sourceCodeStart":257,"sourceCodeEnd":293,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.GenAI.Core/Pipeline/CausalLMPipeline.cs#L257-L293","documentation":"After generating tokens, Generate (streaming wrapper) decodes token ids to text; the tokenizer's Decode returned null for the duplicated-token string, so the pipeline throws InvalidOperationException. This is the SentencePiece branch decoding tokenIds.Concat(tokenIds) with special tokens considered.","triggerScenarios":"The SentencePiece tokenizer's Decode(ids.Concat(ids), considerSpecialTokens:true) returns null for a batch of generated token ids.","commonSituations":"Tokenizer vocabulary/model file mismatched with the model's output ids (e.g. ids out of vocabulary range); special-token ids appearing where the piece model cannot assemble text.","solutions":["Verify the tokenizer model (sentencepiece .model) matches the checkpoint's tokenizer","Check that generated token ids are within the tokenizer's vocabulary size","Handle the null decode result explicitly and fall back to per-token decoding"],"exampleFix":"// before\nvar text = bpeTokenizer.Decode(tokenIds, considerSpecialTokens: true) ?? throw new InvalidOperationException(\"Failed to decode token ids\");\n// after\nvar text = bpeTokenizer.Decode(tokenIds, considerSpecialTokens: true) ?? string.Join(\" \", tokenIds.Select(t => $\"[id {t}]\"));","handlingStrategy":"try-catch","validationCode":"var probe = tokenizer.Decode(new[] { 1, 2, 3 }, considerSpecialTokens: true);\nif (probe is null) throw new InvalidOperationException(\"Tokenizer cannot decode sample ids; vocabulary mismatch suspected\");","typeGuard":"bool CanDecode(ITokenizer t, int[] ids) => t.Decode(ids) is not null;","tryCatchPattern":"try { text = pipeline.Generate(prompt); } catch (InvalidOperationException ex) when (ex.Message == \"Failed to decode token ids\") { // verify tokenizer/model pairing }","preventionTips":["Pair tokenizer files with the exact checkpoint","Validate generated ids fall within vocab size","Test decode round-trip on sample ids before generation"],"tags":["genai","tokenizer","decode"],"backgroundTag":"null-argument","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}