What causes "Logits is null" in dotnet/machinelearning?

Verify the loaded model's forward always populates Logits in CausalLMModelOutput

dotnet/machinelearning · error · InvalidOperationException

Logits is null

Error message

Logits is null

What it means

During streaming generation, the model's forward output must carry a Logits tensor; if output.Logits is null the pipeline cannot sample the next token and throws InvalidOperationException after moving tensors to the correct dispose scope.

Source

Thrown at src/Microsoft.ML.GenAI.Core/Pipeline/CausalLMPipeline.cs:160

        torch.Tensor? logits = default;
        var cache = new DynamicKVCache();
        if (promptLength == totalLen)
        {
            var input = new CausalLMModelInput(inputIds, attentionMask, pastKeyValuesLength: 0)
            {
                OverrideCache = cache,
            };
            var output = this.Model.forward(input);
            logits = output.Logits;
        }
        for (var curPos = promptLength; curPos != totalLen; curPos++)
        {
            var input = new CausalLMModelInput(inputIds[.., prevPos..curPos], attentionMask[.., prevPos..curPos], pastKeyValuesLength: prevPos)
            {
                OverrideCache = cache,
            };
            var output = this.Model.forward(input);
            logits = output.Logits?.MoveToOtherDisposeScope(inputIds) ?? throw new InvalidOperationException("Logits is null");
            torch.Tensor nextToken;
            if (temperature > 0)
            {
                var probs = torch.softmax(logits[.., -1] / temperature, dim: -1);
                nextToken = this.SampleTopP(probs, topP);
            }
            else
            {
                nextToken = torch.argmax(logits[.., -1], dim: -1);
            }

            nextToken = nextToken.reshape(-1);
            inputIds = torch.cat([inputIds, nextToken.unsqueeze(1)], dim: -1).MoveToOtherDisposeScope(inputIds);
            attentionMask = torch.cat([attentionMask, attentionMask.new_ones(attentionMask.shape[0], 1)], dim: -1);
            foreach (var stopSequence in stopTokenSequence)
            {
                // determine if the last n tokens are the stop sequence
                var lastN = inputIds[.., ^stopSequence.Length..];

View on GitHub (pinned to 7b76e69cf9)

Solutions

Verify the loaded model's forward always populates Logits in CausalLMModelOutput
Check that the model head (lm_head) is loaded and attached
Use a stock model implementation for the checkpoint rather than a custom forward override

Example fix

// before
var output = model.forward(input); // custom impl returns Logits = null
// after
class MyOutput : CausalLMModelOutput {
    public override Tensor Logits => lmHead(lastHiddenState); // ensure logits populated
}

Defensive patterns

Strategy: try-catch

Validate before calling

var probe = Model.forward(CausalLMModelInput.CreateTest(batch:1, seq:1));
if (probe.Logits is null) throw new InvalidOperationException("Model forward does not produce logits");

Type guard

bool HasLogits(CausalLMModelOutput o) => o?.Logits is not null && !o.Logits.IsInvalid;

Try / catch

try { await foreach (var t in pipeline.GenerateStreaming(input, mask, stopTokens)) { /* consume */ } } catch (InvalidOperationException ex) when (ex.Message == "Logits is null") { // fallback to a known-good model implementation }

Prevention

Use stock model implementations matching the checkpoint architecture
Verify lm_head is loaded during weight loading
Test a single-token forward before running long generations

When it happens

Trigger: Calling GenerateStreaming (directly or via Generate) when Model.forward returns a CausalLMModelOutput whose Logits property is null for the given input shape/cache configuration.

Common situations: Custom model implementation returning output without logits; model/forward wrapper mismatch where the head is disabled; incorrect CausalLMModelInput construction causing the model to skip logits computation.

Understand the failure class

Background: "invalid response format", "malformed payload", "missing data field": when an API returns 200 but the response shape is wrong — this error's family across 23 libraries.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/57fb8771db735c08. Report an issue: GitHub.