micro/go-micro · error

flow: LLMGrader returned an empty grade

Error message

flow: LLMGrader returned an empty grade

What it means

flow.parseGrade parses the LLM's reply inside LLMGrader and requires a non-empty first line containing PASS or FAIL. This error is returned when the model's reply, after trimming whitespace, is an empty string — meaning the grader received no usable grade text. It surfaces (wrapped) as a 'verify grade attempt N' error to the caller.

Source

Thrown at flow/verify.go:139

			return false, "", fmt.Errorf("flow: LLMGrader requires a flow model (set Provider/APIKey)")
		}
		prompt := fmt.Sprintf("Grade the latest result against this rubric:\n%s\n\nLatest result:\n%s\n\nAnswer with PASS or FAIL on the first line, followed by one short feedback sentence.", rubric, out.String())
		resp, err := d.model.Generate(ctx, &ai.Request{Prompt: prompt})
		if err != nil {
			return false, "", err
		}
		reply := resp.Answer
		if reply == "" {
			reply = resp.Reply
		}
		return parseGrade(reply)
	}
}

func parseGrade(reply string) (bool, string, error) {
	text := strings.TrimSpace(reply)
	if text == "" {
		return false, "", fmt.Errorf("flow: LLMGrader returned an empty grade")
	}
	lines := strings.SplitN(text, "\n", 2)
	first := strings.ToLower(strings.TrimSpace(lines[0]))
	feedback := ""
	if len(lines) > 1 {
		feedback = strings.TrimSpace(lines[1])
	}
	pass := strings.HasPrefix(first, "pass") || isAffirmative(first)
	if !pass && feedback == "" {
		feedback = text
	}
	return pass, feedback, nil
}

func stateWithField(s State, field, value string) (State, error) {
	var obj map[string]any
	if len(s.Data) > 0 && json.Unmarshal(s.Data, &obj) == nil && obj != nil {
		obj[field] = value

View on GitHub (pinned to 24529f1404)

Solutions

1. Retry the grading call — an empty reply is often transient; add a retry wrapper around the grader or configure Verify's attempt loop to tolerate it.
2. Check the model's max_tokens/stop settings and any content-filter configuration that could yield empty output.
3. Inspect the raw API response (logging) to confirm whether the provider returned an empty choices/content array.
4. Use a custom grader that validates the reply and retries once on empty before failing.

Example fix

// before
resp, err := d.model.Generate(ctx, &ai.Request{Prompt: prompt})
// after
resp, err := d.model.Generate(ctx, &ai.Request{Prompt: prompt, MaxTokens: 256})
if err == nil && strings.TrimSpace(resp.Text()) == "" {
    resp, err = d.model.Generate(ctx, &ai.Request{Prompt: prompt, MaxTokens: 256}) // retry once
}

out, err := step(ctx, in)
if err != nil {
    if strings.Contains(err.Error(), "returned an empty grade") {
        time.Sleep(2 * time.Second)
        out, err = step(ctx, in) // retry; empty replies are often transient
    }
    return out, err
}

When it happens

Trigger: The LLM API returns a 200 with an empty response body / empty content (e.g. content filter, stop-reason quirks, empty completion), and LLMGrader passes that reply to parseGrade.

Common situations: Provider returning empty content due to safety filtering; max_tokens set to 0 or extremely small; misconfigured model endpoint returning empty payloads; heavily truncated responses.

Related errors

• LLM step requires a flow model (set Provider/APIKey)
• flow: Verify requires a grader
• verify grade attempt %d: %w
• flow: LLMGrader requires a flow model (set Provider/APIKey)
• no response from API

AI-assisted analysis of micro/go-micro@24529f1404 (2026-09-01). Data as JSON: /api/errors/1f5c349a55ecd0dd. Report an issue: GitHub.