microsoft/semantic-kernel · error · KernelException

BLEU summary evaluation score ({precisions[0]}) is lower tha

Error message

BLEU summary evaluation score ({precisions[0]}) is lower than threshold ({threshold})

What it means

A Semantic Kernel IFunctionInvocationFilter (BleuSummarizationEvaluationFilter) evaluates a generated summary against the source using the BLEU metric (n-gram precision precisions). After retrieving precisions from the evaluation service, it throws a KernelException when precisions[0] (the 1-gram precision) is below the configured threshold.

Source

Thrown at dotnet/samples/Demos/QualityCheck/QualityCheckWithFilters/Filters/BleuSummarizationEvaluationFilter.cs:43

        var summary = context.Result.ToString();

        var request = new SummarizationEvaluationRequest { Sources = [sourceText], Summaries = [summary] };
        var response = await evaluationService.EvaluateAsync<SummarizationEvaluationRequest, BleuSummarizationEvaluationResponse>(request);

        var score = Math.Round(response.Score, 4);
        var precisions = response.Precisions.Select(l => Math.Round(l, 4)).ToList();
        var brevityPenalty = Math.Round(response.BrevityPenalty, 4);
        var lengthRatio = Math.Round(response.LengthRatio, 4);

        logger.LogInformation("[BLEU] Score: {Score}, Precisions: {Precisions}, Brevity penalty: {BrevityPenalty}, Length Ratio: {LengthRatio}",
            score,
            string.Join(", ", precisions),
            brevityPenalty,
            lengthRatio);

        if (precisions[0] < threshold)
        {
            throw new KernelException($"BLEU summary evaluation score ({precisions[0]}) is lower than threshold ({threshold})");
        }
    }
}

View on GitHub (pinned to c028a0c7dc)

Solutions

1. Lower the threshold for precisions[0] in the DI registration to match the model's typical BLEU performance.
2. Use a model or prompt that produces more extractive (literal) summaries, which score higher on BLEU.
3. Switch the quality metric to one more tolerant of paraphrase (e.g., BERTScore or METEOR) if paraphrasing is desired.
4. Catch KernelException at the orchestration layer and implement a retry-with-feedback loop.

Example fix

// before — check only 1-gram precision
if (precisions[0] < threshold)
{
    throw new KernelException($"BLEU summary evaluation score ({precisions[0]}) is lower than threshold ({threshold})");
}

// after — use the BLEU composite score or average of precisions
var avgPrecision = precisions.Average();
if (avgPrecision < threshold)
{
    logger.LogWarning("BLEU avg precision {Avg} below threshold {Threshold}.", avgPrecision, threshold);
}

try { await kernel.InvokeAsync(summarizeFunc); } catch (KernelException ex) when (ex.Message.Contains("BLEU summary evaluation score")) { logger.LogWarning("BLEU below threshold: {Msg}", ex.Message); }

When it happens

Trigger: The evaluation service returns BLEU 1-gram precision below the injected threshold, indicating low word-level overlap between the summary and the source text.

Common situations: Threshold configured too strictly; the model paraphrases heavily (BLEU penalizes exact n-gram matches, so paraphrase-heavy summaries score low); short source text or very short summary; mismatch between source language and summary language.

Related errors

• BERT summary evaluation score ({f1}) is lower than threshold
• METEOR summary evaluation score ({score}) is lower than thre
• COMET translation evaluation score ({score}) is lower than t
• Response is not available.
• Attribute '{node.func.attr}' is not callable in filter expre

AI-assisted analysis of microsoft/semantic-kernel@c028a0c7dc (2026-08-13). Data as JSON: /api/errors/712b057ba8b0f22a. Report an issue: GitHub.