microsoft/semantic-kernel · error · KernelException

METEOR summary evaluation score ({score}) is lower than thre

Error message

METEOR summary evaluation score ({score}) is lower than threshold ({threshold})

What it means

A Semantic Kernel IFunctionInvocationFilter (MeteorSummarizationEvaluationFilter) evaluates a generated summary against the source text using the METEOR metric. It throws a KernelException when the returned single score is below the configured threshold. METEOR accounts for synonyms and stemming, making it more forgiving than BLEU.

Source

Thrown at dotnet/samples/Demos/QualityCheck/QualityCheckWithFilters/Filters/MeteorSummarizationEvaluationFilter.cs:35

    double threshold) : IFunctionInvocationFilter
{
    public async Task OnFunctionInvocationAsync(FunctionInvocationContext context, Func<FunctionInvocationContext, Task> next)
    {
        await next(context);

        var sourceText = context.Result.RenderedPrompt!;
        var summary = context.Result.ToString();

        var request = new SummarizationEvaluationRequest { Sources = [sourceText], Summaries = [summary] };
        var response = await evaluationService.EvaluateAsync<SummarizationEvaluationRequest, MeteorSummarizationEvaluationResponse>(request);

        var score = Math.Round(response.Score, 4);

        logger.LogInformation("[METEOR] Score: {Score}", score);

        if (score < threshold)
        {
            throw new KernelException($"METEOR summary evaluation score ({score}) is lower than threshold ({threshold})");
        }
    }
}

View on GitHub (pinned to c028a0c7dc)

Solutions

1. Lower the threshold to match the model's expected METEOR range.
2. Improve the summarization prompt to encourage better content coverage and conciseness.
3. Catch the KernelException and retry with modified temperature or prompt instructions.
4. Compare against a baseline run to calibrate a realistic threshold before enforcing it.

Example fix

// before
if (score < threshold)
{
    throw new KernelException($"METEOR summary evaluation score ({score}) is lower than threshold ({threshold})");
}

// after — warning-level logging with continued execution
if (score < threshold)
{
    logger.LogWarning("METEOR score {Score} below threshold {Threshold}.", score, threshold);
}

try { await kernel.InvokeAsync(summarizeFunc); } catch (KernelException ex) when (ex.Message.Contains("METEOR summary evaluation score")) { logger.LogWarning("METEOR below threshold: {Msg}", ex.Message); }

When it happens

Trigger: The METEOR score returned by the evaluation service is below the injected threshold, indicating the summary's quality (considering synonyms, word order, and stemming) is insufficient.

Common situations: Threshold too high for the model; very short or very long summaries score poorly on METEOR; the source text and summary diverge in wording or coverage; evaluation service inconsistency or timeout producing low scores.

Related errors

• BERT summary evaluation score ({f1}) is lower than threshold
• BLEU summary evaluation score ({precisions[0]}) is lower tha
• COMET translation evaluation score ({score}) is lower than t
• Response is not available.
• Attribute '{node.func.attr}' is not callable in filter expre

AI-assisted analysis of microsoft/semantic-kernel@c028a0c7dc (2026-08-13). Data as JSON: /api/errors/8c1d39e653803de5. Report an issue: GitHub.