microsoft/semantic-kernel · error · KernelException

BERT summary evaluation score ({f1}) is lower than threshold

Error message

BERT summary evaluation score ({f1}) is lower than threshold ({threshold})

What it means

A Semantic Kernel IFunctionInvocationFilter (BertSummarizationEvaluationFilter) runs after a summarization function completes, sends the source text and generated summary to a BERTScore evaluation service, and throws a KernelException when the computed F1 score falls below the configured threshold. The filter enforces a minimum quality gate on LLM-generated summaries.

Source

Thrown at dotnet/samples/Demos/QualityCheck/QualityCheckWithFilters/Filters/BertSummarizationEvaluationFilter.cs:38

    public async Task OnFunctionInvocationAsync(FunctionInvocationContext context, Func<FunctionInvocationContext, Task> next)
    {
        await next(context);

        var sourceText = context.Result.RenderedPrompt!;
        var summary = context.Result.ToString();

        var request = new SummarizationEvaluationRequest { Sources = [sourceText], Summaries = [summary] };
        var response = await evaluationService.EvaluateAsync<SummarizationEvaluationRequest, BertSummarizationEvaluationResponse>(request);

        var precision = Math.Round(response.Precision[0], 4);
        var recall = Math.Round(response.Recall[0], 4);
        var f1 = Math.Round(response.F1[0], 4);

        logger.LogInformation("[BERT] Precision: {Precision}, Recall: {Recall}, F1: {F1}", precision, recall, f1);

        if (f1 < threshold)
        {
            throw new KernelException($"BERT summary evaluation score ({f1}) is lower than threshold ({threshold})");
        }
    }
}

View on GitHub (pinned to c028a0c7dc)

Solutions

1. Lower the threshold in the filter registration/DI configuration to a value the model can reliably meet.
2. Improve the summarization prompt or switch to a stronger model so the F1 score clears the threshold.
3. Catch the KernelException in the calling code and retry with a different prompt or temperature.
4. Replace the throw with a logging warning + return-the-summary-with-a-quality-flag approach if hard failures are undesirable.

Example fix

// before — hard fail when below threshold
if (f1 < threshold)
{
    throw new KernelException($"BERT summary evaluation score ({f1}) is lower than threshold ({threshold})");
}

// after — log and flag instead of throwing
if (f1 < threshold)
{
    logger.LogWarning("BERT F1 {F1} below threshold {Threshold}; proceeding with low-confidence summary.", f1, threshold);
}

try { await kernel.InvokeAsync(summarizeFunc); } catch (KernelException ex) when (ex.Message.Contains("BERT summary evaluation score")) { logger.LogWarning("Summary below BERT threshold: {Msg}", ex.Message); /* retry or accept */ }

When it happens

Trigger: The threshold (injected via the filter's primary constructor) is exceeded downward: the evaluation service returns F1 < threshold. This happens when the LLM-generated summary diverges significantly from the source text in embedding space.

Common situations: Threshold set too high for the model's typical output quality; a weaker/cheaper model produces summaries that score poorly on BERTScore; the source text is very short or noisy; the evaluation model or service returns unexpected scores; the summary is too brief or off-topic.

Related errors

• BLEU summary evaluation score ({precisions[0]}) is lower tha
• METEOR summary evaluation score ({score}) is lower than thre
• COMET translation evaluation score ({score}) is lower than t
• Response is not available.
• Attribute '{node.func.attr}' is not callable in filter expre

AI-assisted analysis of microsoft/semantic-kernel@c028a0c7dc (2026-08-13). Data as JSON: /api/errors/8f0fe379eeb8abdd. Report an issue: GitHub.