{"record":{"id":"8f0fe379eeb8abdd","repo":"microsoft/semantic-kernel","slug":"bert-summary-evaluation-score-f1-is-lower-than","errorCode":null,"errorMessage":"BERT summary evaluation score ({f1}) is lower than threshold ({threshold})","messagePattern":"BERT summary evaluation score \\((.+?)\\) is lower than threshold \\((.+?)\\)","errorType":"exception","errorClass":"KernelException","httpStatus":null,"severity":"error","filePath":"dotnet/samples/Demos/QualityCheck/QualityCheckWithFilters/Filters/BertSummarizationEvaluationFilter.cs","lineNumber":38,"sourceCode":"    public async Task OnFunctionInvocationAsync(FunctionInvocationContext context, Func<FunctionInvocationContext, Task> next)\n    {\n        await next(context);\n\n        var sourceText = context.Result.RenderedPrompt!;\n        var summary = context.Result.ToString();\n\n        var request = new SummarizationEvaluationRequest { Sources = [sourceText], Summaries = [summary] };\n        var response = await evaluationService.EvaluateAsync<SummarizationEvaluationRequest, BertSummarizationEvaluationResponse>(request);\n\n        var precision = Math.Round(response.Precision[0], 4);\n        var recall = Math.Round(response.Recall[0], 4);\n        var f1 = Math.Round(response.F1[0], 4);\n\n        logger.LogInformation(\"[BERT] Precision: {Precision}, Recall: {Recall}, F1: {F1}\", precision, recall, f1);\n\n        if (f1 < threshold)\n        {\n            throw new KernelException($\"BERT summary evaluation score ({f1}) is lower than threshold ({threshold})\");\n        }\n    }\n}\n","sourceCodeStart":20,"sourceCodeEnd":42,"githubUrl":"https://github.com/microsoft/semantic-kernel/blob/c028a0c7dc4f0814cdcbaba9d998f187a41197bf/dotnet/samples/Demos/QualityCheck/QualityCheckWithFilters/Filters/BertSummarizationEvaluationFilter.cs#L20-L42","documentation":"A Semantic Kernel IFunctionInvocationFilter (BertSummarizationEvaluationFilter) runs after a summarization function completes, sends the source text and generated summary to a BERTScore evaluation service, and throws a KernelException when the computed F1 score falls below the configured threshold. The filter enforces a minimum quality gate on LLM-generated summaries.","triggerScenarios":"The threshold (injected via the filter's primary constructor) is exceeded downward: the evaluation service returns F1 < threshold. This happens when the LLM-generated summary diverges significantly from the source text in embedding space.","commonSituations":"Threshold set too high for the model's typical output quality; a weaker/cheaper model produces summaries that score poorly on BERTScore; the source text is very short or noisy; the evaluation model or service returns unexpected scores; the summary is too brief or off-topic.","solutions":["Lower the threshold in the filter registration/DI configuration to a value the model can reliably meet.","Improve the summarization prompt or switch to a stronger model so the F1 score clears the threshold.","Catch the KernelException in the calling code and retry with a different prompt or temperature.","Replace the throw with a logging warning + return-the-summary-with-a-quality-flag approach if hard failures are undesirable."],"exampleFix":"// before — hard fail when below threshold\nif (f1 < threshold)\n{\n    throw new KernelException($\"BERT summary evaluation score ({f1}) is lower than threshold ({threshold})\");\n}\n\n// after — log and flag instead of throwing\nif (f1 < threshold)\n{\n    logger.LogWarning(\"BERT F1 {F1} below threshold {Threshold}; proceeding with low-confidence summary.\", f1, threshold);\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { await kernel.InvokeAsync(summarizeFunc); } catch (KernelException ex) when (ex.Message.Contains(\"BERT summary evaluation score\")) { logger.LogWarning(\"Summary below BERT threshold: {Msg}\", ex.Message); /* retry or accept */ }","preventionTips":["Calibrate thresholds against a baseline run before enforcing them.","Log all metric scores (precision, recall, F1) so you can reason about threshold adjustments.","Consider a soft-gate (log + flag) before promoting to a hard throw."],"tags":["quality-check","bertscore","evaluation","kernel-exception","filter","summarization"],"backgroundTag":null,"analyzedSha":"c028a0c7dc4f0814cdcbaba9d998f187a41197bf","analyzedAt":"2026-08-13T13:48:05.040Z","schemaVersion":2},"datasetVersion":"2026-08-13T14:17:21.547Z"}