{"record":{"id":"712b057ba8b0f22a","repo":"microsoft/semantic-kernel","slug":"bleu-summary-evaluation-score-precisions-0-is","errorCode":null,"errorMessage":"BLEU summary evaluation score ({precisions[0]}) is lower than threshold ({threshold})","messagePattern":"BLEU summary evaluation score \\((.+?)\\) is lower than threshold \\((.+?)\\)","errorType":"exception","errorClass":"KernelException","httpStatus":null,"severity":"error","filePath":"dotnet/samples/Demos/QualityCheck/QualityCheckWithFilters/Filters/BleuSummarizationEvaluationFilter.cs","lineNumber":43,"sourceCode":"        var summary = context.Result.ToString();\n\n        var request = new SummarizationEvaluationRequest { Sources = [sourceText], Summaries = [summary] };\n        var response = await evaluationService.EvaluateAsync<SummarizationEvaluationRequest, BleuSummarizationEvaluationResponse>(request);\n\n        var score = Math.Round(response.Score, 4);\n        var precisions = response.Precisions.Select(l => Math.Round(l, 4)).ToList();\n        var brevityPenalty = Math.Round(response.BrevityPenalty, 4);\n        var lengthRatio = Math.Round(response.LengthRatio, 4);\n\n        logger.LogInformation(\"[BLEU] Score: {Score}, Precisions: {Precisions}, Brevity penalty: {BrevityPenalty}, Length Ratio: {LengthRatio}\",\n            score,\n            string.Join(\", \", precisions),\n            brevityPenalty,\n            lengthRatio);\n\n        if (precisions[0] < threshold)\n        {\n            throw new KernelException($\"BLEU summary evaluation score ({precisions[0]}) is lower than threshold ({threshold})\");\n        }\n    }\n}\n","sourceCodeStart":25,"sourceCodeEnd":47,"githubUrl":"https://github.com/microsoft/semantic-kernel/blob/c028a0c7dc4f0814cdcbaba9d998f187a41197bf/dotnet/samples/Demos/QualityCheck/QualityCheckWithFilters/Filters/BleuSummarizationEvaluationFilter.cs#L25-L47","documentation":"A Semantic Kernel IFunctionInvocationFilter (BleuSummarizationEvaluationFilter) evaluates a generated summary against the source using the BLEU metric (n-gram precision precisions). After retrieving precisions from the evaluation service, it throws a KernelException when precisions[0] (the 1-gram precision) is below the configured threshold.","triggerScenarios":"The evaluation service returns BLEU 1-gram precision below the injected threshold, indicating low word-level overlap between the summary and the source text.","commonSituations":"Threshold configured too strictly; the model paraphrases heavily (BLEU penalizes exact n-gram matches, so paraphrase-heavy summaries score low); short source text or very short summary; mismatch between source language and summary language.","solutions":["Lower the threshold for precisions[0] in the DI registration to match the model's typical BLEU performance.","Use a model or prompt that produces more extractive (literal) summaries, which score higher on BLEU.","Switch the quality metric to one more tolerant of paraphrase (e.g., BERTScore or METEOR) if paraphrasing is desired.","Catch KernelException at the orchestration layer and implement a retry-with-feedback loop."],"exampleFix":"// before — check only 1-gram precision\nif (precisions[0] < threshold)\n{\n    throw new KernelException($\"BLEU summary evaluation score ({precisions[0]}) is lower than threshold ({threshold})\");\n}\n\n// after — use the BLEU composite score or average of precisions\nvar avgPrecision = precisions.Average();\nif (avgPrecision < threshold)\n{\n    logger.LogWarning(\"BLEU avg precision {Avg} below threshold {Threshold}.\", avgPrecision, threshold);\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { await kernel.InvokeAsync(summarizeFunc); } catch (KernelException ex) when (ex.Message.Contains(\"BLEU summary evaluation score\")) { logger.LogWarning(\"BLEU below threshold: {Msg}\", ex.Message); }","preventionTips":["Understand that BLEU penalizes paraphrasing — calibrate for your model's style.","Consider averaging all n-gram precisions instead of checking only precisions[0].","Use a retry-with-feedback loop to let the model improve its summary."],"tags":["quality-check","bleu","evaluation","kernel-exception","filter","summarization"],"backgroundTag":null,"analyzedSha":"c028a0c7dc4f0814cdcbaba9d998f187a41197bf","analyzedAt":"2026-08-13T13:48:05.040Z","schemaVersion":2},"datasetVersion":"2026-08-13T14:17:21.547Z"}