{"record":{"id":"8c1d39e653803de5","repo":"microsoft/semantic-kernel","slug":"meteor-summary-evaluation-score-score-is-lower","errorCode":null,"errorMessage":"METEOR summary evaluation score ({score}) is lower than threshold ({threshold})","messagePattern":"METEOR summary evaluation score \\((.+?)\\) is lower than threshold \\((.+?)\\)","errorType":"exception","errorClass":"KernelException","httpStatus":null,"severity":"error","filePath":"dotnet/samples/Demos/QualityCheck/QualityCheckWithFilters/Filters/MeteorSummarizationEvaluationFilter.cs","lineNumber":35,"sourceCode":"    double threshold) : IFunctionInvocationFilter\n{\n    public async Task OnFunctionInvocationAsync(FunctionInvocationContext context, Func<FunctionInvocationContext, Task> next)\n    {\n        await next(context);\n\n        var sourceText = context.Result.RenderedPrompt!;\n        var summary = context.Result.ToString();\n\n        var request = new SummarizationEvaluationRequest { Sources = [sourceText], Summaries = [summary] };\n        var response = await evaluationService.EvaluateAsync<SummarizationEvaluationRequest, MeteorSummarizationEvaluationResponse>(request);\n\n        var score = Math.Round(response.Score, 4);\n\n        logger.LogInformation(\"[METEOR] Score: {Score}\", score);\n\n        if (score < threshold)\n        {\n            throw new KernelException($\"METEOR summary evaluation score ({score}) is lower than threshold ({threshold})\");\n        }\n    }\n}\n","sourceCodeStart":17,"sourceCodeEnd":39,"githubUrl":"https://github.com/microsoft/semantic-kernel/blob/c028a0c7dc4f0814cdcbaba9d998f187a41197bf/dotnet/samples/Demos/QualityCheck/QualityCheckWithFilters/Filters/MeteorSummarizationEvaluationFilter.cs#L17-L39","documentation":"A Semantic Kernel IFunctionInvocationFilter (MeteorSummarizationEvaluationFilter) evaluates a generated summary against the source text using the METEOR metric. It throws a KernelException when the returned single score is below the configured threshold. METEOR accounts for synonyms and stemming, making it more forgiving than BLEU.","triggerScenarios":"The METEOR score returned by the evaluation service is below the injected threshold, indicating the summary's quality (considering synonyms, word order, and stemming) is insufficient.","commonSituations":"Threshold too high for the model; very short or very long summaries score poorly on METEOR; the source text and summary diverge in wording or coverage; evaluation service inconsistency or timeout producing low scores.","solutions":["Lower the threshold to match the model's expected METEOR range.","Improve the summarization prompt to encourage better content coverage and conciseness.","Catch the KernelException and retry with modified temperature or prompt instructions.","Compare against a baseline run to calibrate a realistic threshold before enforcing it."],"exampleFix":"// before\nif (score < threshold)\n{\n    throw new KernelException($\"METEOR summary evaluation score ({score}) is lower than threshold ({threshold})\");\n}\n\n// after — warning-level logging with continued execution\nif (score < threshold)\n{\n    logger.LogWarning(\"METEOR score {Score} below threshold {Threshold}.\", score, threshold);\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { await kernel.InvokeAsync(summarizeFunc); } catch (KernelException ex) when (ex.Message.Contains(\"METEOR summary evaluation score\")) { logger.LogWarning(\"METEOR below threshold: {Msg}\", ex.Message); }","preventionTips":["Run a baseline to understand the model's typical METEOR range before setting thresholds.","METEOR is more lenient than BLEU but still requires reasonable content overlap.","Consider combining multiple metrics rather than relying on a single threshold."],"tags":["quality-check","meteor","evaluation","kernel-exception","filter","summarization"],"backgroundTag":null,"analyzedSha":"c028a0c7dc4f0814cdcbaba9d998f187a41197bf","analyzedAt":"2026-08-13T13:48:05.040Z","schemaVersion":2},"datasetVersion":"2026-08-13T14:17:21.547Z"}