{"record":{"id":"ca047e4941d0eee1","repo":"microsoft/semantic-kernel","slug":"comet-translation-evaluation-score-score-is-lo","errorCode":null,"errorMessage":"COMET translation evaluation score ({score}) is lower than threshold ({threshold})","messagePattern":"COMET translation evaluation score \\((.+?)\\) is lower than threshold \\((.+?)\\)","errorType":"exception","errorClass":"KernelException","httpStatus":null,"severity":"error","filePath":"dotnet/samples/Demos/QualityCheck/QualityCheckWithFilters/Filters/CometTranslationEvaluationFilter.cs","lineNumber":37,"sourceCode":"    public async Task OnFunctionInvocationAsync(FunctionInvocationContext context, Func<FunctionInvocationContext, Task> next)\n    {\n        await next(context);\n\n        var sourceText = context.Result.RenderedPrompt!;\n        var translation = context.Result.ToString();\n\n        logger.LogInformation(\"Translation: {Translation}\", translation);\n\n        var request = new TranslationEvaluationRequest { Sources = [sourceText], Translations = [translation] };\n        var response = await evaluationService.EvaluateAsync<TranslationEvaluationRequest, CometTranslationEvaluationResponse>(request);\n\n        var score = Math.Round(response.Scores[0], 4);\n\n        logger.LogInformation(\"[COMET] Score: {Score}\", score);\n\n        if (score < threshold)\n        {\n            throw new KernelException($\"COMET translation evaluation score ({score}) is lower than threshold ({threshold})\");\n        }\n    }\n}\n","sourceCodeStart":19,"sourceCodeEnd":41,"githubUrl":"https://github.com/microsoft/semantic-kernel/blob/c028a0c7dc4f0814cdcbaba9d998f187a41197bf/dotnet/samples/Demos/QualityCheck/QualityCheckWithFilters/Filters/CometTranslationEvaluationFilter.cs#L19-L41","documentation":"A Semantic Kernel IFunctionInvocationFilter (CometTranslationEvaluationFilter) sends source text and its translation to a COMET evaluation service, then throws a KernelException when the returned score falls below the configured threshold. COMET measures translation quality using learned cross-lingual representations.","triggerScenarios":"The COMET score from the evaluation service is below the injected threshold, meaning the translation quality is judged insufficient by the neural metric.","commonSituations":"Threshold set too high for the translation model's quality; using a low-capability translation model or an underpowered LLM; source/translation language pair is challenging; the COMET model on the evaluation server differs from what the threshold was calibrated against.","solutions":["Lower the threshold to a realistic value for the translation model and language pair.","Use a stronger translation model or add few-shot examples to improve translation quality.","Catch the KernelException and retry translation with an alternative prompt or model.","Ensure the evaluation service endpoint and COMET model version match what the threshold was calibrated against."],"exampleFix":"// before\nif (score < threshold)\n{\n    throw new KernelException($\"COMET translation evaluation score ({score}) is lower than threshold ({threshold})\");\n}\n\n// after — log warning and allow caller to decide\nif (score < threshold)\n{\n    logger.LogWarning(\"COMET score {Score} below threshold {Threshold} for translation.\", score, threshold);\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { await kernel.InvokeAsync(translateFunc); } catch (KernelException ex) when (ex.Message.Contains(\"COMET translation evaluation score\")) { logger.LogWarning(\"COMET below threshold: {Msg}\", ex.Message); }","preventionTips":["Calibrate the COMET threshold per language pair.","Ensure the evaluation server's COMET model version matches the threshold's calibration.","Log translations alongside scores for post-hoc analysis."],"tags":["quality-check","comet","evaluation","kernel-exception","filter","translation"],"backgroundTag":null,"analyzedSha":"c028a0c7dc4f0814cdcbaba9d998f187a41197bf","analyzedAt":"2026-08-13T13:48:05.040Z","schemaVersion":2},"datasetVersion":"2026-08-13T14:17:21.547Z"}