microsoft/semantic-kernel · error · KernelException

COMET translation evaluation score ({score}) is lower than t

Error message

COMET translation evaluation score ({score}) is lower than threshold ({threshold})

What it means

A Semantic Kernel IFunctionInvocationFilter (CometTranslationEvaluationFilter) sends source text and its translation to a COMET evaluation service, then throws a KernelException when the returned score falls below the configured threshold. COMET measures translation quality using learned cross-lingual representations.

Source

Thrown at dotnet/samples/Demos/QualityCheck/QualityCheckWithFilters/Filters/CometTranslationEvaluationFilter.cs:37

    public async Task OnFunctionInvocationAsync(FunctionInvocationContext context, Func<FunctionInvocationContext, Task> next)
    {
        await next(context);

        var sourceText = context.Result.RenderedPrompt!;
        var translation = context.Result.ToString();

        logger.LogInformation("Translation: {Translation}", translation);

        var request = new TranslationEvaluationRequest { Sources = [sourceText], Translations = [translation] };
        var response = await evaluationService.EvaluateAsync<TranslationEvaluationRequest, CometTranslationEvaluationResponse>(request);

        var score = Math.Round(response.Scores[0], 4);

        logger.LogInformation("[COMET] Score: {Score}", score);

        if (score < threshold)
        {
            throw new KernelException($"COMET translation evaluation score ({score}) is lower than threshold ({threshold})");
        }
    }
}

View on GitHub (pinned to c028a0c7dc)

Solutions

1. Lower the threshold to a realistic value for the translation model and language pair.
2. Use a stronger translation model or add few-shot examples to improve translation quality.
3. Catch the KernelException and retry translation with an alternative prompt or model.
4. Ensure the evaluation service endpoint and COMET model version match what the threshold was calibrated against.

Example fix

// before
if (score < threshold)
{
    throw new KernelException($"COMET translation evaluation score ({score}) is lower than threshold ({threshold})");
}

// after — log warning and allow caller to decide
if (score < threshold)
{
    logger.LogWarning("COMET score {Score} below threshold {Threshold} for translation.", score, threshold);
}

try { await kernel.InvokeAsync(translateFunc); } catch (KernelException ex) when (ex.Message.Contains("COMET translation evaluation score")) { logger.LogWarning("COMET below threshold: {Msg}", ex.Message); }

When it happens

Trigger: The COMET score from the evaluation service is below the injected threshold, meaning the translation quality is judged insufficient by the neural metric.

Common situations: Threshold set too high for the translation model's quality; using a low-capability translation model or an underpowered LLM; source/translation language pair is challenging; the COMET model on the evaluation server differs from what the threshold was calibrated against.

Related errors

• BERT summary evaluation score ({f1}) is lower than threshold
• BLEU summary evaluation score ({precisions[0]}) is lower tha
• METEOR summary evaluation score ({score}) is lower than thre
• Response is not available.
• Attribute '{node.func.attr}' is not callable in filter expre

AI-assisted analysis of microsoft/semantic-kernel@c028a0c7dc (2026-08-13). Data as JSON: /api/errors/ca047e4941d0eee1. Report an issue: GitHub.