dotnet/machinelearning · error · NotImplementedException

Metric {typeof(TMetrics)} not implemented

Error message

Metric {typeof(TMetrics)} not implemented

What it means

GetAverageMetrics averages per-fold TMetrics across cross-validation runs but only implements aggregation for a fixed set of metric types (binary/classification/regression/ranking). Requesting any other TMetrics type falls through to NotImplementedException.

Source

Thrown at src/Microsoft.ML.AutoML/Experiment/Runners/CrossValSummaryRunner.cs:157

                    l2: GetAverageOfNonNaNScores(newMetrics.Select(x => x.MeanSquaredError)),
                    rms: GetAverageOfNonNaNScores(newMetrics.Select(x => x.RootMeanSquaredError)),
                    lossFunction: GetAverageOfNonNaNScores(newMetrics.Select(x => x.LossFunction)),
                    rSquared: GetAverageOfNonNaNScores(newMetrics.Select(x => x.RSquared)));
                return result as TMetrics;
            }

            if (typeof(TMetrics) == typeof(RankingMetrics))
            {
                var newMetrics = metrics.Select(x => x as RankingMetrics);
                Contracts.Assert(newMetrics != null);

                var result = new RankingMetrics(
                    dcg: GetAverageOfNonNaNScoresInNestedEnumerable(newMetrics.Select(x => x.DiscountedCumulativeGains)),
                    ndcg: GetAverageOfNonNaNScoresInNestedEnumerable(newMetrics.Select(x => x.NormalizedDiscountedCumulativeGains)));
                return result as TMetrics;
            }

            throw new NotImplementedException($"Metric {typeof(TMetrics)} not implemented");
        }

        private static double[] GetAverageOfNonNaNScoresInNestedEnumerable(IEnumerable<IEnumerable<double>> results)
        {
            if (results.All(result => result == null))
            {
                // If all nested enumerables are null, we say the average is a null enumerable as well.
                // This is expected to happen on Multiclass metrics where the TopKAccuracyForAllK
                // array can be null if the topKPredictionCount isn't a valid number.
                // In that case all of the "results" enumerables will be null anyway, and so
                // returning null is the expected solution.
                return null;
            }

            // In case there are only some null elements, we'll ignore them:
            results = results.Where(result => result != null);

            double[] arr = new double[results.ElementAt(0).Count()];

View on GitHub (pinned to 7b76e69cf9)

Solutions

  1. Use one of the supported TMetrics types (BinaryClassificationMetrics, MulticlassClassificationMetrics, RegressionMetrics, RankingMetrics).
  2. Update the library / use a task-specific experiment class whose metrics the runner supports.
  3. If implementing a new metrics type, add an aggregation branch in GetAverageMetrics.

Example fix

// before
var exp = mlContext.Auto().CreateExperiment().SetCrossValSummaryRunner<MyCustomMetrics>(...);
// after
var exp = mlContext.Auto().CreateExperiment().SetCrossValSummaryRunner<RegressionMetrics>(...);
Defensive patterns

Strategy: type-guard

Validate before calling

if (!typeof(IMetrics).IsAssignableFrom(typeof(TMetrics)) || typeof(TMetrics) == typeof(IMetrics))
    throw new NotSupportedException($"{typeof(TMetrics)} aggregation unsupported");

Type guard

bool IsSupportedMetrics<T>() => typeof(T) == typeof(BinaryClassificationMetrics) || typeof(T) == typeof(MulticlassClassificationMetrics) || typeof(T) == typeof(RegressionMetrics) || typeof(T) == typeof(RankingMetrics);

Try / catch

try { avg = runner.GetAverageMetrics(); }
catch (NotImplementedException ex) { logger.LogError(ex, "Unsupported metric aggregation for {Type}", typeof(TMetrics)); }

Prevention

When it happens

Trigger: Running cross-validation experiments with a TMetrics type not covered by the type-switch (e.g. a custom IMetrics implementation or an unsupported metrics class) so the final `throw new NotImplementedException` is reached.

Common situations: Using a newer or custom metrics type with CrossValSummaryRunner after adding a new task/metric family without extending the runner.

Related errors

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/323c5f29812fcee3. Report an issue: GitHub.