dotnet/machinelearning · critical · InvalidOperationException

Training failed with the exception: {_history.Last().Excepti

Error message

Training failed with the exception: {_history.Last().Exception}

What it means

Execute() tolerates individual failed training runs, but if the first 3 runs all fail it concludes the problem is systematic and rethrows the last run's exception wrapped in InvalidOperationException instead of returning empty results.

Source

Thrown at src/Microsoft.ML.AutoML/Experiment/Experiment.cs:191

                    _history.Add(suggestedPipelineRunDetail);
                    WriteIterationLog(pipeline, suggestedPipelineRunDetail, iterationStopwatch);

                    runDetail.RuntimeInSeconds = iterationStopwatch.Elapsed.TotalSeconds;
                    runDetail.PipelineInferenceTimeInSeconds = getPipelineStopwatch.Elapsed.TotalSeconds;

                    ReportProgress(runDetail);
                    iterationResults.Add(runDetail);

                    // if model is perfect, break
                    if (_metricsAgent.IsModelPerfect(suggestedPipelineRunDetail.Score))
                    {
                        break;
                    }

                    // If after third run, all runs have failed so far, throw exception
                    if (_history.Count() == 3 && _history.All(r => !r.RunSucceeded))
                    {
                        throw new InvalidOperationException($"Training failed with the exception: {_history.Last().Exception}");
                    }
                }
                catch (OperationCanceledException e)
                {
                    // This exception is thrown when the IHost/MLContext of the trainer is canceled due to
                    // reaching maximum experiment time. Simply catch this exception and return finished
                    // iteration results.
                    _logger.Warning(_operationCancelledMessage, e.Message);
                    return iterationResults;
                }
                catch (AggregateException e)
                {
                    // This exception is thrown when the IHost/MLContext of the trainer is canceled due to
                    // reaching maximum experiment time. Simply catch this exception and return finished
                    // iteration results. For some trainers, like FastTree, because training is done in parallel
                    // in can throw multiple OperationCancelledExceptions. This causes them to be returned as an
                    // AggregateException and misses the first catch block. This is to handle that case.
                    if (e.InnerExceptions.All(exception => exception is OperationCanceledException))

View on GitHub (pinned to 7b76e69cf9)

Solutions

1. Inspect the inner Exception in the message — fix the root trainer failure (schema, label column, data types).
2. Validate input data (IDataView schema, label column exists, enough rows) before calling Execute().
3. Use UserInputValidationUtil / CrossValidationSplit to confirm folds are non-empty.
4. Catch InvalidOperationException, read _history-style inner exception, and adjust trainer/settings.

Example fix

// before
var result = experiment.Execute(trainData, validationData, labelColumnName: "Target");
// after
if (!trainData.Schema.TryGetColumnIndex("Target", out _))
    throw new ArgumentException("Label column 'Target' missing from training data.");
var result = experiment.Execute(trainData, validationData, labelColumnName: "Target");

if (!trainData.Schema.TryGetColumnIndex(labelColumn, out _)) throw new ArgumentException($"Label '{labelColumn}' missing");
if (rowCount < 3) throw new ArgumentException("Too few rows for AutoML training");

try { var results = experiment.Execute(trainData, labelColumn); }
catch (InvalidOperationException ex) { logger.LogError(ex.InnerException ?? ex, "AutoML training failed on all trials"); throw; }

When it happens

Trigger: Calling experiment.Execute() (e.g. via experiment.Execute(trainData, ...) helpers) where three consecutive trainer runs throw — bad data schema, incompatible labels, or every trainer in the pipeline erroring.

Common situations: Wrong column roles (label/columns not set), featurization failures, dataset too small/empty for any trainer, or unsupported data types causing each trial to fail identically.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Type IPredictionTransformer not implemented by provided type
• The provided model file {filePath} doesn't exist.
• Start must be called on a ModelLoader before it can be used.
• {nameof(ChannelMessageKind)}.{e.Kind} is not yet implemented
• Metric {typeof(TMetrics)} not implemented

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/e3981a3a6b5d8495. Report an issue: GitHub.