dotnet/machinelearning · critical · InvalidOperationException

Training failed with the exception

Error message

Training failed with the exception: {_history.Last().Exception}

What it means

Execute() tolerates individual failed training runs, but if the first 3 runs all fail it concludes the problem is systematic and rethrows the last run's exception wrapped in InvalidOperationException instead of returning empty results.

Solutions

1. Inspect the inner Exception in the message — fix the root trainer failure (schema, label column, data types).
2. Validate input data (IDataView schema, label column exists, enough rows) before calling Execute().
3. Use UserInputValidationUtil / CrossValidationSplit to confirm folds are non-empty.
4. Catch InvalidOperationException, read _history-style inner exception, and adjust trainer/settings.

Example fix

// before
var result = experiment.Execute(trainData, validationData, labelColumnName: "Target");
// after
if (!trainData.Schema.TryGetColumnIndex("Target", out _))
    throw new ArgumentException("Label column 'Target' missing from training data.");
var result = experiment.Execute(trainData, validationData, labelColumnName: "Target");

if (!trainData.Schema.TryGetColumnIndex(labelColumn, out _)) throw new ArgumentException($"Label '{labelColumn}' missing");
if (rowCount < 3) throw new ArgumentException("Too few rows for AutoML training");

try { var results = experiment.Execute(trainData, labelColumn); }
catch (InvalidOperationException ex) { logger.LogError(ex.InnerException ?? ex, "AutoML training failed on all trials"); throw; }

When it happens

Trigger: Calling experiment.Execute() (e.g. via experiment.Execute(trainData, ...) helpers) where three consecutive trainer runs throw — bad data schema, incompatible labels, or every trainer in the pipeline erroring.

Common situations: Wrong column roles (label/columns not set), featurization failures, dataset too small/empty for any trainer, or unsupported data types causing each trial to fail identically.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Type IPredictionTransformer not implemented by provided type
• A PrimitiveDataViewType cannot have a disposable RawType
• Activation function not supported.
• All cross validation folds have empty train or test data…
• argument should not be null.

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/e3981a3a6b5d8495. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.AutoML/Experiment/Experiment.cs:191

                    _history.Add(suggestedPipelineRunDetail);
                    WriteIterationLog(pipeline, suggestedPipelineRunDetail, iterationStopwatch);

                    runDetail.RuntimeInSeconds = iterationStopwatch.Elapsed.TotalSeconds;
                    runDetail.PipelineInferenceTimeInSeconds = getPipelineStopwatch.Elapsed.TotalSeconds;

                    ReportProgress(runDetail);
                    iterationResults.Add(runDetail);

                    // if model is perfect, break
                    if (_metricsAgent.IsModelPerfect(suggestedPipelineRunDetail.Score))
                    {
                        break;
                    }

                    // If after third run, all runs have failed so far, throw exception
                    if (_history.Count() == 3 && _history.All(r => !r.RunSucceeded))
                    {
                        throw new InvalidOperationException($"Training failed with the exception: {_history.Last().Exception}");
                    }
                }
                catch (OperationCanceledException e)
                {
                    // This exception is thrown when the IHost/MLContext of the trainer is canceled due to
                    // reaching maximum experiment time. Simply catch this exception and return finished
                    // iteration results.
                    _logger.Warning(_operationCancelledMessage, e.Message);
                    return iterationResults;
                }
                catch (AggregateException e)
                {
                    // This exception is thrown when the IHost/MLContext of the trainer is canceled due to
                    // reaching maximum experiment time. Simply catch this exception and return finished
                    // iteration results. For some trainers, like FastTree, because training is done in parallel
                    // in can throw multiple OperationCancelledExceptions. This causes them to be returned as an
                    // AggregateException and misses the first catch block. This is to handle that case.
                    if (e.InnerExceptions.All(exception => exception is OperationCanceledException))

View on GitHub (pinned to 7b76e69cf9)