{"record":{"id":"a075eab8aa416c1c","repo":"dotnet/machinelearning","slug":"training-time-finished-without-completing-a-succes","errorCode":null,"errorMessage":"Training time finished without completing a successful trial. Either no trial completed or the metric for all completed trials are NaN or Infinity","messagePattern":"Training time finished without completing a successful trial\\. Either no trial completed or the metric for all completed trials are NaN or Infinity","errorType":"exception","errorClass":"TimeoutException","httpStatus":null,"severity":"critical","filePath":"src/Microsoft.ML.AutoML/AutoMLExperiment/AutoMLExperiment.cs","lineNumber":345,"sourceCode":"                    }\n                    continue;\n                }\n                catch (Exception) when (aggregateTrainingStopManager.IsStopTrainingRequested())\n                {\n                    logger.Trace($\"trial cancelled - {JsonSerializer.Serialize(trialSettings)}, stop training\");\n\n                    break;\n                }\n                finally\n                {\n                    aggregateTrainingStopManager.OnStopTraining -= handler;\n                }\n            }\n\n            trialResultManager?.Save();\n            if (_bestTrialResult == null)\n            {\n                throw new TimeoutException(\"Training time finished without completing a successful trial. Either no trial completed or the metric for all completed trials are NaN or Infinity\");\n            }\n            else\n            {\n                return await Task.FromResult(_bestTrialResult);\n            }\n        }\n\n        private void ValidateSettings()\n        {\n            Contracts.Assert(_settings.MaxExperimentTimeInSeconds > 0, $\"{nameof(ExperimentSettings.MaxExperimentTimeInSeconds)} must be larger than 0\");\n        }\n\n\n        public class AutoMLExperimentSettings : ExperimentSettings\n        {\n            public int? Seed { get; set; }\n\n            public SearchSpace.SearchSpace SearchSpace { get; set; }","sourceCodeStart":327,"sourceCodeEnd":363,"githubUrl":"https://github.com/dotnet/machinelearning/blob/7b76e69cf964daeca3f1377af6bc5543284d56c6/src/Microsoft.ML.AutoML/AutoMLExperiment/AutoMLExperiment.cs#L327-L363","documentation":"At the end of AutoMLExperiment.RunAsync, after the training budget is exhausted, trial results are saved and _bestTrialResult is checked. If it is null - meaning no trial completed successfully or every completed trial's metric was NaN/Infinity - a TimeoutException is thrown. It reports that the whole training window produced no usable model.","triggerScenarios":"RunAsync finishing its allotted time with zero completed trials (too-short budget, all trials failed, cancellation storms) or with all completed trials returning NaN/Infinity metrics (bad data, constant labels, invalid metric choice, featurization failures).","commonSituations":"Setting ExperimentTime too short for even one trial; datasets with NaN labels or degenerate columns; metric functions returning Infinity due to division by zero; all trainers erroring out and only failed trials recorded.","solutions":["Increase the experiment time budget (ExperimentSetTimeout / ExperimentTime) so at least one trial can complete.","Inspect the log's failed-trial reasons (ReportFailTrial) and fix data issues: drop rows with NaN labels, fix schema/column types.","Verify the chosen metric is valid for the data and returns finite values; switch to a more robust metric (e.g. MicroAccuracy).","Reduce trial search space (fewer trainers/features) so individual trials are cheaper.","Call the trialResultManager's saved results to diagnose which trials ran and their metric values."],"exampleFix":"// before\nvar experiment = mlContext.Auto().CreateExperiment();\nexperiment.SetTrainingTime(TimeSpan.FromSeconds(10)); // too short for any trial\nvar best = await experiment.RunAsync();\n// after\nexperiment.SetTrainingTime(TimeSpan.FromMinutes(30));\nvar best = await experiment.RunAsync(); // at least one trial completes","handlingStrategy":"try-catch","validationCode":"// preflight: ensure enough budget and clean data\nif (experimentTime < TimeSpan.FromMinutes(1)) throw new ArgumentException(\"Increase experiment time budget\");\nif (data.GetColumn<float>(labelColumn).Any(float.IsNaN)) throw new ArgumentException(\"Label column contains NaN values\");","typeGuard":null,"tryCatchPattern":"try { var best = await experiment.RunAsync(); }\ncatch (TimeoutException ex) when (ex.Message.Contains(\"successful trial\")) { // no usable model in budget: clean data, raise budget, or switch metric, then retry once\n    experiment.SetTrainingTime(TimeSpan.FromMinutes(60));\n    var best = await experiment.RunAsync(); }","preventionTips":["Give the experiment a time budget comfortably larger than one trial's cost","Scrub NaN/Infinity values from label and feature columns before training","Review failed-trial logs to remove trainers that always error from the search space","Choose a metric that yields finite values for your data distribution"],"tags":["timeout","no-successful-trial","nan","automl","training-budget"],"backgroundTag":"request-timeout","analyzedSha":"7b76e69cf964daeca3f1377af6bc5543284d56c6","analyzedAt":"2026-09-11T12:35:38.930Z","contentChangedAt":"2026-09-11T12:35:38.930Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}