dotnet/machinelearning · critical · TimeoutException

Training time finished without completing a successful…

Error message

Training time finished without completing a successful trial. Either no trial completed or the metric for all completed trials are NaN or Infinity

What it means

At the end of AutoMLExperiment.RunAsync, after the training budget is exhausted, trial results are saved and _bestTrialResult is checked. If it is null - meaning no trial completed successfully or every completed trial's metric was NaN/Infinity - a TimeoutException is thrown. It reports that the whole training window produced no usable model.

Solutions

1. Increase the experiment time budget (ExperimentSetTimeout / ExperimentTime) so at least one trial can complete.
2. Inspect the log's failed-trial reasons (ReportFailTrial) and fix data issues: drop rows with NaN labels, fix schema/column types.
3. Verify the chosen metric is valid for the data and returns finite values; switch to a more robust metric (e.g. MicroAccuracy).
4. Reduce trial search space (fewer trainers/features) so individual trials are cheaper.
5. Call the trialResultManager's saved results to diagnose which trials ran and their metric values.

Example fix

// before
var experiment = mlContext.Auto().CreateExperiment();
experiment.SetTrainingTime(TimeSpan.FromSeconds(10)); // too short for any trial
var best = await experiment.RunAsync();
// after
experiment.SetTrainingTime(TimeSpan.FromMinutes(30));
var best = await experiment.RunAsync(); // at least one trial completes

// preflight: ensure enough budget and clean data
if (experimentTime < TimeSpan.FromMinutes(1)) throw new ArgumentException("Increase experiment time budget");
if (data.GetColumn<float>(labelColumn).Any(float.IsNaN)) throw new ArgumentException("Label column contains NaN values");

try { var best = await experiment.RunAsync(); }
catch (TimeoutException ex) when (ex.Message.Contains("successful trial")) { // no usable model in budget: clean data, raise budget, or switch metric, then retry once
    experiment.SetTrainingTime(TimeSpan.FromMinutes(60));
    var best = await experiment.RunAsync(); }

When it happens

Trigger: RunAsync finishing its allotted time with zero completed trials (too-short budget, all trials failed, cancellation storms) or with all completed trials returning NaN/Infinity metrics (bad data, constant labels, invalid metric choice, featurization failures).

Common situations: Setting ExperimentTime too short for even one trial; datasets with NaN labels or degenerate columns; metric functions returning Infinity due to division by zero; all trainers erroring out and only failed trials recorded.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

• column must be of data type
• Duplicate column name
• File at path ' ' cannot be empty
• File ' ' does not exist
• If provided, must be the same as for Ranking Experiments

AI-assisted analysis of dotnet/machinelearning@7b76e69cf9 (2026-09-11). Data as JSON: /api/errors/a075eab8aa416c1c. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.ML.AutoML/AutoMLExperiment/AutoMLExperiment.cs:345

                    }
                    continue;
                }
                catch (Exception) when (aggregateTrainingStopManager.IsStopTrainingRequested())
                {
                    logger.Trace($"trial cancelled - {JsonSerializer.Serialize(trialSettings)}, stop training");

                    break;
                }
                finally
                {
                    aggregateTrainingStopManager.OnStopTraining -= handler;
                }
            }

            trialResultManager?.Save();
            if (_bestTrialResult == null)
            {
                throw new TimeoutException("Training time finished without completing a successful trial. Either no trial completed or the metric for all completed trials are NaN or Infinity");
            }
            else
            {
                return await Task.FromResult(_bestTrialResult);
            }
        }

        private void ValidateSettings()
        {
            Contracts.Assert(_settings.MaxExperimentTimeInSeconds > 0, $"{nameof(ExperimentSettings.MaxExperimentTimeInSeconds)} must be larger than 0");
        }


        public class AutoMLExperimentSettings : ExperimentSettings
        {
            public int? Seed { get; set; }

            public SearchSpace.SearchSpace SearchSpace { get; set; }

View on GitHub (pinned to 7b76e69cf9)