microsoft/aspire · error · InvalidOperationException

Toolbox ' ' changed concurrently: the Toolbox was no longer…

Error message

Toolbox '{name}' changed concurrently: the Toolbox was no longer visible after promotion. No further changes were made.

What it means

FoundryToolboxReconciler.PromoteOwnedVersionAsync performs an optimistic-concurrency check after promoting a Toolbox version: it re-fetches the Toolbox via administration.GetAsync and throws this error if the Toolbox no longer exists (was deleted or renamed by someone else between promotion and verification). The reconciler makes no further changes so external state stays consistent. This is a defensive guard so Aspire never claims success on a resource that vanished mid-reconcile.

Solutions

Re-run the reconcile/deploy after confirming who deleted the Toolbox — a fresh run will recreate it since Aspire treats it as new
Check audit logs or the Foundry portal to identify the concurrent actor that removed the Toolbox and stop overlapping pipelines
Serialize reconciles for the same environment (single deploy pipeline, or a lock) so two processes cannot race
If the Toolbox was intentionally removed, remove it from the Aspire AppHost model instead of reconciling it

Example fix

// before: two pipelines deploying the same AppHost concurrently
// (pipeline A deletes stale Toolboxes while pipeline B promotes a version)

// after: ensure only one reconcile runs at a time for the environment
# .github/workflows/deploy.yml
concurrency:
  group: aspire-deploy-${{ github.ref }}
  cancel-in-progress: false

Defensive patterns

Strategy: retry

Validate before calling

// Before reconciling, confirm the Toolbox still exists and note its identity
var existing = await administration.GetAsync(definition.Name, ct);
if (existing is null) { /* recreate instead of promoting */ }

Type guard

bool ToolboxExists(FoundryToolboxState? state) => state is not null;

Try / catch

try
{
    await reconciler.ReconcileAsync(definition, ct);
}
catch (ConcurrentChangeException) // or the toolbox concurrent-change exception type
{
    // No partial changes were made; re-run the reconcile after resolving the external actor.
    await Task.Delay(TimeSpan.FromSeconds(5), ct);
    await reconciler.ReconcileAsync(definition, ct);
}

Prevention

Serialize deploys per environment with a CI concurrency group
Do not manually delete Toolboxes in the portal while a deploy is running
Re-run the deploy after any concurrent-change error; it is designed to be safe to retry

When it happens

Trigger: Calling ReconcileAsync on a FoundryToolboxDeploymentDefinition when another actor (another pipeline run, another developer, a portal operation) deletes the Toolbox in the window between the PromoteVersionAsync call and the immediately following GetAsync verification call returning null.

Common situations: Two CI pipelines reconciling the same Aspire app concurrently; someone deletes the Toolbox from the Foundry portal while a deploy is running; a cleanup job removes Toolboxes mid-promotion; stale local model being reconciled against an environment changed by a teammate.

Related errors

AI-assisted analysis of microsoft/aspire@25830f84bd (2026-09-16). Data as JSON: /api/errors/e2820061054956cd. Report an issue: GitHub.

Appendix: source

Thrown at src/Aspire.Hosting.Foundry/Toolbox/FoundryToolboxReconciler.cs:448

        if (observedDefaultVersion is null ||
            !string.Equals(before.DefaultVersion, observedDefaultVersion, StringComparison.Ordinal))
        {
            var expectedDefault = observedDefaultVersion is null
                ? "no default version"
                : $"default version '{observedDefaultVersion}'";
            throw CreateConcurrentChangeException(
                definition.Name,
                $"expected {expectedDefault}, but version '{before.DefaultVersion}' is now the default");
        }

        await administration.PromoteVersionAsync(
            definition.Name,
            version,
            cancellationToken).ConfigureAwait(false);

        var after = await administration.GetAsync(definition.Name, cancellationToken).ConfigureAwait(false)
            ?? throw CreateConcurrentChangeException(
                definition.Name,
                "the Toolbox was no longer visible after promotion");

        ValidateOwnedDefault(definition.Name, after, concurrentChange: true);
        ValidatePromotionTarget(definition, after, version);
        if (!string.Equals(after.DefaultVersion, version, StringComparison.Ordinal))
        {
            throw CreateConcurrentChangeException(
                definition.Name,
                $"version '{version}' was promoted, but version '{after.DefaultVersion}' is now the default");
        }
    }

    private static void ValidateOwnedDefault(
        string name,
        FoundryToolboxState state,
        bool concurrentChange = false)
    {

View on GitHub (pinned to 25830f84bd)