fullstackhero/dotnet-starter-kit · error · CustomException

Provisioning already running for tenant

Error message

Provisioning already running for tenant {tenantId}.

What it means

TenantProvisioningService.StartAsync refuses to start a new provisioning run when the latest TenantProvisioning record for the tenant is still in Running or Pending status. Provisioning is treated as a single-flight operation per tenant, so a concurrent or stale in-flight run blocks new ones. The thrown CustomException surfaces to the caller as a 400/409-style business error naming the tenant.

Solutions

  1. Wait for the current provisioning run to finish (poll GetStatusAsync until Status is Completed or Failed) before retrying.
  2. If the record is stuck in Running/Pending from a crashed job, mark it Failed or delete/reset the TenantProvisioning row, then call StartAsync again.
  3. Add idempotency on the client: check GetStatusAsync before calling start, or serialize provisioning behind a queue.
  4. If the Hangfire job died, verify job storage (TryEnsureJobStorage) and requeue the job so the record is driven to a terminal status.

Example fix

// before
await provisioningService.StartAsync(tenantId, ct);

// after
var status = await provisioningService.GetStatusAsync(tenantId, ct);
if (status is not { Status: TenantProvisioningStatus.Running or TenantProvisioningStatus.Pending })
{
    await provisioningService.StartAsync(tenantId, ct);
}
Defensive patterns

Strategy: try-catch

Validate before calling

var latest = await provisioningService.GetStatusAsync(tenantId, ct);
if (latest.Status is TenantProvisioningStatus.Running or TenantProvisioningStatus.Pending)
    return; // already in flight

Type guard

bool CanStart(TenantProvisioningStatusDto? s) => s is null || s.Status is not (TenantProvisioningStatus.Running or TenantProvisioningStatus.Pending);

Try / catch

try
{
    await provisioningService.StartAsync(tenantId, ct);
}
catch (CustomException ex) when (ex.Message.Contains("Provisioning already running"))
{
    logger.LogInformation("Provisioning already in flight for {TenantId}", tenantId);
}

Prevention

When it happens

Trigger: Calling StartAsync (directly or via RetryAsync) while a TenantProvisioning row for tenantId exists with Status == TenantProvisioningStatus.Running or Pending. Typical when a Hangfire provisioning job is still executing, or a previous run was never marked Failed/Completed (e.g. API crashed mid-run) and no cleanup happened.

Common situations: Double-clicking a 'provision tenant' button; two admins provisioning simultaneously; a crashed or hung background job leaving the record stuck in Running; retrying before the prior run finishes; deploying a new provisioning job while the old one is pending.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of fullstackhero/dotnet-starter-kit@3f2959e683 (2026-09-15). Data as JSON: /api/errors/640c9406e231a843. Report an issue: GitHub.

Appendix: source

Thrown at src/Modules/Multitenancy/Modules.Multitenancy/Provisioning/TenantProvisioningService.cs:44

        IServiceScopeFactory scopeFactory,
        ILogger<TenantProvisioningService> logger)
    {
        _dbContext = dbContext;
        _tenantStore = tenantStore;
        _jobService = jobService;
        _scopeFactory = scopeFactory;
        _logger = logger;
    }

    public async Task<TenantProvisioning> StartAsync(string tenantId, CancellationToken cancellationToken)
    {
        var tenant = await _tenantStore.GetAsync(tenantId).ConfigureAwait(false)
            ?? throw new NotFoundException($"Tenant {tenantId} not found for provisioning.");

        var existing = await GetLatestAsync(tenantId, cancellationToken).ConfigureAwait(false);
        if (existing is not null && (existing.Status is TenantProvisioningStatus.Running or TenantProvisioningStatus.Pending))
        {
            throw new CustomException($"Provisioning already running for tenant {tenantId}.");
        }

        var correlationId = Guid.NewGuid().ToString();
        var provisioning = new TenantProvisioning(tenant.Id, correlationId);

        provisioning.Steps.Add(new TenantProvisioningStep(provisioning.Id, TenantProvisioningStepName.Database));
        provisioning.Steps.Add(new TenantProvisioningStep(provisioning.Id, TenantProvisioningStepName.Migrations));
        provisioning.Steps.Add(new TenantProvisioningStep(provisioning.Id, TenantProvisioningStepName.Seeding));
        provisioning.Steps.Add(new TenantProvisioningStep(provisioning.Id, TenantProvisioningStepName.CacheWarm));

        _dbContext.Add(provisioning);
        await _dbContext.SaveChangesAsync(cancellationToken).ConfigureAwait(false);

        if (!TryEnsureJobStorage())
        {
            _logger.LogWarning("Background job storage not available; running provisioning inline for tenant {TenantId}.", tenantId);
            provisioning.SetJobId("inline");
            await _dbContext.SaveChangesAsync(cancellationToken).ConfigureAwait(false);

View on GitHub (pinned to 3f2959e683)