elsa-workflows/elsa-core · error · TimeoutException

Could not acquire distributed lock with key

Error message

Could not acquire distributed lock with key '{lockKey}' within the configured timeout of {distributedLockingOptions.Value.LockAcquisitionTimeout}.

What it means

WorkflowResumer.ResumeAsync wraps the entire bookmark-resume operation in a distributed lock named 'workflow-resumer:{filter}'. If the lock cannot be acquired within DistributedLockingOptions.LockAcquisitionTimeout, the underlying Medallion TimeoutException is rethrown with this message naming the lock key and configured timeout. This prevents concurrent resume of the same bookmarks by multiple workers.

Solutions

Increase DistributedLockingOptions.LockAcquisitionTimeout in app configuration to cover worst-case lock hold time.
Check which process/thread holds the lock and why it is slow (long-running workflow resume, DB contention); fix that bottleneck.
Ensure a shared, correctly configured distributed lock provider (e.g. SQL Server/Postgres/Redis) is registered; a misconfigured backend can behave like a permanently held lock.
Retry the resume after the timeout with backoff, ideally with idempotent handling since another worker may have already resumed the bookmark.

Example fix

// before
services.Configure<DistributedLockingOptions>(o => o.LockAcquisitionTimeout = TimeSpan.FromSeconds(1));
// after
services.Configure<DistributedLockingOptions>(o => o.LockAcquisitionTimeout = TimeSpan.FromSeconds(30));

Defensive patterns

Strategy: retry

Validate before calling

var opts = serviceProvider.GetRequiredService<IOptions<DistributedLockingOptions>>().Value;
if (opts.LockAcquisitionTimeout < TimeSpan.FromSeconds(5)) throw new InvalidOperationException("LockAcquisitionTimeout too low for resume operations");

Try / catch

try { await resumer.ResumeAsync(filter, options, ct); }
catch (TimeoutException) { logger.LogWarning("Resume lock '{Key}' timed out; will retry", lockKey); await Task.Delay(TimeSpan.FromSeconds(5), ct); /* retry or surface */ }

Prevention

Size LockAcquisitionTimeout above worst-case resume duration.
Use a robust distributed lock backend shared by all workers.
Monitor lock contention metrics and alert on frequent timeouts.
Make resume handling idempotent so safe retries are possible.

When it happens

Trigger: Calling any IWorkflowResumer.ResumeAsync overload (by bookmark id, by TActivity + stimulus, by BookmarkFilter, or via ResumeBookmarkRequest) while another process/thread holds the lock 'workflow-resumer:<hashable filter string>' for longer than the configured LockAcquisitionTimeout.

Common situations: Multiple Elsa workers contending for the same event-driven bookmark; a long-running resume that exceeds the timeout; a stuck/crashed holder leaving the lock until expiry; LockAcquisitionTimeout configured too low for slow environments (e.g. database latency).

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of elsa-workflows/elsa-core@fe9217bdfa (2026-09-13). Data as JSON: /api/errors/5bd6042fc9202aea. Report an issue: GitHub.

Appendix: source

Thrown at src/modules/Elsa.Workflows.Runtime/Services/WorkflowResumer.cs:132

                try
                {
                    var response = await workflowClient.RunInstanceAsync(runRequest, cancellationToken);
                    logger.LogDebug("Resumed workflow instance {WorkflowInstanceId} with bookmark {BookmarkId}", bookmark.WorkflowInstanceId, bookmark.Id);
                    responses.Add(response);
                }
                catch (WorkflowInstanceNotFoundException)
                {
                    // The workflow instance does not (yet) exist in the DB.
                    logger.LogDebug("No workflow instance with ID {WorkflowInstanceId} found for bookmark {BookmarkId} at this time.", bookmark.WorkflowInstanceId, bookmark.Id);
                }
            }

            return responses;
        }
        catch (TimeoutException e)
        {
            // Rethrow but with a more specific message.
            throw new TimeoutException($"Could not acquire distributed lock with key '{lockKey}' within the configured timeout of {distributedLockingOptions.Value.LockAcquisitionTimeout}.", e);
        }
    }
}

View on GitHub (pinned to fe9217bdfa)