HangfireIO/Hangfire · warning · DistributedLockTimeoutException

Timeout expired. The timeout elapsed prior to obtaining a di

Error message

Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the '{resource}' resource.

What it means

Hangfire exhausted the configured timeout while polling for a SQL Server application lock because another process continuously held it. The exception is DistributedLockTimeoutException (derived from TimeoutException) and is expected under lock contention. Hangfire itself catches this in several internal services (ExpirationManager, DelayedJobScheduler, RecurringJobScheduler) and treats it as non-fatal.

Source

Thrown at src/Hangfire.SqlServer/SqlServerDistributedLock.cs:220

                command.ExecuteNonQuery();

                var lockResult = (int)resultParameter.Value;

                if (lockResult >= 0)
                {
                    // The lock has been successfully obtained on the specified resource.
                    return;
                }

                if (lockResult == -999 /* Indicates a parameter validation or other call error. */)
                {
                    throw new SqlServerDistributedLockException(
                        $"Could not place a lock on the resource '{resource}': {(LockErrorMessages.TryGetValue(lockResult, out var message) ? message : $"Server returned the '{lockResult}' error.")}.");
                }
            } while (started.Elapsed < timeout);

            throw new DistributedLockTimeoutException(resource);
        }

        internal static void Release(DbConnection connection, string resource)
        {
            using (var command = CreateReleaseCommand(connection, resource, out var resultParameter))
            {
                command.ExecuteNonQuery();

                var releaseResult = (int)resultParameter.Value;

                if (releaseResult < 0)
                {
                    throw new SqlServerDistributedLockException(
                        $"Could not release a lock on the resource '{resource}': Server returned the '{releaseResult}' error.");
                }
            }
        }

View on GitHub (pinned to c236dd0f93)

Solutions

Reduce the number of Hangfire server instances competing for the same database, or shard across multiple databases.
Optimize the Hangfire schema: run schema migrations, enable UseIgnoreDupKeyOption, increase DeleteExpiredBatchSize to speed up expiration sweeps.
Monitor SQL Server for blocking sessions and long-running queries during the timeout window.
If calling distributed-lock APIs directly, catch DistributedLockTimeoutException and retry with backoff or skip gracefully as Hangfire's own services do.

Example fix

// before — unhandled timeout propagates and crashes the operation
using (storage.GetConnection().AcquireDistributedLock("my-lock", TimeSpan.FromSeconds(5)))
{
    DoWork();
}
// after — catch and retry/skip gracefully
try
{
    using (storage.GetConnection().AcquireDistributedLock("my-lock", TimeSpan.FromSeconds(5)))
    {
        DoWork();
    }
}
catch (DistributedLockTimeoutException)
{
    // Lock is held by another instance; skip this cycle.
}

Defensive patterns

Strategy: retry

Try / catch

int attempts = 0;
while (true)
{
    try
    {
        using (storage.GetConnection().AcquireDistributedLock(resource, TimeSpan.FromSeconds(5)))
        {
            DoWork();
            return;
        }
    }
    catch (DistributedLockTimeoutException) when (attempts++ < maxRetries)
    {
        Thread.Sleep(TimeSpan.FromSeconds(backoffSeconds));
    }
}

Prevention

Limit the number of Hangfire server instances sharing a single database to reduce lock contention.
Keep expiration and aggregation batches fast: enable schema optimizations and tune batch sizes.
Treat DistributedLockTimeoutException as recoverable — Hangfire's own services (ExpirationManager, RecurringJobScheduler, DelayedJobScheduler) already swallow it internally.
Monitor SQL Server for blocking chains during peak load.

When it happens

Trigger: Multiple Hangfire server instances or background workers competing for the same lock resource simultaneously; a long-running maintenance query (large expiration batch, heavy counters aggregation) holding the lock longer than the timeout; database under heavy load causing each sp_getapplock attempt (1-second LockTimeout per iteration) to fail within the overall timeout window.

Common situations: Scaling out to many server instances all hitting the same Hangfire database; large job tables making expiration sweeps slow; SQL Azure throttling or resource governor connection termination; deadlocks on the lock resource during peak load.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of HangfireIO/Hangfire@c236dd0f93 (2026-08-13). Data as JSON: /api/errors/13b1a4030a5fc09e. Report an issue: GitHub.