SigNoz/signoz · error

CodeTimeout

CodeTimeout

Error message

timed out waiting for lock

What it means

Before running or rolling back migrations, the migrator polls an advisory lock every Lock.Interval until Lock.Timeout elapses. If the lock is still held (or the lock call keeps failing), the operation aborts with this timeout error and a log entry 'cannot acquire lock'.

Source

Thrown at pkg/sqlmigrator/migrator.go:128

	return nil
}

func (migrator *migrator) Lock(ctx context.Context) error {
	if err := migrator.migrator.Lock(ctx); err == nil {
		migrator.settings.Logger().InfoContext(ctx, "acquired migration lock", slog.String("dialect", migrator.dialect))
		return nil
	}

	timer := time.NewTimer(migrator.config.Lock.Timeout)
	defer timer.Stop()

	ticker := time.NewTicker(migrator.config.Lock.Interval)
	defer ticker.Stop()

	for {
		select {
		case <-timer.C:
			err := errors.New(errors.TypeTimeout, errors.CodeTimeout, "timed out waiting for lock")
			migrator.settings.Logger().ErrorContext(ctx, "cannot acquire lock", errors.Attr(err), slog.String("lock_timeout", migrator.config.Lock.Timeout.String()), slog.String("dialect", migrator.dialect))
			return err
		case <-ticker.C:
			var err error
			if err = migrator.migrator.Lock(ctx); err == nil {
				migrator.settings.Logger().InfoContext(ctx, "acquired migration lock", slog.String("dialect", migrator.dialect))
				return nil
			}
			migrator.settings.Logger().ErrorContext(ctx, "attempt to acquire lock failed", errors.Attr(err), slog.String("lock_interval", migrator.config.Lock.Interval.String()), slog.String("dialect", migrator.dialect))
		case <-ctx.Done():
			return ctx.Err()
		}
	}
}

View on GitHub (pinned to 5069bf80b0)

Solutions

1. Increase Lock.Timeout so it exceeds the expected longest migration
2. Ensure only one migration job runs at a time (deploy serialization / leader election)
3. If a stale lock is suspected, clear the advisory lock (e.g. pg_advisory_unlock or terminate the holding session) after confirming no active migration
4. Check the error log for the underlying lock-call failures (dialect, permissions)

Example fix

// before
cfg.Lock.Timeout = 10 * time.Second

// after
cfg.Lock.Timeout = 10 * time.Minute

// pre-flight: check no other migration holds the lock
// and ensure timeout comfortably exceeds longest migration
if cfg.Lock.Timeout <= longestExpectedMigration {
    cfg.Lock.Timeout = 10 * time.Minute
}

err := m.Migrate(ctx)
if err != nil && strings.Contains(err.Error(), "timed out waiting for lock") {
    // backoff, verify no active migration, then retry once
}

When it happens

Trigger: Concurrent deployments/migration jobs where one process holds the migration lock longer than the configured timeout, or a stale lock left by a crashed process, or the database lock call erroring (permissions, connectivity) each tick.

Common situations: Two CI/CD pipelines deploying simultaneously, a long-running migration blocking others, a previous migration process killed without releasing the lock, or Lock.Timeout configured too low.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• ErrCodeInvalidSQLMigratorConfig
• CodeInvalidInput
• ErrCodeIncorrectPassword
• CodeTooManyRequests
• CodeInvalidInput

AI-assisted analysis of SigNoz/signoz@5069bf80b0 (2026-08-28). Data as JSON: /api/errors/5e865cfab415bc13. Report an issue: GitHub.