What causes "can't lock shard: SIGTERM received" in vitessio/vitess?

Let the process exit; the recovery will be re-attempted by the restarted vtorc

can't lock shard: SIGTERM received

Error message

can't lock shard: SIGTERM received

What it means

LockShard refuses to take a shard lock once the vtorc process has received SIGTERM (shutdown in progress). This prevents new recoveries from acquiring locks during graceful shutdown when locks could never be released properly.

Solutions

Let the process exit; the recovery will be re-attempted by the restarted vtorc
If it recurs constantly, adjust recovery check intervals so recoveries start before shutdown
Ensure graceful shutdown completes quickly so the window is small

Defensive patterns

Strategy: retry

Validate before calling

if hasReceivedSIGTERM.Load() > 0 {
    return nil // do not start new recoveries during shutdown
}

Try / catch

ctx, unlock, err := logic.LockShard(ctx, ks, shard, action)
if err != nil && strings.Contains(err.Error(), "SIGTERM received") {
    log.Info("recovery skipped: shutdown in progress")
    return nil
}
if err != nil { return err }

Prevention

Skip initiating new recoveries once shutdown signals are trapped
Keep recovery intervals sized so they finish before deploys
Graceful-stop vtorc outside peak recovery windows

When it happens

Trigger: A recovery check fires in the window after SIGTERM was delivered to vtorc but before the process fully exits; executeCheckAndRecoverFunction calls LockShard and hasReceivedSIGTERM is non-zero.

Common situations: Deployments restarting vtorc while an incapacitated-primary detection is in flight; Kubernetes pod termination racing with recovery ticks; systemd restarts during active failover monitoring.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/f4e4b4e5a7bc76b9. Report an issue: GitHub.

Appendix: source

Thrown at go/vt/vtorc/logic/topology_recovery.go:249

func initializeTopologyRecoveryPostConfiguration() {
	config.WaitForConfigurationToBeLoaded()
}

func getLockAction(tabletAlias *topodatapb.TabletAlias, code inst.AnalysisCode) string {
	return fmt.Sprintf("VTOrc Recovery for %v on %v", code, topoproto.TabletAliasString(tabletAlias))
}

// LockShard locks the keyspace-shard preventing others from performing conflicting actions.
func LockShard(ctx context.Context, keyspace, shard, lockAction string) (context.Context, func(*error), error) {
	if keyspace == "" {
		return nil, nil, errors.New("can't lock shard: keyspace is unspecified")
	}
	if shard == "" {
		return nil, nil, errors.New("can't lock shard: shard name is unspecified")
	}
	if hasReceivedSIGTERM.Load() > 0 {
		return nil, nil, errors.New("can't lock shard: SIGTERM received")
	}

	startTime := time.Now()
	defer func() {
		lockTime := time.Since(startTime)
		shardLockTimings.Add("Lock", lockTime)
	}()

	shardsLockCounter.Add(1)
	ctx, unlock, err := ts.TryLockShard(ctx, keyspace, shard, lockAction)
	if err != nil {
		shardsLockCounter.Add(-1)
		return nil, nil, err
	}
	return ctx, func(e *error) {
		startTime := time.Now()
		defer func() {
			shardsLockCounter.Add(-1)

View on GitHub (pinned to 01a25a7d17)