{"record":{"id":"f4e4b4e5a7bc76b9","repo":"vitessio/vitess","slug":"can-t-lock-shard-sigterm-received","errorCode":null,"errorMessage":"can't lock shard: SIGTERM received","messagePattern":"can't lock shard: SIGTERM received","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"go/vt/vtorc/logic/topology_recovery.go","lineNumber":249,"sourceCode":"\nfunc initializeTopologyRecoveryPostConfiguration() {\n\tconfig.WaitForConfigurationToBeLoaded()\n}\n\nfunc getLockAction(tabletAlias *topodatapb.TabletAlias, code inst.AnalysisCode) string {\n\treturn fmt.Sprintf(\"VTOrc Recovery for %v on %v\", code, topoproto.TabletAliasString(tabletAlias))\n}\n\n// LockShard locks the keyspace-shard preventing others from performing conflicting actions.\nfunc LockShard(ctx context.Context, keyspace, shard, lockAction string) (context.Context, func(*error), error) {\n\tif keyspace == \"\" {\n\t\treturn nil, nil, errors.New(\"can't lock shard: keyspace is unspecified\")\n\t}\n\tif shard == \"\" {\n\t\treturn nil, nil, errors.New(\"can't lock shard: shard name is unspecified\")\n\t}\n\tif hasReceivedSIGTERM.Load() > 0 {\n\t\treturn nil, nil, errors.New(\"can't lock shard: SIGTERM received\")\n\t}\n\n\tstartTime := time.Now()\n\tdefer func() {\n\t\tlockTime := time.Since(startTime)\n\t\tshardLockTimings.Add(\"Lock\", lockTime)\n\t}()\n\n\tshardsLockCounter.Add(1)\n\tctx, unlock, err := ts.TryLockShard(ctx, keyspace, shard, lockAction)\n\tif err != nil {\n\t\tshardsLockCounter.Add(-1)\n\t\treturn nil, nil, err\n\t}\n\treturn ctx, func(e *error) {\n\t\tstartTime := time.Now()\n\t\tdefer func() {\n\t\t\tshardsLockCounter.Add(-1)","sourceCodeStart":231,"sourceCodeEnd":267,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/vtorc/logic/topology_recovery.go#L231-L267","documentation":"LockShard refuses to take a shard lock once the vtorc process has received SIGTERM (shutdown in progress). This prevents new recoveries from acquiring locks during graceful shutdown when locks could never be released properly.","triggerScenarios":"A recovery check fires in the window after SIGTERM was delivered to vtorc but before the process fully exits; executeCheckAndRecoverFunction calls LockShard and hasReceivedSIGTERM is non-zero.","commonSituations":"Deployments restarting vtorc while an incapacitated-primary detection is in flight; Kubernetes pod termination racing with recovery ticks; systemd restarts during active failover monitoring.","solutions":["Let the process exit; the recovery will be re-attempted by the restarted vtorc","If it recurs constantly, adjust recovery check intervals so recoveries start before shutdown","Ensure graceful shutdown completes quickly so the window is small"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"if hasReceivedSIGTERM.Load() > 0 {\n    return nil // do not start new recoveries during shutdown\n}","typeGuard":null,"tryCatchPattern":"ctx, unlock, err := logic.LockShard(ctx, ks, shard, action)\nif err != nil && strings.Contains(err.Error(), \"SIGTERM received\") {\n    log.Info(\"recovery skipped: shutdown in progress\")\n    return nil\n}\nif err != nil { return err }","preventionTips":["Skip initiating new recoveries once shutdown signals are trapped","Keep recovery intervals sized so they finish before deploys","Graceful-stop vtorc outside peak recovery windows"],"tags":["vtorc","shutdown","sigterm","shard-lock"],"backgroundTag":"shutdown-in-progress","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}