{"record":{"id":"5ae119305b1cee11","repo":"k3s-io/k3s","slug":"snapshot-save-already-in-progress","errorCode":null,"errorMessage":"snapshot save already in progress","messagePattern":"snapshot save already in progress","errorType":"exception","errorClass":null,"httpStatus":500,"severity":"warning","filePath":"pkg/etcd/snapshot.go","lineNumber":226,"sourceCode":"// subcommand for prune that can be run manually if the user wants to remove old snapshots.\n// Returns metadata about the new and pruned snapshots.\nfunc (e *ETCD) Snapshot(ctx context.Context) (*managed.SnapshotResult, error) {\n\tres, err := e.snapshot(ctx)\n\tif err != nil {\n\t\treturn res, err\n\t}\n\treturn res, e.reconcileSnapshotData(ctx, res)\n}\n\n// snapshot is the actual snapshot save/upload implementation.\n// This is not inline in the Snapshot function so that the save and reconcile operation\n// metrics do not overlap.\nfunc (e *ETCD) snapshot(ctx context.Context) (_ *managed.SnapshotResult, rerr error) {\n\tsnapshotStart := time.Now()\n\tdefer metrics.ObserveWithStatus(snapshotmetrics.SaveCount, snapshotStart, rerr)\n\n\tif !e.snapshotMu.TryLock() {\n\t\treturn nil, errors.New(\"snapshot save already in progress\")\n\t}\n\tdefer e.snapshotMu.Unlock()\n\t// make sure the core.Factory is initialized before attempting to add snapshot metadata\n\tvar extraMetadata *v1.ConfigMap\n\tif e.config.Runtime.Core == nil {\n\t\tlogrus.Debugf(\"Cannot retrieve extra metadata from %s ConfigMap: runtime core not ready\", snapshot.ExtraMetadataConfigMapName)\n\t} else {\n\t\tlogrus.Debugf(\"Attempting to retrieve extra metadata from %s ConfigMap\", snapshot.ExtraMetadataConfigMapName)\n\t\tif snapshotExtraMetadataConfigMap, err := e.config.Runtime.Core.Core().V1().ConfigMap().Get(metav1.NamespaceSystem, snapshot.ExtraMetadataConfigMapName, metav1.GetOptions{}); err != nil {\n\t\t\tlogrus.Debugf(\"Error encountered attempting to retrieve extra metadata from %s ConfigMap, error: %v\", snapshot.ExtraMetadataConfigMapName, err)\n\t\t} else {\n\t\t\tlogrus.Debugf(\"Setting extra metadata from %s ConfigMap\", snapshot.ExtraMetadataConfigMapName)\n\t\t\textraMetadata = snapshotExtraMetadataConfigMap\n\t\t}\n\t}\n\n\tendpoints := getEndpoints(e.config)\n\tstatus, err := e.client.Status(ctx, endpoints[0])","sourceCodeStart":208,"sourceCodeEnd":244,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/etcd/snapshot.go#L208-L244","documentation":"The snapshot save path is serialized with a mutex and uses TryLock so a second save that arrives while one is running fails immediately instead of queueing. The error means another snapshot save (scheduled or manual) currently holds the lock.","triggerScenarios":"Invoking etcd-snapshot save (or POST /db/snapshot with operation=save) while the retention timer's periodic snapshot or a previous manual save is still in flight; tight retry loops from automation that treat the 500 as fatal.","commonSituations":"Cron-driven manual snapshots overlapping the built-in schedule; large/slow S3 uploads extending save duration; HA controllers or scripts racing each other to trigger saves.","solutions":["Retry after the in-flight save completes; watch server logs or metrics (snapshot save duration) for completion.","Space out triggers: offset manual crons from the built-in snapshot schedule, or disable one of them.","If saves are abnormally long, check S3 throughput and etcd db size and prune old snapshots."],"exampleFix":"# before: fire-and-forget cron that collides\n* * * * * k3s etcd-snapshot save --s3\n# after: back off when a save is in progress\nfor i in 1 2 3 4 5; do k3s etcd-snapshot save --s3 && break; sleep 300; done","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"res, err := e.snapshot(ctx)\nif err != nil && strings.Contains(err.Error(), \"snapshot save already in progress\") {\n    // not a failure: wait for the running save to finish, then either\n    // return its result via metrics/logs or retry once after the save window\n    time.Sleep(saveRetryInterval)\n    res, err = e.snapshot(ctx)\n}","preventionTips":["Offset manual snapshot crons from the built-in schedule.","Treat this error as backpressure, not as an alert-worthy failure."],"tags":["etcd","snapshot","concurrency"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}