k3s-io/k3s · warning

snapshot save already in progress

Error message

snapshot save already in progress

What it means

The snapshot save path is serialized with a mutex and uses TryLock so a second save that arrives while one is running fails immediately instead of queueing. The error means another snapshot save (scheduled or manual) currently holds the lock.

Solutions

1. Retry after the in-flight save completes; watch server logs or metrics (snapshot save duration) for completion.
2. Space out triggers: offset manual crons from the built-in snapshot schedule, or disable one of them.
3. If saves are abnormally long, check S3 throughput and etcd db size and prune old snapshots.

Example fix

# before: fire-and-forget cron that collides
* * * * * k3s etcd-snapshot save --s3
# after: back off when a save is in progress
for i in 1 2 3 4 5; do k3s etcd-snapshot save --s3 && break; sleep 300; done

res, err := e.snapshot(ctx)
if err != nil && strings.Contains(err.Error(), "snapshot save already in progress") {
    // not a failure: wait for the running save to finish, then either
    // return its result via metrics/logs or retry once after the save window
    time.Sleep(saveRetryInterval)
    res, err = e.snapshot(ctx)
}

When it happens

Trigger: Invoking etcd-snapshot save (or POST /db/snapshot with operation=save) while the retention timer's periodic snapshot or a previous manual save is still in flight; tight retry loops from automation that treat the 500 as fatal.

Common situations: Cron-driven manual snapshots overlapping the built-in schedule; large/slow S3 uploads extending save duration; HA controllers or scripts racing each other to trigger saves.

Related errors

• etcd: snapshot path does not exist
• etcd: snapshot path must be a file, not a directory
• etcd-snapshot-reconcile-interval must be greater than 0s
• invalid output format:
• no snapshots given for removal

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/5ae119305b1cee11. Report an issue: GitHub.

Appendix: source

Thrown at pkg/etcd/snapshot.go:226

// subcommand for prune that can be run manually if the user wants to remove old snapshots.
// Returns metadata about the new and pruned snapshots.
func (e *ETCD) Snapshot(ctx context.Context) (*managed.SnapshotResult, error) {
	res, err := e.snapshot(ctx)
	if err != nil {
		return res, err
	}
	return res, e.reconcileSnapshotData(ctx, res)
}

// snapshot is the actual snapshot save/upload implementation.
// This is not inline in the Snapshot function so that the save and reconcile operation
// metrics do not overlap.
func (e *ETCD) snapshot(ctx context.Context) (_ *managed.SnapshotResult, rerr error) {
	snapshotStart := time.Now()
	defer metrics.ObserveWithStatus(snapshotmetrics.SaveCount, snapshotStart, rerr)

	if !e.snapshotMu.TryLock() {
		return nil, errors.New("snapshot save already in progress")
	}
	defer e.snapshotMu.Unlock()
	// make sure the core.Factory is initialized before attempting to add snapshot metadata
	var extraMetadata *v1.ConfigMap
	if e.config.Runtime.Core == nil {
		logrus.Debugf("Cannot retrieve extra metadata from %s ConfigMap: runtime core not ready", snapshot.ExtraMetadataConfigMapName)
	} else {
		logrus.Debugf("Attempting to retrieve extra metadata from %s ConfigMap", snapshot.ExtraMetadataConfigMapName)
		if snapshotExtraMetadataConfigMap, err := e.config.Runtime.Core.Core().V1().ConfigMap().Get(metav1.NamespaceSystem, snapshot.ExtraMetadataConfigMapName, metav1.GetOptions{}); err != nil {
			logrus.Debugf("Error encountered attempting to retrieve extra metadata from %s ConfigMap, error: %v", snapshot.ExtraMetadataConfigMapName, err)
		} else {
			logrus.Debugf("Setting extra metadata from %s ConfigMap", snapshot.ExtraMetadataConfigMapName)
			extraMetadata = snapshotExtraMetadataConfigMap
		}
	}

	endpoints := getEndpoints(e.config)
	status, err := e.client.Status(ctx, endpoints[0])

View on GitHub (pinned to 6ba341e396)