k3s-io/k3s · warning

snapshot save already in progress

Error message

snapshot save already in progress

What it means

The snapshot save path is serialized with a mutex and uses TryLock so a second save that arrives while one is running fails immediately instead of queueing. The error means another snapshot save (scheduled or manual) currently holds the lock.

Source

Thrown at pkg/etcd/snapshot.go:226

// subcommand for prune that can be run manually if the user wants to remove old snapshots.
// Returns metadata about the new and pruned snapshots.
func (e *ETCD) Snapshot(ctx context.Context) (*managed.SnapshotResult, error) {
	res, err := e.snapshot(ctx)
	if err != nil {
		return res, err
	}
	return res, e.reconcileSnapshotData(ctx, res)
}

// snapshot is the actual snapshot save/upload implementation.
// This is not inline in the Snapshot function so that the save and reconcile operation
// metrics do not overlap.
func (e *ETCD) snapshot(ctx context.Context) (_ *managed.SnapshotResult, rerr error) {
	snapshotStart := time.Now()
	defer metrics.ObserveWithStatus(snapshotmetrics.SaveCount, snapshotStart, rerr)

	if !e.snapshotMu.TryLock() {
		return nil, errors.New("snapshot save already in progress")
	}
	defer e.snapshotMu.Unlock()
	// make sure the core.Factory is initialized before attempting to add snapshot metadata
	var extraMetadata *v1.ConfigMap
	if e.config.Runtime.Core == nil {
		logrus.Debugf("Cannot retrieve extra metadata from %s ConfigMap: runtime core not ready", snapshot.ExtraMetadataConfigMapName)
	} else {
		logrus.Debugf("Attempting to retrieve extra metadata from %s ConfigMap", snapshot.ExtraMetadataConfigMapName)
		if snapshotExtraMetadataConfigMap, err := e.config.Runtime.Core.Core().V1().ConfigMap().Get(metav1.NamespaceSystem, snapshot.ExtraMetadataConfigMapName, metav1.GetOptions{}); err != nil {
			logrus.Debugf("Error encountered attempting to retrieve extra metadata from %s ConfigMap, error: %v", snapshot.ExtraMetadataConfigMapName, err)
		} else {
			logrus.Debugf("Setting extra metadata from %s ConfigMap", snapshot.ExtraMetadataConfigMapName)
			extraMetadata = snapshotExtraMetadataConfigMap
		}
	}

	endpoints := getEndpoints(e.config)
	status, err := e.client.Status(ctx, endpoints[0])

View on GitHub (pinned to 6ba341e396)

Solutions

1. Retry after the in-flight save completes; watch server logs or metrics (snapshot save duration) for completion.
2. Space out triggers: offset manual crons from the built-in snapshot schedule, or disable one of them.
3. If saves are abnormally long, check S3 throughput and etcd db size and prune old snapshots.

Example fix

# before: fire-and-forget cron that collides
* * * * * k3s etcd-snapshot save --s3
# after: back off when a save is in progress
for i in 1 2 3 4 5; do k3s etcd-snapshot save --s3 && break; sleep 300; done

res, err := e.snapshot(ctx)
if err != nil && strings.Contains(err.Error(), "snapshot save already in progress") {
    // not a failure: wait for the running save to finish, then either
    // return its result via metrics/logs or retry once after the save window
    time.Sleep(saveRetryInterval)
    res, err = e.snapshot(ctx)
}

When it happens

Trigger: Invoking etcd-snapshot save (or POST /db/snapshot with operation=save) while the retention timer's periodic snapshot or a previous manual save is still in flight; tight retry loops from automation that treat the 500 as fatal.

Common situations: Cron-driven manual snapshots overlapping the built-in schedule; large/slow S3 uploads extending save duration; HA controllers or scripts racing each other to trigger saves.

Related errors

• no snapshots given for removal
• invalid output format:
• etcd-snapshot-reconcile-interval must be greater than 0s
• unexpected compressed etcd snapshot contents
• etcd: snapshot path does not exist: %s

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/5ae119305b1cee11. Report an issue: GitHub.