What causes "there are still keys to be synchronized" in cilium/cilium?

Inspect logs for which keys remain in pendingSync and why their backend updates fail

there are still keys to be synchronized

Error message

there are still keys to be synchronized

What it means

wqSyncStore.handleSync reports synchronization progress; if the pendingSync set is still non-empty it returns "there are still keys to be synchronized" instead of writing the synced-marker key to the backend. This is a soft/error state indicating the work queue has not drained all queued key upserts yet, so the global last-synced timestamp is not updated.

Source

Thrown at pkg/kvstore/store/syncstore.go:309

		return err
	}

	wss.log.Debug("Deleted key from kvstore",
		logfields.Key, key,
	)
	return nil
}

func (wss *wqSyncStore) handleSync(ctx context.Context, skipCallbacks bool) error {
	// This could be replaced by wss.toSync.Len() == 0 if it only existed...
	syncCompleted := true
	wss.pendingSync.Range(func(string, struct{}) bool {
		syncCompleted = false
		return false
	})

	if !syncCompleted {
		return fmt.Errorf("there are still keys to be synchronized")
	}

	key := wss.getSyncedKey()

	err := wss.backend.Update(ctx, key, []byte(time.Now().Format(time.RFC3339)), wss.withLease)
	if err != nil {
		wss.log.Warn("Failed upserting synced key in kvstore. Retrying...",
			logfields.Error, err,
			logfields.Key, key,
		)
		return err
	}

	wss.log.Info("Initial synchronization from the external source completed",
		logfields.Key, key,
	)
	wss.syncedMetric.Set(metrics.BoolToFloat64(true))

View on GitHub (pinned to ac7b90affa)

Solutions

Inspect logs for which keys remain in pendingSync and why their backend updates fail
Restore kvstore connectivity/permissions; the work queue will retry and drain pendingSync automatically
Check backend Update errors for rate limits or lease issues (wss.withLease) that keep writes failing

Defensive patterns

Strategy: retry

Try / catch

if err := handleSync(ctx); err != nil {
    if strings.Contains(err.Error(), "there are still keys to be synchronized") {
        // pending keys not yet acked by backend; the work queue retries automatically
        log.Debug("sync not complete, pending keys remain", "pending", wss.pendingSync)
        return err // let the caller retry
    }
    return err
}

Prevention

Monitor kvstore write errors; persistent failures keep pendingSync populated
Avoid shutting down the agent while sync is in progress (graceful shutdown)
Alert on repeated 'there are still keys to be synchronized' messages — indicates backend write issues

When it happens

Trigger: handle (the work-queue handler) invokes handleSync while one or more keys remain in pendingSync — i.e. previous key upserts have not been retried/acknowledged by the backend; repeated kvstore write failures leave entries in pendingSync.

Common situations: etcd/consul temporarily unavailable while the sync work queue keeps retrying; an individual key's Upsert failing repeatedly and blocking completion of the sync cycle; agent shutting down mid-sync.

Related errors

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/c535fd3f329cf47a. Report an issue: GitHub.