{"record":{"id":"c535fd3f329cf47a","repo":"cilium/cilium","slug":"there-are-still-keys-to-be-synchronized","errorCode":null,"errorMessage":"there are still keys to be synchronized","messagePattern":"there are still keys to be synchronized","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/kvstore/store/syncstore.go","lineNumber":309,"sourceCode":"\t\treturn err\n\t}\n\n\twss.log.Debug(\"Deleted key from kvstore\",\n\t\tlogfields.Key, key,\n\t)\n\treturn nil\n}\n\nfunc (wss *wqSyncStore) handleSync(ctx context.Context, skipCallbacks bool) error {\n\t// This could be replaced by wss.toSync.Len() == 0 if it only existed...\n\tsyncCompleted := true\n\twss.pendingSync.Range(func(string, struct{}) bool {\n\t\tsyncCompleted = false\n\t\treturn false\n\t})\n\n\tif !syncCompleted {\n\t\treturn fmt.Errorf(\"there are still keys to be synchronized\")\n\t}\n\n\tkey := wss.getSyncedKey()\n\n\terr := wss.backend.Update(ctx, key, []byte(time.Now().Format(time.RFC3339)), wss.withLease)\n\tif err != nil {\n\t\twss.log.Warn(\"Failed upserting synced key in kvstore. Retrying...\",\n\t\t\tlogfields.Error, err,\n\t\t\tlogfields.Key, key,\n\t\t)\n\t\treturn err\n\t}\n\n\twss.log.Info(\"Initial synchronization from the external source completed\",\n\t\tlogfields.Key, key,\n\t)\n\twss.syncedMetric.Set(metrics.BoolToFloat64(true))\n","sourceCodeStart":291,"sourceCodeEnd":327,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/pkg/kvstore/store/syncstore.go#L291-L327","documentation":"wqSyncStore.handleSync reports synchronization progress; if the pendingSync set is still non-empty it returns \"there are still keys to be synchronized\" instead of writing the synced-marker key to the backend. This is a soft/error state indicating the work queue has not drained all queued key upserts yet, so the global last-synced timestamp is not updated.","triggerScenarios":"handle (the work-queue handler) invokes handleSync while one or more keys remain in pendingSync — i.e. previous key upserts have not been retried/acknowledged by the backend; repeated kvstore write failures leave entries in pendingSync.","commonSituations":"etcd/consul temporarily unavailable while the sync work queue keeps retrying; an individual key's Upsert failing repeatedly and blocking completion of the sync cycle; agent shutting down mid-sync.","solutions":["Inspect logs for which keys remain in pendingSync and why their backend updates fail","Restore kvstore connectivity/permissions; the work queue will retry and drain pendingSync automatically","Check backend Update errors for rate limits or lease issues (wss.withLease) that keep writes failing"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := handleSync(ctx); err != nil {\n    if strings.Contains(err.Error(), \"there are still keys to be synchronized\") {\n        // pending keys not yet acked by backend; the work queue retries automatically\n        log.Debug(\"sync not complete, pending keys remain\", \"pending\", wss.pendingSync)\n        return err // let the caller retry\n    }\n    return err\n}","preventionTips":["Monitor kvstore write errors; persistent failures keep pendingSync populated","Avoid shutting down the agent while sync is in progress (graceful shutdown)","Alert on repeated 'there are still keys to be synchronized' messages — indicates backend write issues"],"tags":["kvstore","sync","workqueue","cilium"],"backgroundTag":"sync-pending-keys","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}