{"record":{"id":"f01f7cf72d84bd30","repo":"containerd/containerd","slug":"failed-to-update-sandbox-s-in-core-store-w","errorCode":null,"errorMessage":"failed to update sandbox %s in core store: %w","messagePattern":"failed to update sandbox (.+?) in core store: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/cri/server/sandbox_update_resources.go","lineNumber":76,"sourceCode":"\t)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to update sandbox store: %w\", err)\n\t}\n\n\tsandboxInfo, err := c.client.SandboxStore().Get(ctx, sandbox.ID)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to get sandbox %s from sandbox store: %w\", sandbox.ID, err)\n\t}\n\n\tupdatedRes := podsandbox.UpdatedResources{\n\t\tOverhead:  overhead,\n\t\tResources: resources,\n\t}\n\tif err := sandboxInfo.AddExtension(podsandbox.UpdatedResourcesKey, &updatedRes); err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to add updated sandbox resources extension: %w\", err)\n\t}\n\tif _, err := c.client.SandboxStore().Update(ctx, sandboxInfo, \"extensions\"); err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to update sandbox %s in core store: %w\", sandbox.ID, err)\n\t}\n\n\tif err := c.sandboxService.UpdateSandbox(ctx, sandboxInfo.Sandboxer, sandboxInfo.ID, sandboxInfo, \"extensions\"); err != nil {\n\t\t// Tolerate these errors for older sandbox controllers that may not support this yet.\n\t\tif !errdefs.IsNotImplemented(err) {\n\t\t\treturn nil, fmt.Errorf(\"failed to update sandbox controller: %w\", err)\n\t\t}\n\t\tlog.G(ctx).Tracef(\"sandbox controller %q does not implement Update endpoint\", sandboxInfo.Sandboxer)\n\t}\n\n\terr = c.nri.PostUpdatePodSandboxResources(ctx, &sandbox)\n\tif err != nil {\n\t\tlog.G(ctx).WithError(err).Errorf(\"NRI post-update notification failed\")\n\t}\n\n\treturn &runtime.UpdatePodSandboxResourcesResponse{}, nil\n}\n","sourceCodeStart":58,"sourceCodeEnd":94,"githubUrl":"https://github.com/containerd/containerd/blob/4246446a2bf7d03837b0244118d858799393bd80/internal/cri/server/sandbox_update_resources.go#L58-L94","documentation":"After adding the extension, the code persists it with c.client.SandboxStore().Update(ctx, sandboxInfo, \"extensions\"); failure is wrapped as `failed to update sandbox %s in core store: %w`. The CRI status was updated but the durable record could not be written, so the update is incomplete and will be lost on restart.","triggerScenarios":"Core SandboxStore.Update with fieldpath \"extensions\" fails — concurrent writers causing optimistic-concurrency/ETag conflicts, the sandbox record vanished between Get and Update, or the metadata DB write errored (disk full/locked).","commonSituations":"Kubelet and another controller updating the same pod concurrently; containerd metadata boltDB lock contention; disk-full on /var/lib/containerd; pod deleted between the Get and Update calls.","solutions":["Retry UpdatePodSandboxResources — transient concurrent-update conflicts usually resolve.","Verify disk space and metadata DB health (`df -h /var/lib/containerd`, containerd logs for boltDB errors).","Confirm the sandbox still exists (`crictl pods`) — if deleted concurrently, recreate instead of retrying.","Restart containerd if the metadata store is stuck in a locked/inconsistent state."],"exampleFix":"// before\nsudo crictl updatep ...  # fails while disk full\n// after\ndf -h /var/lib/containerd && sudo journalctl -u containerd --since -5m | grep -i bolt\nsudo crictl updatep --memory 536870912 \"$id\"","handlingStrategy":"retry","validationCode":"// pre-checks: disk free and sandbox still present\n// df -h /var/lib/containerd && crictl pods | grep <id>","typeGuard":null,"tryCatchPattern":"try {\n  await client.updatePodSandboxResources(req);\n} catch (err) {\n  if (String(err).includes('in core store')) {\n    // likely concurrent writer/delete — short backoff, verify existence, retry once\n    await sleep(500);\n    if (await podExists(id)) return client.updatePodSandboxResources(req);\n  }\n  throw err;\n}","preventionTips":["Serialize mutating operations per pod (no kubelet + custom controller races).","Monitor disk on /var/lib/containerd.","Handle concurrent pod deletion gracefully.","Watch boltDB lock contention in containerd logs."],"tags":["cri","sandbox","persistence","core-store","containerd"],"backgroundTag":"sandbox-store-update-failed","analyzedSha":"4246446a2bf7d03837b0244118d858799393bd80","analyzedAt":"2026-09-02T00:14:43.053Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}