{"record":{"id":"3ac944ec8dd4689f","repo":"kopia/kopia","slug":"unable-to-schedule-delayed-cleanup-of-blobs","errorCode":null,"errorMessage":"unable to schedule delayed cleanup of blobs","messagePattern":"unable to schedule delayed cleanup of blobs","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"repo/content/indexblob/index_blob_manager_v0.go","lineNumber":324,"sourceCode":"\t\tlogparam.Time(\"compactionLogServerTimeCutoff\", compactionLogServerTimeCutoff))\n\n\tcompactionBlobEntries, err := m.getCompactionLogEntries(ctx, compactionBlobs)\n\tif err != nil {\n\t\treturn errors.Wrap(err, \"unable to get compaction log entries\")\n\t}\n\n\tindexBlobsToDelete := m.findIndexBlobsToDelete(ctx, latestBlob.Timestamp, compactionBlobEntries, maxEventualConsistencySettleTime)\n\n\t// note that we must always delete index blobs first before compaction logs\n\t// otherwise we may inadvertently resurrect an index blob that should have been removed.\n\tif err := m.deleteBlobsFromStorageAndCache(ctx, indexBlobsToDelete); err != nil {\n\t\treturn errors.Wrap(err, \"unable to delete compaction logs\")\n\t}\n\n\tcompactionLogBlobsToDelayCleanup := m.findCompactionLogBlobsToDelayCleanup(ctx, compactionBlobs)\n\n\tif err := m.delayCleanupBlobs(ctx, compactionLogBlobsToDelayCleanup, latestBlob.Timestamp); err != nil {\n\t\treturn errors.Wrap(err, \"unable to schedule delayed cleanup of blobs\")\n\t}\n\n\treturn nil\n}\n\nfunc (m *ManagerV0) findIndexBlobsToDelete(ctx context.Context, latestServerBlobTime time.Time, entries map[blob.ID]*compactionLogEntry, maxEventualConsistencySettleTime time.Duration) []blob.ID {\n\ttmp := map[blob.ID]bool{}\n\n\tfor _, cl := range entries {\n\t\t// are the input index blobs in this compaction eligible for deletion?\n\t\tif age := latestServerBlobTime.Sub(cl.metadata.Timestamp); age < maxEventualConsistencySettleTime {\n\t\t\tcontentlog.Log3(ctx, m.log,\n\t\t\t\t\"not deleting compacted index blob used as inputs for compaction\",\n\t\t\t\tblobparam.BlobID(\"blobID\", cl.metadata.BlobID),\n\t\t\t\tlogparam.Duration(\"age\", age),\n\t\t\t\tlogparam.Duration(\"maxEventualConsistencySettleTime\", maxEventualConsistencySettleTime))\n\n\t\t\tcontinue","sourceCodeStart":306,"sourceCodeEnd":342,"githubUrl":"https://github.com/kopia/kopia/blob/82495e54b584c1ef6073c9e1be048f57f8aef078/repo/content/indexblob/index_blob_manager_v0.go#L306-L342","documentation":"ManagerV0.deleteOldBlobs fails to schedule deferred deletion of compaction-log blobs. After deleting old index blobs, it must write a cleanup-entry blob recording which compaction logs to delete later; if delayCleanupBlobs returns an error, this wrapper is returned to registerCompaction and thus to Compact. Compaction itself succeeded, but garbage-collection bookkeeping failed, so old compaction logs linger and cleanup is not scheduled.","triggerScenarios":"Calling Compact() on an index blob manager when the underlying storage write in delayCleanupBlobs (EncryptAndWriteBlob with V0CleanupBlobPrefix) fails, or JSON marshaling of the cleanupEntry fails; the inner error is wrapped with this message by deleteOldBlobs.","commonSituations":"Blob storage backend outage or throttling during compaction (S3/Blob Storage 5xx), expired or revoked cloud credentials, network partition between the kopia client and the blob store, or a read-only/misconfigured storage destination.","solutions":["Inspect the wrapped inner error for the actual storage failure (credentials, connectivity, quota) and fix it.","Retry Compact(); delayed-cleanup scheduling is idempotent-safe to retry since it only writes a cleanup entry blob.","Verify the storage destination is writable and not in a read-only or expired state.","Check server clock sanity: cleanup schedule time comes from the latest blob's server timestamp."],"exampleFix":"// before\nif err := m.delayCleanupBlobs(ctx, compactionLogBlobsToDelayCleanup, latestBlob.Timestamp); err != nil {\n\treturn errors.Wrap(err, \"unable to schedule delayed cleanup of blobs\")\n}\n// after\nif err := m.delayCleanupBlobs(ctx, compactionLogBlobsToDelayCleanup, latestBlob.Timestamp); err != nil {\n\tm.log.Errorw(\"delayed cleanup scheduling failed; will retry on next compaction\", \"err\", err)\n\treturn errors.Wrap(err, \"unable to schedule delayed cleanup of blobs\")\n}","handlingStrategy":"retry","validationCode":"// Go: check storage writability before Compact\nif err := storageTest(ctx, blobStore); err != nil {\n\treturn fmt.Errorf(\"blob storage not writable: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"err := mgr.Compact(ctx); if err != nil && strings.Contains(err.Error(), \"unable to schedule delayed cleanup\") { // retry after storage recovers\n\tretry.Do(func() error { return mgr.Compact(ctx) }, retry.Attempts(3))\n}","preventionTips":["Run compaction during stable network/credential windows","Monitor blob-store write error rates and alert before compaction jobs","Keep storage credentials rotated with overlap so they never expire mid-job"],"tags":["blob-storage","compaction","garbage-collection"],"backgroundTag":"file-write-failed","analyzedSha":"82495e54b584c1ef6073c9e1be048f57f8aef078","analyzedAt":"2026-09-07T20:35:21.689Z","contentChangedAt":"2026-09-07T20:35:21.689Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}