kopia/kopia · error
unable to delete blob
Error message
unable to delete blob %v
What it means
deleteBlobsFromStorageAndCache failed to delete a specific blob from the underlying blob storage (m.st.DeleteBlob). Errors equal to blob.ErrBlobNotFound are deliberately tolerated (already deleted); any other error is wrapped with the blob ID. This aborts the delete loop, leaving remaining blobs undeleted.
Solutions
- Read the blob ID from the message and the wrapped provider error for the root cause.
- Verify the storage credentials/role permit DeleteBlob operations.
- Check for immutability/retention policies (S3 Object Lock, Azure immutability) on the bucket/container.
- Retry the operation; missing blobs are already tolerated so a rerun only targets the remaining blobs.
Example fix
// before
if err := m.st.DeleteBlob(ctx, blobID); err != nil && !errors.Is(err, blob.ErrBlobNotFound) {
return errors.Wrapf(err, "unable to delete blob %v", blobID)
}
// after
if err := m.st.DeleteBlob(ctx, blobID); err != nil && !errors.Is(err, blob.ErrBlobNotFound) {
if retry.IsTransient(err) {
continue // or back off and retry the whole pass
}
return errors.Wrapf(err, "unable to delete blob %v", blobID)
} Defensive patterns
Strategy: retry
Validate before calling
// Go: verify delete permission before GC passes
if err := probeBlobDelete(ctx, m.st, testBlobID); err != nil {
return fmt.Errorf("storage delete probe failed: %w", err)
} Try / catch
if err != nil && strings.Contains(err.Error(), "unable to delete blob ") { // back off and retry; ErrBlobNotFound tolerated on rerun
backoff.Retry(func() error { return mgr.Compact(ctx) }, backoff.WithMaxRetries(3, time.Second))
} Prevention
- Grant storage credentials delete permissions, not just read/write
- Avoid S3 Object Lock/immutability policies on kopia-managed buckets used for GC
- Alert on provider delete-API error rates
When it happens
Trigger: m.st.DeleteBlob(ctx, blobID) returns a non-ErrBlobNotFound error during deleteOldBlobs or cleanup — e.g. storage API failure, permission denial, network error, or a conditional-delete conflict.
Common situations: Cloud credentials lacking delete permission (write-only or read-only policy), blob locked by an immutability/retention policy (S3 Object Lock, WORM), transient provider 5xx, or eventual-consistency races where a replica reports the blob in a conflicting state.
Related errors
AI-assisted analysis of kopia/kopia@82495e54b5 (2026-09-07).
Data as JSON: /api/errors/6fa64b907a2bf2d6.
Report an issue: GitHub.
Appendix: source
Thrown at repo/content/indexblob/index_blob_manager_v0.go:412
BlobIDs: blobIDs,
CleanupScheduleTime: cleanupScheduleTime,
})
if err != nil {
return errors.Wrap(err, "unable to marshal cleanup log bytes")
}
if _, err := m.enc.EncryptAndWriteBlob(ctx, gather.FromSlice(payload), V0CleanupBlobPrefix, ""); err != nil {
return errors.Wrap(err, "unable to cleanup log")
}
return nil
}
func (m *ManagerV0) deleteBlobsFromStorageAndCache(ctx context.Context, blobIDs []blob.ID) error {
for _, blobID := range blobIDs {
if err := m.st.DeleteBlob(ctx, blobID); err != nil && !errors.Is(err, blob.ErrBlobNotFound) {
contentlog.Log2(ctx, m.log, "delete-blob failed", blobparam.BlobID("blobID", blobID), logparam.Error("err", err))
return errors.Wrapf(err, "unable to delete blob %v", blobID)
}
contentlog.Log1(ctx, m.log, "delete-blob succeeded", blobparam.BlobID("blobID", blobID))
}
return nil
}
func (m *ManagerV0) cleanup(ctx context.Context, maxEventualConsistencySettleTime time.Duration) error {
allCleanupBlobs, err := blob.ListAllBlobs(ctx, m.st, V0CleanupBlobPrefix)
if err != nil {
return errors.Wrap(err, "error listing cleanup blobs")
}
// determine latest storage write time of a cleanup blob
var latestStorageWriteTimestamp time.Time
for _, cb := range allCleanupBlobs {View on GitHub (pinned to 82495e54b5)