What causes "no blob found to mark delete failed, ID:%d, digest:%s" in goharbor/harbor?

Serialize GC executions — let Harbor's GC lock order scheduled and manual runs instead of bypassing it

goharbor/harbor · error · lib/errors.Error

NOT_FOUND

NOT_FOUND

Error message

no blob found to mark delete failed, ID:%d, digest:%s

What it means

During garbage collection, when deleting a blob fails, GarbageCollector.markDeleteFailed (src/jobservice/job/impl/gc/garbage_collection.go:728) flags the blob StatusDeleteFailed via blobMgr.UpdateBlobStatus, which returns the count of updated rows. A count of 0 means no row matched the blob's ID/digest, so a NotFound-coded error is returned: the database row vanished between candidate selection and the status update. The usual cause is a concurrent GC or an external process deleting the blob row first.

Source

Thrown at src/jobservice/job/impl/gc/garbage_collection.go:728

				return blobs, err
			}
			blobs = append(blobs, artBlobs...)
		}
	}

	return blobs, err
}

// markDeleteFailed set the blob status to StatusDeleteFailed
func (gc *GarbageCollector) markDeleteFailed(ctx job.Context, blob *blobModels.Blob) error {
	blob.Status = blobModels.StatusDeleteFailed
	count, err := gc.blobMgr.UpdateBlobStatus(ctx.SystemContext(), blob)
	if err != nil {
		gc.logger.Errorf("failed to mark gc candidate delete failed: %s, %s", blob.Digest, blob.Status)
		return errors.Wrapf(err, "failed to mark gc candidate delete failed: %s, %s", blob.Digest, blob.Status)
	}
	if count == 0 {
		return errors.New(nil).WithMessagef("no blob found to mark delete failed, ID:%d, digest:%s", blob.ID, blob.Digest).WithCode(errors.NotFoundCode)
	}
	return nil
}

func (gc *GarbageCollector) shouldStop(ctx job.Context) bool {
	opCmd, exit := ctx.OPCommand()
	if exit && opCmd.IsStop() {
		return true
	}
	return false
}

func saveGCRes(ctx job.Context, sweepSize, blobs, manifests int64) error {
	gcObj := struct {
		SweepSize int64 `json:"freed_space"`
		Blobs     int64 `json:"purged_blobs"`
		Manifests int64 `json:"purged_manifests"`
	}{

View on GitHub (pinned to 7b2fd08cc5)

Solutions

Serialize GC executions — let Harbor's GC lock order scheduled and manual runs instead of bypassing it
Treat the error as usually benign: the blob is already gone; rerun GC and the candidate list will no longer include it
Find and stop whatever else deletes blob rows (second instance, cron job, manual DELETE)
If it persists, query the blob table for the reported ID/digest to confirm the row is gone

Defensive patterns

Strategy: try-catch

Try / catch

if err := gc.markDeleteFailed(ctx, blob); err != nil {
    if liberrors.IsNotFound(err) { // UpdateBlobStatus affected 0 rows
        gc.logger.Warningf("blob %d/%s already removed by a concurrent run; skipping", blob.ID, blob.Digest)
        continue
    }
    return err
}

Prevention

Run exactly one GC at a time — rely on Harbor's GC lock and avoid direct gcctl/DB access during scheduled GC
Do not manually DELETE blob rows while GC is executing
Alert on overlapping GC schedules so the race is fixed, not just tolerated

When it happens

Trigger: Two GC executions overlapping on the same database (manual gcctl run during the scheduled GC, or a bypassed distributed lock); an external script/DBA deleting blob rows while GC holds stale candidates; stale references after restoring the DB from backup.

Common situations: Running offline GC while online GC is scheduled; multiple Harbor instances sharing one database; manual database surgery on the blob table.

Related errors

AI-assisted analysis of goharbor/harbor@7b2fd08cc5 (2026-08-16). Data as JSON: /api/errors/82107e372e7bb4b3. Report an issue: GitHub.