argoproj/argo-workflows · error

delete %s: %w

Error message

delete %s: %w

What it means

This error is thrown by deleteObject when the GCS API client's Object.Delete call fails while removing an artifact object from a bucket. It wraps the storage library error with the object key so the operator knows which object could not be deleted. ArtifactDriver.Delete retries it via waitutil.Backoff, so reaching the caller means the error persisted (or is non-transient, e.g. NotFound or PermissionDenied).

Source

Thrown at workflow/artifacts/gcs/gcs.go:335

			logger := logging.RequireLoggerFromContext(ctx)
			logger.WithField("path", localPath).WithError(closeErr).Error(ctx, "Error closing file")
		}
	}()
	wc := client.Bucket(bucket).Object(key).NewWriter(ctx)
	if _, err = io.Copy(wc, f); err != nil {
		return fmt.Errorf("io copy: %w", err)
	}
	if err := wc.Close(); err != nil {
		return fmt.Errorf("writer close: %w", err)
	}
	return nil
}

// delete an object from GCS
func deleteObject(ctx context.Context, client *storage.Client, bucket, key string) error {
	err := client.Bucket(bucket).Object(key).Delete(ctx)
	if err != nil {
		return fmt.Errorf("delete %s: %w", key, err)
	}
	return nil
}

// Delete deletes an artifact from GCS
func (h *ArtifactDriver) Delete(ctx context.Context, s *wfv1.Artifact) error {
	err := waitutil.Backoff(defaultRetry,
		func() (bool, error) {
			client, err := h.newGCSClient(ctx)
			if err != nil {
				return !isTransientGCSErr(ctx, err), err
			}
			defer client.Close()
			err = deleteObject(ctx, client, s.GCS.Bucket, s.GCS.Key)
			if err != nil {
				return !isTransientGCSErr(ctx, err), err
			}
			return true, nil

View on GitHub (pinned to 35bff19146)

Solutions

1. Check the wrapped error code: 404 is usually harmless (object already gone) — consider treating storage.ErrObjectNotExist as success in cleanup paths.
2. Grant storage.objectAdmin (or objectDelete) on the bucket to the workflow's service account: `gsutil iam ch serviceAccount:<sa>:roles/storage.objectAdmin gs://<bucket>`.
3. Verify the key/bucket in the artifact record is correct — a stale or mis-encoded key can point at a nonexistent object.
4. Check bucket retention policy/lock: `gsutil retention ls gs://<bucket>`; wait out or lift the retention period if deletion is genuinely required.
5. Retry later if the wrapped error is 429/5xx; the driver's defaultRetry backoff may need more attempts for heavily throttled buckets.

Example fix

// before: cleanup fails the workflow when the object is already gone
err = deleteObject(ctx, client, s.GCS.Bucket, s.GCS.Key)
if err != nil {
    return fmt.Errorf("delete %s: %w", key, err)
}
// after: tolerate NotFound during cleanup
err = deleteObject(ctx, client, s.GCS.Bucket, s.GCS.Key)
if err != nil {
    if status.Code(err) == codes.NotFound {
        return nil
    }
    return fmt.Errorf("delete %s: %w", key, err)
}

// check the object exists and is deletable before calling Delete
handle := client.Bucket(bucket).Object(key)
if _, err := handle.Attrs(ctx); err != nil {
    if status.Code(err) == codes.NotFound {
        return nil // nothing to delete
    }
    return fmt.Errorf("cannot access gs://%s/%s: %w", bucket, key, err)
}

func isDeleteWrapErr(err error) (*googleapi.Error, bool) {
    var gerr *googleapi.Error
    if errors.As(err, &gerr) && strings.Contains(err.Error(), "delete ") {
        return gerr, true
    }
    return nil, false
}

err := driver.Delete(ctx, artifact)
if err != nil {
    var gerr *googleapi.Error
    if errors.As(err, &gerr) {
        switch gerr.Code {
        case 404:
            // object already gone: treat as success in cleanup paths
        case 403:
            // IAM: grant storage.objects.delete / objectAdmin
        default:
            // 429/5xx: backoff and retry
        }
    }
}

When it happens

Trigger: client.Bucket(bucket).Object(key).Delete(ctx) returns an error: object does not exist (storage.ErrObjectNotExist / 404), service account lacks storage.objects.delete (403), bucket is versioned with retention lock, rate-limited (429), or the context was canceled (workflow deleted mid-cleanup).

Common situations: Deleting an artifact that was never uploaded (garbage-collection cleanup of failed workflows); IAM role downgraded to read-only on the artifact bucket; bucket-level retention policy or Object Lock preventing deletion; deleting already-pruned objects twice; transient GCS 5xx exhausting the retry backoff.

Related errors

• unable to delete Azure Blob %s: %w
• new bucket reader: %w
• os create %s: %w
• upload %s: %w
• writer close: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/cfd6204b902f890d. Report an issue: GitHub.