argoproj/argo-workflows · error

plugin %s delete failed: %w

Error message

plugin %s delete failed: %w

What it means

Driver.Delete forwards an artifact deletion to the artifact plugin over gRPC (client.Delete). When the RPC itself fails — transport error, context deadline, Unimplemented, etc. — the error is wrapped as "plugin %s delete failed: %w". This is distinct from error 512, which fires when the RPC succeeds but the plugin reports a logical failure via resp.Success=false.

Source

Thrown at workflow/artifacts/plugin/plugin.go:333

	return resp.GetSupportsSaveStream(), nil
}

// saveStreamViaTempFile is the fallback used when the plugin doesn't implement
// streaming SaveStream: buffer to a temp file and call the existing unary Save.
func (d *Driver) saveStreamViaTempFile(ctx context.Context, reader io.Reader, outputArtifact *wfv1.Artifact) error {
	return common.SaveStreamViaTempFile(reader, "plugin-upload-*", func(path string) error {
		return d.Save(ctx, path, outputArtifact)
	})
}

// Delete implements ArtifactDriver.Delete by calling the plugin service
func (d *Driver) Delete(ctx context.Context, artifactRef *wfv1.Artifact) error {
	grpcArtifact := convertToGRPC(artifactRef)
	resp, err := d.client.Delete(ctx, &artifact.DeleteArtifactRequest{
		Artifact: grpcArtifact,
	})
	if err != nil {
		return fmt.Errorf("plugin %s delete failed: %w", d.pluginName, err)
	}
	if !resp.Success {
		return fmt.Errorf("plugin %s delete failed: %s", d.pluginName, resp.Error)
	}
	return nil
}

// ListObjects implements ArtifactDriver.ListObjects by calling the plugin service
func (d *Driver) ListObjects(ctx context.Context, artifactRef *wfv1.Artifact) ([]string, error) {
	grpcArtifact := convertToGRPC(artifactRef)
	resp, err := d.client.ListObjects(ctx, &artifact.ListObjectsRequest{
		Artifact: grpcArtifact,
	})
	if err != nil {
		return nil, fmt.Errorf("plugin %s list objects failed: %w", d.pluginName, err)
	}
	if resp.Error != "" {
		return nil, fmt.Errorf("plugin %s list objects failed: %s", d.pluginName, resp.Error)

View on GitHub (pinned to 35bff19146)

Solutions

  1. Check the wrapped (%w) gRPC status: Unavailable means fix plugin connectivity/restart the plugin; Unimplemented means upgrade the plugin to a build that implements Delete.
  2. Verify the artifact reference is complete and valid (key/bucket fields) so the plugin receives a well-formed request.
  3. Check plugin logs for the server-side cause (storage credentials revoked, object already gone).
  4. If deletion is best-effort garbage collection, tolerate the error and retry later; Delete is idempotent for most storage backends.

Example fix

// caller tolerating best-effort cleanup
// before:
if err := driver.Delete(ctx, art); err != nil { return err }
// after:
if err := driver.Delete(ctx, art); err != nil {
	if status.Code(err) == codes.Unimplemented || status.Code(err) == codes.Unavailable {
		log.Warn(ctx, "defer artifact delete", "err", err) // retry via GC later
		return nil
	}
	return err
}
Defensive patterns

Strategy: retry

Validate before calling

// validate artifact ref before delete
func validForDelete(a *wfv1.Artifact) error {
	if a == nil || a.GetName() == "" {
		return errors.New("artifact ref incomplete: name required")
	}
	return nil
}

Type guard

func isRetryableDeleteErr(err error) bool {
	switch status.Code(errors.Unwrap(err)) {
	case codes.Unavailable, codes.DeadlineExceeded, codes.ResourceExhausted:
		return true
	default:
		return false
	}
}

Try / catch

err := driver.Delete(ctx, art)
if err != nil {
	if isRetryableDeleteErr(err) {
		return retry.WithExponentialBackoff(ctx, func() error { return driver.Delete(ctx, art) }, 3)
	}
	return fmt.Errorf("artifact delete not retryable: %w", err)
}

Prevention

When it happens

Trigger: Calling Driver.Delete(ctx, artifactRef) when the plugin's Delete RPC returns a gRPC error: plugin unreachable (Unavailable), method not implemented (Unimplemented, old plugin binary), ctx deadline exceeded, or the artifact ref failed conversion/serialization causing an InvalidArgument.

Common situations: Garbage-collecting artifacts after workflow deletion with the plugin offline; plugin upgraded to a version without Delete (Unimplemented); artifact reference pointing at storage the plugin can't reach, surfaced as a deadline or internal error; network partition between controller and plugin service.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/e482409a8b90c050. Report an issue: GitHub.