argoproj/argo-workflows · error

plugin %s save failed: %w

Error message

plugin %s save failed: %w

What it means

The unary Save RPC to the artifact plugin failed at the transport/gRPC level. The driver calls client.Save with the artifact path and metadata; any error returned by the gRPC call (network failure, unimplemented method, context timeout, plugin crash) is wrapped as 'plugin %s save failed: %w'. The wrapped gRPC status is the real diagnosis.

Source

Thrown at workflow/artifacts/plugin/plugin.go:212

					writer.CloseWithError(fmt.Errorf("plugin %s stream write failed: %w", d.pluginName, writeErr))
					return
				}
			}
		}
	}()

	return reader, nil
}

// Save implements ArtifactDriver.Save by calling the plugin service
func (d *Driver) Save(ctx context.Context, path string, outputArtifact *wfv1.Artifact) error {
	grpcArtifact := convertToGRPC(outputArtifact)
	resp, err := d.client.Save(ctx, &artifact.SaveArtifactRequest{
		Path:           path,
		OutputArtifact: grpcArtifact,
	})
	if err != nil {
		return fmt.Errorf("plugin %s save failed: %w", d.pluginName, err)
	}
	if !resp.Success {
		return fmt.Errorf("plugin %s save failed: %s", d.pluginName, resp.Error)
	}
	return nil
}

// saveStreamChunkSize is the size of each chunk sent over the streaming SaveStream RPC.
// 2MiB stays well under gRPC's default 4MiB max message size while keeping the
// per-chunk marshal/syscall overhead low for multi-GB artifacts.
const saveStreamChunkSize = 2 * 1024 * 1024

// SaveStream implements ArtifactDriver.SaveStream. If the plugin advertises
// streaming support (per GetCapabilities), the reader is streamed directly with no
// local buffering. Otherwise it falls back to buffering to a temp file and calling
// the existing unary Save, so a plugin that predates streaming keeps working.
//
// Capability is checked before reader is touched: once GetCapabilities confirms

View on GitHub (pinned to 35bff19146)

Solutions

  1. Decode the wrapped gRPC status (status.FromError) to see the exact code (Unavailable, DeadlineExceeded, Unimplemented, ResourceExhausted)
  2. Check the plugin container logs and restart state (kubectl get pods, describe)
  3. If Unavailable, verify the plugin is running and its unix socket path matches configuration
  4. For DeadlineExceeded, increase the controller's timeout or check plugin save performance
Defensive patterns

Strategy: retry

Validate before calling

// Pre-check plugin connectivity before issuing Save:
connState := driver.ConnState() // if exposed, or probe socket:
if _, err := os.Stat(socketPath); err != nil {
    return fmt.Errorf("plugin socket missing: %w", err)
}

Type guard

func grpcUnavailable(err error) bool {
    st, ok := status.FromError(errors.Unwrap(err))
    return ok && st.Code() == codes.Unavailable
}

Try / catch

err := driver.Save(ctx, path, artifact)
if err != nil {
    if grpcUnavailable(err) {
        // backoff and retry: plugin may be restarting
        return retryWithBackoff(func() error { return driver.Save(ctx, path, artifact) })
    }
    return err
}

Prevention

When it happens

Trigger: Calling Driver.Save (workflow/artifacts/plugin/plugin.go:205) when the plugin pod is down/restarting, the unix socket connection broke, the context deadline expired, or the plugin returns a gRPC error status.

Common situations: Plugin OOM-killed during a large artifact save, plugin redeployed mid-workflow, artifact save exceeds gRPC 4MiB message limit for huge inline metadata, or node/network issues on the unix socket.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/a7572b22d74591f4. Report an issue: GitHub.