What causes "plugin %s save stream failed to open: %w" in argoproj/argo-workflows?

Decode the wrapped gRPC status for the specific code (Unavailable = plugin down, Unimplemented = version mismatch)

plugin %s save stream failed to open: %w

Error message

plugin %s save stream failed to open: %w

What it means

SaveStream could not even open the bidirectional SaveStream gRPC stream to the plugin. This happens before any data is sent, so it is a connection/RPC-establishment failure with the plugin service over its unix socket.

Source

Thrown at workflow/artifacts/plugin/plugin.go:251

// partially consumed and cannot be rewound.
func (d *Driver) SaveStream(ctx context.Context, reader io.Reader, outputArtifact *wfv1.Artifact) error {
	supported, err := d.supportsSaveStream(ctx)
	if err != nil {
		return err
	}
	if !supported {
		return d.saveStreamViaTempFile(ctx, reader, outputArtifact)
	}

	// Cancelled on every return path (including reader errors, which don't close the
	// stream themselves) so the server's blocking Recv() is released instead of
	// waiting on the caller's ctx, which may outlive this call.
	ctx, cancel := context.WithCancel(ctx)
	defer cancel()

	stream, err := d.client.SaveStream(ctx)
	if err != nil {
		return fmt.Errorf("plugin %s save stream failed to open: %w", d.pluginName, err)
	}

	// Once the server aborts the stream, grpc-go makes Send return io.EOF instead
	// of the failure; the plugin's actual error is only retrievable via CloseAndRecv.
	sendFrame := func(req *artifact.SaveStreamArtifactRequest, action string) error {
		sendErr := stream.Send(req)
		if sendErr == nil {
			return nil
		}
		if _, recvErr := stream.CloseAndRecv(); recvErr != nil {
			return fmt.Errorf("plugin %s save stream failed %s: %w", d.pluginName, action, recvErr)
		}
		return fmt.Errorf("plugin %s save stream failed %s: %w", d.pluginName, action, sendErr)
	}

	if sendErr := sendFrame(&artifact.SaveStreamArtifactRequest{OutputArtifact: convertToGRPC(outputArtifact)}, "to send metadata"); sendErr != nil {
		return sendErr
	}

View on GitHub (pinned to 35bff19146)

Solutions

Decode the wrapped gRPC status for the specific code (Unavailable = plugin down, Unimplemented = version mismatch)
Verify the plugin pod is running and the unix socket exists (ls -la on the socket path)
Confirm the deployed plugin build actually implements SaveStream and matches its advertised capabilities
Retry the artifact save once the plugin is healthy

Defensive patterns

Strategy: retry

Validate before calling

// Verify the plugin socket exists and the plugin responds before SaveStream:
if _, err := os.Stat(socketPath); err != nil {
    return fmt.Errorf("plugin socket %s not present: %w", socketPath, err)
}

Type guard

func streamOpenFailed(err error) bool {
    return strings.Contains(err.Error(), "save stream failed to open")
}

Try / catch

err := driver.SaveStream(ctx, reader, artifact)
if err != nil {
    if streamOpenFailed(err) && grpcUnavailable(err) {
        // plugin likely restarting; wait and retry with a FRESH reader
        time.Sleep(backoff)
        return reopenReaderAndSave()
    }
    return err
}

Prevention

Check plugin pod health before long artifact-save phases
Align plugin builds so advertised capabilities match implemented RPCs
Avoid cancelling the caller context during save setup
Retry with a fresh reader — the original reader may have been partially consumed

When it happens

Trigger: Calling Driver.SaveStream when the plugin connection is broken (pod restarting, socket removed), the method is unimplemented despite the capability check passing (version skew), or the per-call context was already cancelled.

Common situations: Plugin restarted between the GetCapabilities check and SaveStream call, plugin build advertises streaming capability but lacks the RPC (custom build mismatch), controller-to-plugin socket deleted during plugin shutdown.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/45d9a679e025f831. Report an issue: GitHub.