argoproj/argo-workflows · error

plugin %s save stream failed %s: %w

Error message

plugin %s save stream failed %s: %w

What it means

Inside SaveStream's sendFrame helper, a stream.Send failed and, when the driver tried to recover the plugin's real error with stream.CloseAndRecv, that call ALSO returned an error. The recvErr is wrapped here because grpc-go makes Send return only io.EOF after the server aborts; the CloseAndRecv error is the plugin's actual failure cause (or a secondary failure).

Source

Thrown at workflow/artifacts/plugin/plugin.go:262

	// stream themselves) so the server's blocking Recv() is released instead of
	// waiting on the caller's ctx, which may outlive this call.
	ctx, cancel := context.WithCancel(ctx)
	defer cancel()

	stream, err := d.client.SaveStream(ctx)
	if err != nil {
		return fmt.Errorf("plugin %s save stream failed to open: %w", d.pluginName, err)
	}

	// Once the server aborts the stream, grpc-go makes Send return io.EOF instead
	// of the failure; the plugin's actual error is only retrievable via CloseAndRecv.
	sendFrame := func(req *artifact.SaveStreamArtifactRequest, action string) error {
		sendErr := stream.Send(req)
		if sendErr == nil {
			return nil
		}
		if _, recvErr := stream.CloseAndRecv(); recvErr != nil {
			return fmt.Errorf("plugin %s save stream failed %s: %w", d.pluginName, action, recvErr)
		}
		return fmt.Errorf("plugin %s save stream failed %s: %w", d.pluginName, action, sendErr)
	}

	if sendErr := sendFrame(&artifact.SaveStreamArtifactRequest{OutputArtifact: convertToGRPC(outputArtifact)}, "to send metadata"); sendErr != nil {
		return sendErr
	}

	buf := make([]byte, saveStreamChunkSize)
	for {
		n, readErr := reader.Read(buf)
		if n > 0 {
			// gRPC forbids mutating a message after Send (a lazy stats handler may
			// read it later), so copy the chunk instead of aliasing buf, which the
			// next Read overwrites.
			chunk := make([]byte, n)
			copy(chunk, buf[:n])
			if sendErr := sendFrame(&artifact.SaveStreamArtifactRequest{Chunk: chunk}, "mid-transfer"); sendErr != nil {

View on GitHub (pinned to 35bff19146)

Solutions

1. Inspect the wrapped recvErr — it carries the plugin's termination status (e.g. Unavailable, Canceled, Internal)
2. Check plugin container logs around the failure time for a crash or explicit error
3. Retry the artifact save; if it recurs on large artifacts, suspect plugin memory limits or storage timeouts
4. Ensure the context passed to SaveStream is not being cancelled early by your workflow/controller logic

func streamRecvErr(err error) (error, bool) {
    if strings.Contains(err.Error(), "save stream failed ") {
        return errors.Unwrap(err), true
    }
    return nil, false
}

if err := driver.SaveStream(ctx, reader, artifact); err != nil {
    var inner error
    if errors.As(err, &inner) {
        if st, ok := status.FromError(inner); ok && st.Code() == codes.Unavailable {
            // plugin died mid-stream: check plugin logs, restart, retry with fresh reader
        }
    }
    return err
}

When it happens

Trigger: Calling Driver.SaveStream where sending metadata ('to send metadata') or a data chunk ('mid-transfer') fails — e.g. the plugin died mid-stream — and CloseAndRecv then also fails to deliver the server's status.

Common situations: Plugin pod crashed or was OOM-killed while streaming a large artifact, unix socket connection reset, plugin returned a gRPC error status mid-stream, caller context cancelled by controller shutdown.

Related errors

• plugin %s stream error: %s
• plugin %s save stream failed to open: %w
• plugin %s save stream failed: %w
• plugin %s save failed: %w
• plugin %s save stream failed to read artifact content: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/ed5f7595db0ddd63. Report an issue: GitHub.