What causes "plugin %s save stream failed: %w" in argoproj/argo-workflows?

Decode the wrapped gRPC status from CloseAndRecv for the specific failure code

plugin %s save stream failed: %w

Error message

plugin %s save stream failed: %w

What it means

All chunks were sent successfully, but the final stream.CloseAndRecv — which blocks for the plugin's SaveResponse — returned a gRPC/transport error instead of a response. This means the plugin failed (or the connection broke) after receiving the full artifact, during its commit/finalize phase.

Source

Thrown at workflow/artifacts/plugin/plugin.go:294

			// read it later), so copy the chunk instead of aliasing buf, which the
			// next Read overwrites.
			chunk := make([]byte, n)
			copy(chunk, buf[:n])
			if sendErr := sendFrame(&artifact.SaveStreamArtifactRequest{Chunk: chunk}, "mid-transfer"); sendErr != nil {
				return sendErr
			}
		}
		if errors.Is(readErr, io.EOF) {
			break
		}
		if readErr != nil {
			return fmt.Errorf("plugin %s save stream failed to read artifact content: %w", d.pluginName, readErr)
		}
	}

	resp, err := stream.CloseAndRecv()
	if err != nil {
		return fmt.Errorf("plugin %s save stream failed: %w", d.pluginName, err)
	}
	if !resp.Success {
		return fmt.Errorf("plugin %s save stream failed: %s", d.pluginName, resp.Error)
	}
	return nil
}

// supportsSaveStream reports whether the plugin advertises streaming support.
// A plugin that predates GetCapabilities returns codes.Unimplemented, which maps to
// (false, nil) so SaveStream falls back to the buffered Save. Any other error is
// returned rather than silently downgrading to buffering a potentially large artifact
// to disk before the real failure would resurface via Save.
func (d *Driver) supportsSaveStream(ctx context.Context) (bool, error) {
	resp, err := d.client.GetCapabilities(ctx, &artifact.GetCapabilitiesRequest{})
	if err != nil {
		if status.Code(err) == codes.Unimplemented {
			return false, nil
		}

View on GitHub (pinned to 35bff19146)

Solutions

Decode the wrapped gRPC status from CloseAndRecv for the specific failure code
Check plugin logs for errors during finalization/commit of the upload
Retry the save — plugin-side finalize failures are frequently transient
If timeouts recur on large artifacts, increase plugin-side finalization timeouts

Defensive patterns

Strategy: retry

Type guard

func finalizeFailed(err error) bool {
    return strings.Contains(err.Error(), "save stream failed:") && !strings.Contains(err.Error(), "failed ")
}

Try / catch

err := driver.SaveStream(ctx, reader, artifact)
if err != nil {
    if st, ok := status.FromError(errors.Unwrap(err)); ok && isRetryable(st.Code()) {
        // finalize-phase failure after full upload: retry with a fresh reader
        return retryWithBackoff(func() error {
            r, _ := regenerateReader()
            return driver.SaveStream(ctx, r, artifact)
        })
    }
    return err
}

Prevention

Give plugins adequate finalization timeouts for very large artifacts
Monitor plugin logs during the commit/multipart-complete phase
Retry finalize failures — full data transfer succeeded, so they are often transient
Keep the unix socket stable during long saves (avoid plugin redeploys mid-workflow)

When it happens

Trigger: Calling Driver.SaveStream where the entire reader was consumed and sent without error, but CloseAndRecv fails — the plugin errored while finalizing (e.g. completing a multipart upload) or the stream status was non-OK.

Common situations: Plugin-side multipart upload completion failure, plugin timeout during finalization of very large artifacts, plugin crash after receiving all bytes, socket reset during the final handshake.

Related errors

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/9c2f9d6707e4646f. Report an issue: GitHub.