argoproj/argo-workflows · error

plugin %s stream receive failed: %w

Error message

plugin %s stream receive failed: %w

What it means

Mid-stream failure inside OpenStream's pump goroutine: stream.Recv() returned an error other than io.EOF while reading artifact chunks. The error is delivered to the caller via writer.CloseWithError, so a Read on the returned pipe fails with this message. The stream broke partway through the transfer.

Source

Thrown at workflow/artifacts/plugin/plugin.go:182

	grpcArtifact := convertToGRPC(a)
	stream, err := d.client.OpenStream(ctx, &artifact.OpenStreamRequest{
		Artifact: grpcArtifact,
	})
	if err != nil {
		return nil, fmt.Errorf("plugin %s open stream failed: %w", d.pluginName, err)
	}

	reader, writer := io.Pipe()

	go func() {
		defer writer.Close()
		for {
			resp, err := stream.Recv()
			if errors.Is(err, io.EOF) {
				break
			}
			if err != nil {
				writer.CloseWithError(fmt.Errorf("plugin %s stream receive failed: %w", d.pluginName, err))
				return
			}
			if resp.Error != "" {
				writer.CloseWithError(fmt.Errorf("plugin %s stream error: %s", d.pluginName, resp.Error))
				return
			}
			if resp.IsEnd {
				break
			}
			if len(resp.Data) > 0 {
				if _, writeErr := writer.Write(resp.Data); writeErr != nil {
					writer.CloseWithError(fmt.Errorf("plugin %s stream write failed: %w", d.pluginName, writeErr))
					return
				}
			}
		}
	}()

View on GitHub (pinned to 35bff19146)

Solutions

1. Check plugin logs/pod events at the failure timestamp (OOMKilled, restart)
2. Ensure the reader consuming the pipe respects context cancellation and drains promptly so the stream isn't stalled past its deadline
3. Keep chunk sizes under gRPC's 4MiB default max message size (client and server)
4. Retry the whole OpenStream operation — the pipe cannot be resumed mid-stream
5. Increase the step/node timeout if long streams are being cancelled

Example fix

// before: reading without handling mid-stream error
data, _ := io.ReadAll(rc)
// after
if _, err := io.Copy(dst, rc); err != nil {
    // err is 'plugin <name> stream receive failed: ...' from CloseWithError
    if status.Code(errors.Unwrap(err)) == codes.Unavailable {
        // restart plugin / retry OpenStream from scratch
    }
    return err
}

// prefer a bounded copy so a stalled stream can't hang past your deadline
ctx, cancel := context.WithTimeout(ctx, streamTimeout)
defer cancel()

_, err := io.Copy(dst, rc)
if err != nil {
    // pipe reader surfaces 'plugin %s stream receive failed: ...' from CloseWithError
    var ret *os.PathError
    if !errors.As(err, &ret) && status.Code(context.Cause(ctx)) == codes.DeadlineExceeded {
        return retryFromScratch(ctx, a) // stream not resumable; reopen OpenStream
    }
    return fmt.Errorf("artifact stream aborted: %w", err)
}

When it happens

Trigger: The plugin closes the stream abnormally mid-transfer (crash, OOM kill), the context is cancelled or deadline exceeded during streaming, a gRPC message-size limit is hit, or the network/uds connection drops between chunks.

Common situations: Plugin OOM-killed while streaming a multi-GB artifact, workflow node timeout cancelling ctx mid-stream, plugin hitting gRPC's default 4MiB message limit on an oversized chunk, socket disruption during plugin restart.

Related errors

• plugin %s open stream failed: %w
• failed to dial plugin %s at %q: %w
• plugin %s connection shutdown (socket=%q)
• timeout waiting for plugin %s connection to become ready, la
• plugin %s load failed: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/bd538ff0c894b1c4. Report an issue: GitHub.