{"record":{"id":"bd538ff0c894b1c4","repo":"argoproj/argo-workflows","slug":"plugin-s-stream-receive-failed-w","errorCode":null,"errorMessage":"plugin %s stream receive failed: %w","messagePattern":"plugin (.+?) stream receive failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/artifacts/plugin/plugin.go","lineNumber":182,"sourceCode":"\tgrpcArtifact := convertToGRPC(a)\n\tstream, err := d.client.OpenStream(ctx, &artifact.OpenStreamRequest{\n\t\tArtifact: grpcArtifact,\n\t})\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"plugin %s open stream failed: %w\", d.pluginName, err)\n\t}\n\n\treader, writer := io.Pipe()\n\n\tgo func() {\n\t\tdefer writer.Close()\n\t\tfor {\n\t\t\tresp, err := stream.Recv()\n\t\t\tif errors.Is(err, io.EOF) {\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tif err != nil {\n\t\t\t\twriter.CloseWithError(fmt.Errorf(\"plugin %s stream receive failed: %w\", d.pluginName, err))\n\t\t\t\treturn\n\t\t\t}\n\t\t\tif resp.Error != \"\" {\n\t\t\t\twriter.CloseWithError(fmt.Errorf(\"plugin %s stream error: %s\", d.pluginName, resp.Error))\n\t\t\t\treturn\n\t\t\t}\n\t\t\tif resp.IsEnd {\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tif len(resp.Data) > 0 {\n\t\t\t\tif _, writeErr := writer.Write(resp.Data); writeErr != nil {\n\t\t\t\t\twriter.CloseWithError(fmt.Errorf(\"plugin %s stream write failed: %w\", d.pluginName, writeErr))\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n","sourceCodeStart":164,"sourceCodeEnd":200,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/artifacts/plugin/plugin.go#L164-L200","documentation":"Mid-stream failure inside OpenStream's pump goroutine: stream.Recv() returned an error other than io.EOF while reading artifact chunks. The error is delivered to the caller via writer.CloseWithError, so a Read on the returned pipe fails with this message. The stream broke partway through the transfer.","triggerScenarios":"The plugin closes the stream abnormally mid-transfer (crash, OOM kill), the context is cancelled or deadline exceeded during streaming, a gRPC message-size limit is hit, or the network/uds connection drops between chunks.","commonSituations":"Plugin OOM-killed while streaming a multi-GB artifact, workflow node timeout cancelling ctx mid-stream, plugin hitting gRPC's default 4MiB message limit on an oversized chunk, socket disruption during plugin restart.","solutions":["Check plugin logs/pod events at the failure timestamp (OOMKilled, restart)","Ensure the reader consuming the pipe respects context cancellation and drains promptly so the stream isn't stalled past its deadline","Keep chunk sizes under gRPC's 4MiB default max message size (client and server)","Retry the whole OpenStream operation — the pipe cannot be resumed mid-stream","Increase the step/node timeout if long streams are being cancelled"],"exampleFix":"// before: reading without handling mid-stream error\ndata, _ := io.ReadAll(rc)\n// after\nif _, err := io.Copy(dst, rc); err != nil {\n    // err is 'plugin <name> stream receive failed: ...' from CloseWithError\n    if status.Code(errors.Unwrap(err)) == codes.Unavailable {\n        // restart plugin / retry OpenStream from scratch\n    }\n    return err\n}","handlingStrategy":"try-catch","validationCode":"// prefer a bounded copy so a stalled stream can't hang past your deadline\nctx, cancel := context.WithTimeout(ctx, streamTimeout)\ndefer cancel()","typeGuard":null,"tryCatchPattern":"_, err := io.Copy(dst, rc)\nif err != nil {\n    // pipe reader surfaces 'plugin %s stream receive failed: ...' from CloseWithError\n    var ret *os.PathError\n    if !errors.As(err, &ret) && status.Code(context.Cause(ctx)) == codes.DeadlineExceeded {\n        return retryFromScratch(ctx, a) // stream not resumable; reopen OpenStream\n    }\n    return fmt.Errorf(\"artifact stream aborted: %w\", err)\n}","preventionTips":["Keep streamed chunks under gRPC's 4MiB default message limit on both sides","Size node timeouts to cover the full artifact transfer plus slack","Watch plugin memory limits; OOM kills mid-stream show up as this error on the consumer side","Treat streamed transfers as all-or-nothing: always retry by reopening the stream, never resume a broken pipe"],"tags":["grpc","streaming","mid-stream-failure","io-pipe","plugin-driver"],"backgroundTag":"grpc-stream-broken","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}