{"record":{"id":"ed5f7595db0ddd63","repo":"argoproj/argo-workflows","slug":"plugin-s-save-stream-failed-s-w","errorCode":null,"errorMessage":"plugin %s save stream failed %s: %w","messagePattern":"plugin (.+?) save stream failed (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/artifacts/plugin/plugin.go","lineNumber":262,"sourceCode":"\t// stream themselves) so the server's blocking Recv() is released instead of\n\t// waiting on the caller's ctx, which may outlive this call.\n\tctx, cancel := context.WithCancel(ctx)\n\tdefer cancel()\n\n\tstream, err := d.client.SaveStream(ctx)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"plugin %s save stream failed to open: %w\", d.pluginName, err)\n\t}\n\n\t// Once the server aborts the stream, grpc-go makes Send return io.EOF instead\n\t// of the failure; the plugin's actual error is only retrievable via CloseAndRecv.\n\tsendFrame := func(req *artifact.SaveStreamArtifactRequest, action string) error {\n\t\tsendErr := stream.Send(req)\n\t\tif sendErr == nil {\n\t\t\treturn nil\n\t\t}\n\t\tif _, recvErr := stream.CloseAndRecv(); recvErr != nil {\n\t\t\treturn fmt.Errorf(\"plugin %s save stream failed %s: %w\", d.pluginName, action, recvErr)\n\t\t}\n\t\treturn fmt.Errorf(\"plugin %s save stream failed %s: %w\", d.pluginName, action, sendErr)\n\t}\n\n\tif sendErr := sendFrame(&artifact.SaveStreamArtifactRequest{OutputArtifact: convertToGRPC(outputArtifact)}, \"to send metadata\"); sendErr != nil {\n\t\treturn sendErr\n\t}\n\n\tbuf := make([]byte, saveStreamChunkSize)\n\tfor {\n\t\tn, readErr := reader.Read(buf)\n\t\tif n > 0 {\n\t\t\t// gRPC forbids mutating a message after Send (a lazy stats handler may\n\t\t\t// read it later), so copy the chunk instead of aliasing buf, which the\n\t\t\t// next Read overwrites.\n\t\t\tchunk := make([]byte, n)\n\t\t\tcopy(chunk, buf[:n])\n\t\t\tif sendErr := sendFrame(&artifact.SaveStreamArtifactRequest{Chunk: chunk}, \"mid-transfer\"); sendErr != nil {","sourceCodeStart":244,"sourceCodeEnd":280,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/artifacts/plugin/plugin.go#L244-L280","documentation":"Inside SaveStream's sendFrame helper, a stream.Send failed and, when the driver tried to recover the plugin's real error with stream.CloseAndRecv, that call ALSO returned an error. The recvErr is wrapped here because grpc-go makes Send return only io.EOF after the server aborts; the CloseAndRecv error is the plugin's actual failure cause (or a secondary failure).","triggerScenarios":"Calling Driver.SaveStream where sending metadata ('to send metadata') or a data chunk ('mid-transfer') fails — e.g. the plugin died mid-stream — and CloseAndRecv then also fails to deliver the server's status.","commonSituations":"Plugin pod crashed or was OOM-killed while streaming a large artifact, unix socket connection reset, plugin returned a gRPC error status mid-stream, caller context cancelled by controller shutdown.","solutions":["Inspect the wrapped recvErr — it carries the plugin's termination status (e.g. Unavailable, Canceled, Internal)","Check plugin container logs around the failure time for a crash or explicit error","Retry the artifact save; if it recurs on large artifacts, suspect plugin memory limits or storage timeouts","Ensure the context passed to SaveStream is not being cancelled early by your workflow/controller logic"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"func streamRecvErr(err error) (error, bool) {\n    if strings.Contains(err.Error(), \"save stream failed \") {\n        return errors.Unwrap(err), true\n    }\n    return nil, false\n}","tryCatchPattern":"if err := driver.SaveStream(ctx, reader, artifact); err != nil {\n    var inner error\n    if errors.As(err, &inner) {\n        if st, ok := status.FromError(inner); ok && st.Code() == codes.Unavailable {\n            // plugin died mid-stream: check plugin logs, restart, retry with fresh reader\n        }\n    }\n    return err\n}","preventionTips":["Monitor plugin pod restarts/OOMKills; raise memory limits for large artifact streams","Keep save-phase context deadlines generous relative to artifact size","Treat this as 'plugin aborted mid-transfer' — always inspect the wrapped recvErr","Retry with a fresh reader since bytes already sent cannot be replayed"],"tags":["grpc","plugin","streaming","artifacts"],"backgroundTag":"grpc-stream-aborted","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}