{"record":{"id":"9c2f9d6707e4646f","repo":"argoproj/argo-workflows","slug":"plugin-s-save-stream-failed-w","errorCode":null,"errorMessage":"plugin %s save stream failed: %w","messagePattern":"plugin (.+?) save stream failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/artifacts/plugin/plugin.go","lineNumber":294,"sourceCode":"\t\t\t// read it later), so copy the chunk instead of aliasing buf, which the\n\t\t\t// next Read overwrites.\n\t\t\tchunk := make([]byte, n)\n\t\t\tcopy(chunk, buf[:n])\n\t\t\tif sendErr := sendFrame(&artifact.SaveStreamArtifactRequest{Chunk: chunk}, \"mid-transfer\"); sendErr != nil {\n\t\t\t\treturn sendErr\n\t\t\t}\n\t\t}\n\t\tif errors.Is(readErr, io.EOF) {\n\t\t\tbreak\n\t\t}\n\t\tif readErr != nil {\n\t\t\treturn fmt.Errorf(\"plugin %s save stream failed to read artifact content: %w\", d.pluginName, readErr)\n\t\t}\n\t}\n\n\tresp, err := stream.CloseAndRecv()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"plugin %s save stream failed: %w\", d.pluginName, err)\n\t}\n\tif !resp.Success {\n\t\treturn fmt.Errorf(\"plugin %s save stream failed: %s\", d.pluginName, resp.Error)\n\t}\n\treturn nil\n}\n\n// supportsSaveStream reports whether the plugin advertises streaming support.\n// A plugin that predates GetCapabilities returns codes.Unimplemented, which maps to\n// (false, nil) so SaveStream falls back to the buffered Save. Any other error is\n// returned rather than silently downgrading to buffering a potentially large artifact\n// to disk before the real failure would resurface via Save.\nfunc (d *Driver) supportsSaveStream(ctx context.Context) (bool, error) {\n\tresp, err := d.client.GetCapabilities(ctx, &artifact.GetCapabilitiesRequest{})\n\tif err != nil {\n\t\tif status.Code(err) == codes.Unimplemented {\n\t\t\treturn false, nil\n\t\t}","sourceCodeStart":276,"sourceCodeEnd":312,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/artifacts/plugin/plugin.go#L276-L312","documentation":"All chunks were sent successfully, but the final stream.CloseAndRecv — which blocks for the plugin's SaveResponse — returned a gRPC/transport error instead of a response. This means the plugin failed (or the connection broke) after receiving the full artifact, during its commit/finalize phase.","triggerScenarios":"Calling Driver.SaveStream where the entire reader was consumed and sent without error, but CloseAndRecv fails — the plugin errored while finalizing (e.g. completing a multipart upload) or the stream status was non-OK.","commonSituations":"Plugin-side multipart upload completion failure, plugin timeout during finalization of very large artifacts, plugin crash after receiving all bytes, socket reset during the final handshake.","solutions":["Decode the wrapped gRPC status from CloseAndRecv for the specific failure code","Check plugin logs for errors during finalization/commit of the upload","Retry the save — plugin-side finalize failures are frequently transient","If timeouts recur on large artifacts, increase plugin-side finalization timeouts"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":"func finalizeFailed(err error) bool {\n    return strings.Contains(err.Error(), \"save stream failed:\") && !strings.Contains(err.Error(), \"failed \")\n}","tryCatchPattern":"err := driver.SaveStream(ctx, reader, artifact)\nif err != nil {\n    if st, ok := status.FromError(errors.Unwrap(err)); ok && isRetryable(st.Code()) {\n        // finalize-phase failure after full upload: retry with a fresh reader\n        return retryWithBackoff(func() error {\n            r, _ := regenerateReader()\n            return driver.SaveStream(ctx, r, artifact)\n        })\n    }\n    return err\n}","preventionTips":["Give plugins adequate finalization timeouts for very large artifacts","Monitor plugin logs during the commit/multipart-complete phase","Retry finalize failures — full data transfer succeeded, so they are often transient","Keep the unix socket stable during long saves (avoid plugin redeploys mid-workflow)"],"tags":["grpc","plugin","streaming","artifacts"],"backgroundTag":"grpc-stream-finalize-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}