{"record":{"id":"448c17bccf4e078a","repo":"apache/beam","slug":"chunk-send-failed","errorCode":null,"errorMessage":"chunk send failed","messagePattern":"chunk send failed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go","lineNumber":109,"sourceCode":"\tctx := stream.Context()\n\tclient, err := gcsx.NewClient(ctx, storage.ScopeReadOnly)\n\tif err != nil {\n\t\treturn errors.Wrapf(err, \"Failed to create client for %v\", key)\n\t}\n\n\t// Stream artifact in up to 1MB chunks.\n\tr, err := client.Bucket(bucket).Object(object).NewReader(ctx)\n\tif err != nil {\n\t\treturn errors.Wrapf(err, \"Failed to read object for %v\", key)\n\t}\n\tdefer r.Close()\n\n\tdata := make([]byte, 1<<20)\n\tfor {\n\t\tn, err := r.Read(data)\n\t\tif n > 0 {\n\t\t\tif err := stream.Send(&jobpb.ArtifactChunk{Data: data[:n]}); err != nil {\n\t\t\t\treturn errors.Wrap(err, \"chunk send failed\")\n\t\t\t}\n\t\t}\n\t\tif err == io.EOF {\n\t\t\tbreak\n\t\t}\n\t\tif err != nil {\n\t\t\treturn errors.Wrapf(err, \"failed to read from %v\", blob)\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc validate(md *jobpb.ProxyManifest) error {\n\tkeys := make(map[string]bool)\n\tfor _, a := range md.GetManifest().GetArtifact() {\n\t\tif _, seen := keys[a.Name]; seen {\n\t\t\treturn errors.Errorf(\"multiple artifact with name %v\", a.Name)\n\t\t}","sourceCodeStart":91,"sourceCodeEnd":127,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go#L91-L127","documentation":"GetArtifact streams a GCS blob back to the client over the ArtifactRetrieval gRPC stream in 1MB chunks. This error wraps any failure returned by stream.Send while pushing a chunk, meaning the gRPC connection to the client broke mid-transfer (client cancelled, network dropped, or server-side stream issue). It is not about reading the blob — that path produces 'failed to read from'.","triggerScenarios":"stream.Send(&jobpb.ArtifactChunk{Data: data[:n]}) returns non-nil while streaming artifact bytes in GetArtifact, typically when the client disconnects or cancels the RPC context mid-stream, deadlines expire, or the gRPC transport fails.","commonSituations":"Runner crashes or is killed while fetching artifacts during pipeline submission; network partitions between job submission client and artifact server; client-side context deadline exceeded before the (possibly large) artifact finishes streaming; flaky load balancers terminating idle/large streams.","solutions":["Check the wrapped err with status.FromContextError / status.Code to identify cancellation, deadline-exceeded, or transport failure and handle each appropriately.","Retry GetArtifact from the client side; the read restarts from the beginning of the blob so retries are safe.","Verify the client keeps its context alive for the full transfer (don't cancel the ctx before retrieval completes).","Reduce artifact size or chunk frequency issues by checking network stability between client and artifact server."],"exampleFix":"// before\nif err := stream.Send(&jobpb.ArtifactChunk{Data: data[:n]}); err != nil {\n\treturn errors.Wrap(err, \"chunk send failed\")\n}\n// after\nif err := stream.Send(&jobpb.ArtifactChunk{Data: data[:n]}); err != nil {\n\tif st, ok := status.FromError(err); ok && st.Code() == codes.Canceled {\n\t\treturn status.Error(codes.Canceled, \"client canceled artifact retrieval\")\n\t}\n\treturn errors.Wrap(err, \"chunk send failed\")\n}","handlingStrategy":"try-catch","validationCode":"// Go: nothing to validate pre-call, but ensure ctx has adequate deadline\nctx, cancel := context.WithTimeout(ctx, 10*time.Minute)\ndefer cancel()","typeGuard":null,"tryCatchPattern":"if err := retriever.GetArtifact(ctx, req, stream); err != nil {\n\tif st, ok := status.FromError(errors.Unwrap(err)); ok {\n\t\tswitch st.Code() {\n\t\tcase codes.Canceled, codes.DeadlineExceeded:\n\t\t\t// retry or reschedule retrieval\n\t\tdefault:\n\t\t\tlog.Printf(\"artifact stream failed: %v\", err)\n\t\t}\n\t}\n}","preventionTips":["Give artifact retrieval RPCs generous deadlines proportional to artifact size.","Keep the client context alive until streaming completes; never cancel early.","Retry retrieval on transport errors — reads restart from scratch, so it's idempotent.","Monitor network stability between submission client and artifact server."],"tags":["grpc","gcs","streaming","network"],"backgroundTag":"network-request-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}