apache/beam · error

chunk send failed

Error message

chunk send failed

What it means

GetArtifact streams a GCS blob back to the client over the ArtifactRetrieval gRPC stream in 1MB chunks. This error wraps any failure returned by stream.Send while pushing a chunk, meaning the gRPC connection to the client broke mid-transfer (client cancelled, network dropped, or server-side stream issue). It is not about reading the blob — that path produces 'failed to read from'.

Source

Thrown at sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go:109

	ctx := stream.Context()
	client, err := gcsx.NewClient(ctx, storage.ScopeReadOnly)
	if err != nil {
		return errors.Wrapf(err, "Failed to create client for %v", key)
	}

	// Stream artifact in up to 1MB chunks.
	r, err := client.Bucket(bucket).Object(object).NewReader(ctx)
	if err != nil {
		return errors.Wrapf(err, "Failed to read object for %v", key)
	}
	defer r.Close()

	data := make([]byte, 1<<20)
	for {
		n, err := r.Read(data)
		if n > 0 {
			if err := stream.Send(&jobpb.ArtifactChunk{Data: data[:n]}); err != nil {
				return errors.Wrap(err, "chunk send failed")
			}
		}
		if err == io.EOF {
			break
		}
		if err != nil {
			return errors.Wrapf(err, "failed to read from %v", blob)
		}
	}
	return nil
}

func validate(md *jobpb.ProxyManifest) error {
	keys := make(map[string]bool)
	for _, a := range md.GetManifest().GetArtifact() {
		if _, seen := keys[a.Name]; seen {
			return errors.Errorf("multiple artifact with name %v", a.Name)
		}

View on GitHub (pinned to 12126d8942)

Solutions

1. Check the wrapped err with status.FromContextError / status.Code to identify cancellation, deadline-exceeded, or transport failure and handle each appropriately.
2. Retry GetArtifact from the client side; the read restarts from the beginning of the blob so retries are safe.
3. Verify the client keeps its context alive for the full transfer (don't cancel the ctx before retrieval completes).
4. Reduce artifact size or chunk frequency issues by checking network stability between client and artifact server.

Example fix

// before
if err := stream.Send(&jobpb.ArtifactChunk{Data: data[:n]}); err != nil {
	return errors.Wrap(err, "chunk send failed")
}
// after
if err := stream.Send(&jobpb.ArtifactChunk{Data: data[:n]}); err != nil {
	if st, ok := status.FromError(err); ok && st.Code() == codes.Canceled {
		return status.Error(codes.Canceled, "client canceled artifact retrieval")
	}
	return errors.Wrap(err, "chunk send failed")
}

// Go: nothing to validate pre-call, but ensure ctx has adequate deadline
ctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
defer cancel()

if err := retriever.GetArtifact(ctx, req, stream); err != nil {
	if st, ok := status.FromError(errors.Unwrap(err)); ok {
		switch st.Code() {
		case codes.Canceled, codes.DeadlineExceeded:
			// retry or reschedule retrieval
		default:
			log.Printf("artifact stream failed: %v", err)
		}
	}
}

When it happens

Trigger: stream.Send(&jobpb.ArtifactChunk{Data: data[:n]}) returns non-nil while streaming artifact bytes in GetArtifact, typically when the client disconnects or cancels the RPC context mid-stream, deadlines expire, or the gRPC transport fails.

Common situations: Runner crashes or is killed while fetching artifacts during pipeline submission; network partitions between job submission client and artifact server; client-side context deadline exceeded before the (possibly large) artifact finishes streaming; flaky load balancers terminating idle/large streams.

Related errors

• failed to receive header
• Logging stream terminated unexpectedly before it was closed
• failed to retrieve chunk for %v
• failed to send header for %v
• failed to send chunks for %v; close error: %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/448c17bccf4e078a. Report an issue: GitHub.