apache/beam · error

chunk send failed

Error message

chunk send failed

What it means

stageChunks reads the local file in 1MB chunks and sends each as a PutArtifactRequest over the gRPC stream. If stream.Send returns io.EOF the stream was closed by the server (returned verbatim); any other non-nil error is wrapped as 'chunk send failed'. This indicates the gRPC client stream to the artifact staging service broke while transferring artifact data.

Source

Thrown at sdks/go/pkg/beam/artifact/stage.go:196

		n, err := r.Read(data)
		if n > 0 {
			if _, err := sha256W.Write(data[:n]); err != nil {
				panic(err) // cannot fail
			}

			chunk := &jobpb.PutArtifactRequest{
				Content: &jobpb.PutArtifactRequest_Data{
					Data: &jobpb.ArtifactChunk{
						Data: data[:n],
					},
				},
			}
			err := stream.Send(chunk)
			if err == io.EOF {
				return "", err
			}
			if err != nil {
				return "", errors.Wrap(err, "chunk send failed")
			}
		}
		if err == io.EOF {
			break
		}
		if err != nil {
			return "", err
		}
	}
	return hex.EncodeToString(sha256W.Sum(nil)), nil
}

// KeyedFile is a key and filename pair.
type KeyedFile struct {
	Key, Filename string
}

func scan(dir string) ([]KeyedFile, error) {

View on GitHub (pinned to 12126d8942)

Solutions

1. Read the wrapped gRPC status from the error (errors.Unwrap + status.Code) to distinguish transient (Unavailable, DeadlineExceeded) from fatal (InvalidArgument, Unauthenticated) causes.
2. Let MultiStage retry — it retries Stage 3 times with jittered backoff for transient failures; persistent failures need the root cause fixed.
3. Increase the context timeout for large artifacts and confirm no proxy/firewall is killing long-lived gRPC streams (e.g. idle/connection timeouts).
4. Verify the staging session token is valid for the duration of the upload.

Example fix

// before
err := stream.Send(chunk)
if err == io.EOF {
	return "", err
}
if err != nil {
	return "", errors.Wrap(err, "chunk send failed")
}
// after (caller-side: retry only transient send failures)
var lastErr error
for attempt := 0; attempt < 3; attempt++ {
	_, err := artifact.Stage(ctx, client, key, filename, token)
	if err == nil {
		break
	}
	if st, ok := status.FromError(errors.Unwrap(err)); ok &&
		(st.Code() == codes.Unavailable || st.Code() == codes.DeadlineExceeded) {
		lastErr = err
		time.Sleep(time.Duration(attempt+1) * time.Second)
		continue
	}
	return err
}
_ = lastErr

// pre-check connectivity and context budget
if ctx.Err() != nil { return ctx.Err() }
if _, err := grpc.DialContext(ctx, addr, grpc.WithBlock(), grpc.WithTransportCredentials(insecure.NewCredentials())); err != nil {
	return fmt.Errorf("cannot reach artifact staging service: %w", err)
}

_, err := artifact.Stage(ctx, client, key, filename, token)
if err != nil && strings.Contains(err.Error(), "chunk send failed") {
	if st, ok := status.FromError(errors.Unwrap(err)); ok && (st.Code() == codes.Unavailable || st.Code() == codes.DeadlineExceeded) {
		// retry with exponential backoff; MultiStage does this internally
	}
}

When it happens

Trigger: stream.Send(chunk) returns a non-EOF error mid-upload: server cancelled the PutArtifact RPC (invalid staging session token, metadata rejection), transport failure/connection reset, context deadline or cancellation, or message size limits on very large files.

Common situations: Network partition between the runner and the artifact service; artifact service pod restart (e.g. Kubernetes) mid-upload; context timeout too short for large files; server-side cancellation because the staging session token was revoked or the manifest commit already happened.

Related errors

• failed to send chunks for %v; close error: %v
• failed to close stream for %v; response: %v
• failed to receive header
• failed to retrieve chunk for %v
• failed to connect to state service %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/47dc0ef9d875e0e3. Report an issue: GitHub.