apache/beam · error

failed to retrieve chunk for %v

Error message

failed to retrieve chunk for %v

What it means

Wraps any error from writeChunks during streaming download in artifact.retrieve: a gRPC stream Recv failed mid-transfer, or a chunk could not be written to the buffered writer. The artifact file is closed and partially downloaded content is abandoned. The filename is included in the message.

Source

Thrown at sdks/go/pkg/beam/artifact/materialize.go:244

	if err := os.MkdirAll(filepath.Dir(filename), os.ModePerm); err != nil {
		return err
	}

	stream, err := a.client.GetArtifact(ctx, &jobpb.GetArtifactRequest{Artifact: a.dep})
	if err != nil {
		return err
	}

	fd, err := os.OpenFile(filename, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0755)
	if err != nil {
		return err
	}
	w := bufio.NewWriter(fd)

	sha256Hash, err := writeChunks(stream, w)
	if err != nil {
		fd.Close() // drop any buffered content
		return errors.Wrapf(err, "failed to retrieve chunk for %v", filename)
	}
	if err := w.Flush(); err != nil {
		fd.Close()
		return errors.Wrapf(err, "failed to flush chunks for %v", filename)
	}
	stat, _ := fd.Stat()
	log.Printf("Downloaded: %v (sha256: %v, size: %v)", filename, sha256Hash, stat.Size())

	if err := fd.Close(); err != nil {
		return err
	}

	if isArtifactValidationEnabled(ctx) {
		if a.expectedSha256 == "" {
			log.Printf("WARN: Artifact validation skipped for file: %v", filename)
		} else if sha256Hash != a.expectedSha256 {
			return errors.Errorf("bad SHA256 for %v: %v, want %v", filename, sha256Hash, a.expectedSha256)
		}

View on GitHub (pinned to 12126d8942)

Solutions

1. MultiRetrieve already retries up to 3 times; verify the artifact service is healthy and the network is stable, then rerun the pipeline.
2. Check free disk space on the worker for dest and enlarge the volume or clean old artifacts.
3. Increase gRPC/context deadlines if large artifacts are timing out.
4. Confirm the artifact server endpoint is reachable and not behind a proxy dropping long streams.

Example fix

// before: default ctx without deadline control
ctx := context.Background()
// after: give retrieval ample time
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
defer cancel()

// Check endpoint reachability and disk headroom before retrieval:
conn, err := net.DialTimeout("tcp", endpointHost, 5*time.Second)
if err != nil { return err }
conn.Close()
if st, err := os.Stat(dest); err != nil || !st.IsDir() { return fmt.Errorf("dest missing: %s", dest) }

err := artifact.Materialize(ctx, endpoint, deps, rt, dest)
for attempt := 0; err != nil && attempt < 3; attempt++ {
	if strings.Contains(err.Error(), "failed to retrieve chunk") {
		time.Sleep(time.Duration(attempt+1) * 10 * time.Second) // backoff; transient stream drop
		err = artifact.Materialize(ctx, endpoint, deps, rt, dest)
		continue
	}
	break
}

When it happens

Trigger: Calling Materialize/Retrieve where the GetArtifact server stream errors mid-transfer: artifact staging service restarted, network drop between worker and staging endpoint, gRPC deadline exceeded, or the local disk filled so bufio writes failed.

Common situations: Unstable network between worker and artifact server; large artifacts exceeding deadlines; the staging service crashing while serving; disk quota/full volume on the worker node.

Related errors

• failed to get manifest
• failed to send chunks for %v; close error: %v
• chunk send failed
• Logging stream terminated unexpectedly before it was closed
• chunk send failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2ecab1d9b8e6ba3d. Report an issue: GitHub.