apache/beam · error

failed to retrieve %v in %v attempts: %v

Error message

failed to retrieve %v in %v attempts: %v

What it means

MultiRetrieve sets this error (via errorx.GuardedError) when an artifact fails to be retrieved more than 3 times; all individual failure messages are joined with ';' into one aggregate message. It means the artifact could not be fetched from the legacy artifact retrieval service after retries with 1-5 second randomized backoff.

Source

Thrown at sdks/go/pkg/beam/artifact/materialize.go:371

		wg.Add(1)
		go func() {
			defer wg.Done()
			for a := range q {
				if permErr.Error() != nil {
					continue
				}

				const attempts = 3

				var failures []string
				for {
					err := a.retrieve(ctx, dest)
					if err == nil || permErr.Error() != nil {
						break // done or give up
					}
					failures = append(failures, err.Error())
					if len(failures) > attempts {
						permErr.TrySetError(errors.Errorf("failed to retrieve %v in %v attempts: %v", dest, attempts, strings.Join(failures, "; ")))
						break // give up
					}
					time.Sleep(time.Duration(rand.Intn(5)+1) * time.Second)
				}
			}
		}()
	}
	wg.Wait()

	return permErr.Error()
}

type retrievable interface {
	retrieve(ctx context.Context, dest string) error
}

// LegacyMultiRetrieve is exported for testing.
func LegacyMultiRetrieve(ctx context.Context, client jobpb.LegacyArtifactRetrievalServiceClient, cpus int, list []*jobpb.ArtifactMetadata, rt string, dest string) error {

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the joined sub-errors in the message to find the root cause (gRPC error vs SHA mismatch vs file error).
2. Verify the retrieval token is fresh and the artifact service endpoint is reachable from the worker.
3. Check disk permissions and free space at the destination directory.
4. Retry the job; the library already does 3 attempts with jittered backoff, so persistent failure means a systemic issue, not transient flake.

conn, err := grpc.DialContext(ctx, endpoint, grpc.WithBlock(), grpc.WithTimeout(5*time.Second))
if err != nil {
    return fmt.Errorf("artifact service unreachable before retrieval: %w", err)
}
conn.Close()

err := artifact.Materialize(ctx, endpoint, rt, dest)
if err != nil {
    var backoff = time.Second
    for i := 0; i < 2 && err != nil; i++ {
        time.Sleep(backoff)
        backoff *= 2
        err = artifact.Materialize(ctx, endpoint, rt, dest)
    }
}

When it happens

Trigger: Calling Materialize/Retrieve/LegacyMultiRetrieve where `a.retrieve(ctx, dest)` (i.e. Retrieve) fails on attempt 4+, accumulating len(failures) > attempts=3 — e.g. repeated gRPC GetArtifact stream errors, bad retrieval token, or SHA mismatch on every attempt.

Common situations: Artifact service endpoint unreachable or flaky network during pipeline startup; stale/invalid retrieval token after a runner restart; corrupt artifact on the server causing persistent bad-SHA256 failures; dest directory permissions blocking writes on every retry.

Related errors

• failed to stage %v in %v attempts: %v
• failed to send header for %v
• Logging stream terminated unexpectedly before it was closed
• Failed to call Rate Limit Service
• Failed to get response from Rate Limit Service

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/dd74e26eedc47c40. Report an issue: GitHub.