{"record":{"id":"dd74e26eedc47c40","repo":"apache/beam","slug":"failed-to-retrieve-v-in-v-attempts-v","errorCode":null,"errorMessage":"failed to retrieve %v in %v attempts: %v","messagePattern":"failed to retrieve (.+?) in (.+?) attempts: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/artifact/materialize.go","lineNumber":371,"sourceCode":"\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tdefer wg.Done()\n\t\t\tfor a := range q {\n\t\t\t\tif permErr.Error() != nil {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\n\t\t\t\tconst attempts = 3\n\n\t\t\t\tvar failures []string\n\t\t\t\tfor {\n\t\t\t\t\terr := a.retrieve(ctx, dest)\n\t\t\t\t\tif err == nil || permErr.Error() != nil {\n\t\t\t\t\t\tbreak // done or give up\n\t\t\t\t\t}\n\t\t\t\t\tfailures = append(failures, err.Error())\n\t\t\t\t\tif len(failures) > attempts {\n\t\t\t\t\t\tpermErr.TrySetError(errors.Errorf(\"failed to retrieve %v in %v attempts: %v\", dest, attempts, strings.Join(failures, \"; \")))\n\t\t\t\t\t\tbreak // give up\n\t\t\t\t\t}\n\t\t\t\t\ttime.Sleep(time.Duration(rand.Intn(5)+1) * time.Second)\n\t\t\t\t}\n\t\t\t}\n\t\t}()\n\t}\n\twg.Wait()\n\n\treturn permErr.Error()\n}\n\ntype retrievable interface {\n\tretrieve(ctx context.Context, dest string) error\n}\n\n// LegacyMultiRetrieve is exported for testing.\nfunc LegacyMultiRetrieve(ctx context.Context, client jobpb.LegacyArtifactRetrievalServiceClient, cpus int, list []*jobpb.ArtifactMetadata, rt string, dest string) error {","sourceCodeStart":353,"sourceCodeEnd":389,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/artifact/materialize.go#L353-L389","documentation":"MultiRetrieve sets this error (via errorx.GuardedError) when an artifact fails to be retrieved more than 3 times; all individual failure messages are joined with ';' into one aggregate message. It means the artifact could not be fetched from the legacy artifact retrieval service after retries with 1-5 second randomized backoff.","triggerScenarios":"Calling Materialize/Retrieve/LegacyMultiRetrieve where `a.retrieve(ctx, dest)` (i.e. Retrieve) fails on attempt 4+, accumulating len(failures) > attempts=3 — e.g. repeated gRPC GetArtifact stream errors, bad retrieval token, or SHA mismatch on every attempt.","commonSituations":"Artifact service endpoint unreachable or flaky network during pipeline startup; stale/invalid retrieval token after a runner restart; corrupt artifact on the server causing persistent bad-SHA256 failures; dest directory permissions blocking writes on every retry.","solutions":["Inspect the joined sub-errors in the message to find the root cause (gRPC error vs SHA mismatch vs file error).","Verify the retrieval token is fresh and the artifact service endpoint is reachable from the worker.","Check disk permissions and free space at the destination directory.","Retry the job; the library already does 3 attempts with jittered backoff, so persistent failure means a systemic issue, not transient flake."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"conn, err := grpc.DialContext(ctx, endpoint, grpc.WithBlock(), grpc.WithTimeout(5*time.Second))\nif err != nil {\n    return fmt.Errorf(\"artifact service unreachable before retrieval: %w\", err)\n}\nconn.Close()","typeGuard":null,"tryCatchPattern":"err := artifact.Materialize(ctx, endpoint, rt, dest)\nif err != nil {\n    var backoff = time.Second\n    for i := 0; i < 2 && err != nil; i++ {\n        time.Sleep(backoff)\n        backoff *= 2\n        err = artifact.Materialize(ctx, endpoint, rt, dest)\n    }\n}","preventionTips":["Ensure workers can reach the artifact/runner endpoint before job submission","Use fresh retrieval tokens; never reuse tokens across runner sessions","Pass a context with a generous timeout for large artifact sets","Parse the joined sub-errors to triage network vs checksum causes"],"tags":["retry-exhausted","network","grpc","artifact"],"backgroundTag":"retry-exhausted","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}