{"record":{"id":"2e8cc9bfbef90458","repo":"apache/beam","slug":"failed-to-stage-v-in-v-attempts-v","errorCode":null,"errorMessage":"failed to stage %v in %v attempts: %v","messagePattern":"failed to stage (.+?) in (.+?) attempts: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/artifact/stage.go","lineNumber":105,"sourceCode":"\t\t\t\tif permErr.Error() != nil {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\n\t\t\t\tconst attempts = 3\n\n\t\t\t\tvar failures []string\n\t\t\t\tfor {\n\t\t\t\t\ta, err := Stage(ctx, client, f.Key, f.Filename, st)\n\t\t\t\t\tif err == nil {\n\t\t\t\t\t\tret <- a\n\t\t\t\t\t\tbreak\n\t\t\t\t\t}\n\t\t\t\t\tif permErr.Error() != nil {\n\t\t\t\t\t\tbreak // give up\n\t\t\t\t\t}\n\t\t\t\t\tfailures = append(failures, err.Error())\n\t\t\t\t\tif len(failures) > attempts {\n\t\t\t\t\t\tpermErr.TrySetError(errors.Errorf(\"failed to stage %v in %v attempts: %v\", f.Filename, attempts, strings.Join(failures, \"; \")))\n\t\t\t\t\t\tbreak // give up\n\t\t\t\t\t}\n\t\t\t\t\ttime.Sleep(time.Duration(rand.Intn(5)+1) * time.Second)\n\t\t\t\t}\n\t\t\t}\n\t\t}()\n\t}\n\twg.Wait()\n\tclose(ret)\n\n\treturn queue2slice(ret), permErr.Error()\n}\n\n// Stage stages a local file as an artifact with the given key. It computes\n// the SHA256 and returns the full artifact metadata.\nfunc Stage(ctx context.Context, client jobpb.LegacyArtifactStagingServiceClient, key, filename, st string) (*jobpb.ArtifactMetadata, error) {\n\tstat, err := os.Stat(filename)\n\tif err != nil {","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/artifact/stage.go#L87-L123","documentation":"MultiStage sets this error (via errorx.GuardedError) when a file fails to stage after more than 3 attempts; the individual failure messages are joined with ';' into the aggregate message. It means the artifact could not be uploaded to the legacy artifact staging service despite retries with jittered backoff.","triggerScenarios":"MultiStage -> Stage(ctx, client, key, filename, st) fails on attempt 4+: PutArtifact stream send errors, staging session token rejected, or the file vanished/became unreadable between attempts.","commonSituations":"Staging service endpoint unreachable from the submitting machine; invalid/expired staging session token; source file deleted or permission-revoked mid-retry; proxy blocking long gRPC uploads; TLS misconfiguration against the runner endpoint.","solutions":["Inspect the joined sub-errors in the message to identify the root cause (send error vs open error vs token rejection).","Verify the staging token is current and the staging service endpoint is reachable (curl/openssl the endpoint).","Ensure all staged files exist and are readable for the whole retry window.","Retry submission; persistent failure indicates a systemic endpoint/auth issue, not a transient one."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"for _, f := range files {\n    if _, err := os.Stat(f.Filename); err != nil {\n        return fmt.Errorf(\"file to stage missing: %w\", err)\n    }\n}\nconn, err := grpc.DialContext(ctx, endpoint, grpc.WithBlock(), grpc.WithTimeout(5*time.Second))\nif err != nil {\n    return fmt.Errorf(\"staging endpoint unreachable: %w\", err)\n}\nconn.Close()","typeGuard":null,"tryCatchPattern":"_, err := artifact.MultiStage(ctx, client, cpus, files, st)\nif err != nil {\n    var backoff = time.Second\n    for i := 0; i < 2 && err != nil; i++ {\n        time.Sleep(backoff)\n        backoff *= 2\n        _, err = artifact.MultiStage(ctx, client, cpus, files, st)\n    }\n}","preventionTips":["Keep staged files present and readable for the duration of staging","Use valid, unexpired staging session tokens","Verify network path and TLS config to the staging endpoint beforehand","Prefer MultiStage (built-in retries with backoff) over raw Stage calls"],"tags":["retry-exhausted","grpc","upload","artifact"],"backgroundTag":"retry-exhausted","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}