{"record":{"id":"6faf4a8a69b3ccc7","repo":"apache/beam","slug":"failed-to-stage-artifacts-for-token-v-in-v-attempts-v","errorCode":null,"errorMessage":"failed to stage artifacts for token %v in %v attempts: %v","messagePattern":"failed to stage artifacts for token (.+?) in (.+?) attempts: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/runners/universal/runnerlib/stage.go","lineNumber":68,"sourceCode":"\n\treturn StageViaLegacyAPI(ctx, cc, binary, st)\n}\n\n// StageViaPortableAPI is a beam internal function for uploading artifacts to the staging service\n// via the portable API.\n//\n// It will be unexported at a later time.\nfunc StageViaPortableAPI(ctx context.Context, cc *grpc.ClientConn, binary, st string) (retErr error) {\n\tconst attempts = 3\n\tvar failures []string\n\tfor {\n\t\terr := stageFiles(ctx, cc, binary, st)\n\t\tif err == nil {\n\t\t\treturn nil // success!\n\t\t}\n\t\tfailures = append(failures, err.Error())\n\t\tif len(failures) > attempts {\n\t\t\treturn errors.Errorf(\"failed to stage artifacts for token %v in %v attempts: %v\", st, attempts, strings.Join(failures, \";\\n\"))\n\t\t}\n\t}\n}\n\nfunc stageFiles(ctx context.Context, cc *grpc.ClientConn, binary, st string) error {\n\tclient := jobpb.NewArtifactStagingServiceClient(cc)\n\tstream, err := client.ReverseArtifactRetrievalService(ctx)\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer func() {\n\t\tif err := stream.CloseSend(); err != nil {\n\t\t\tlog.Error(ctx, \"StageViaPortableApi CloseSend error: \", err)\n\t\t}\n\t}()\n\n\tif err := stream.Send(&jobpb.ArtifactResponseWrapper{StagingToken: st}); err != nil {\n\t\treturn errors.Wrapf(err, \"failed to send staging token\")","sourceCodeStart":50,"sourceCodeEnd":86,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/runners/universal/runnerlib/stage.go#L50-L86","documentation":"StageViaPortableAPI retries stageFiles up to a configured number of attempts; if every attempt fails it aggregates all per-attempt error strings and returns this error. It means artifact staging to the JobService's ArtifactStagingService repeatedly failed.","triggerScenarios":"stageFiles returns an error on every retry (attempt limit exceeded), typically because the staging gRPC stream to the artifact service fails — unreachable endpoint, missing binary file, or the artifact service rejecting the staging token.","commonSituations":"JobService endpoint misconfigured (artifact staging endpoint unreachable), worker binary path wrong or not built, network/firewall blocking the artifact port, or artifact service not supporting the reverse retrieval stream.","solutions":["Read the joined failure list in the error to see the underlying per-attempt cause","Verify the artifact staging endpoint (--artifactEndpoint / environment endpoint) is reachable","Ensure the Go worker binary was built and exists at the expected path","Increase attempts or fix the persistent network/permission problem before resubmitting"],"exampleFix":"// before\nreturn errors.Errorf(\"failed to stage artifacts for token %v in %v attempts: %v\", st, attempts, strings.Join(failures, \";\\n\"))\n// after: fix connectivity, e.g. ensure job server exposes the artifact port:\n// --artifactEndpoint=host:50051 and verify with `nc -zv host 50051`","handlingStrategy":"retry","validationCode":"// pre-flight: verify artifact endpoint reachable\nconn, err := grpc.Dial(artifactEndpoint, grpc.WithInsecure())\nif err != nil { return fmt.Errorf(\"artifact endpoint unreachable: %w\", err) }\nconn.Close()","typeGuard":null,"tryCatchPattern":"if err := stage(...); err != nil {\n  if strings.Contains(err.Error(), \"in \") && strings.Contains(err.Error(), \"attempts\") {\n    // all retries exhausted: check joined failure causes, fix network/binary, resubmit\n  }\n}","preventionTips":["Verify artifact staging endpoint connectivity before submission","Build the worker binary before submitting the pipeline","Ensure firewall rules allow the artifact service port"],"tags":["grpc","artifact-staging","network","beam-runner"],"backgroundTag":"api-request-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}