apache/beam · error

failed to stage artifacts for token

Error message

failed to stage artifacts for token %v in %v attempts: %v

What it means

StageViaPortableAPI retries stageFiles up to a configured number of attempts; if every attempt fails it aggregates all per-attempt error strings and returns this error. It means artifact staging to the JobService's ArtifactStagingService repeatedly failed.

Solutions

1. Read the joined failure list in the error to see the underlying per-attempt cause
2. Verify the artifact staging endpoint (--artifactEndpoint / environment endpoint) is reachable
3. Ensure the Go worker binary was built and exists at the expected path
4. Increase attempts or fix the persistent network/permission problem before resubmitting

Example fix

// before
return errors.Errorf("failed to stage artifacts for token %v in %v attempts: %v", st, attempts, strings.Join(failures, ";\n"))
// after: fix connectivity, e.g. ensure job server exposes the artifact port:
// --artifactEndpoint=host:50051 and verify with `nc -zv host 50051`

// pre-flight: verify artifact endpoint reachable
conn, err := grpc.Dial(artifactEndpoint, grpc.WithInsecure())
if err != nil { return fmt.Errorf("artifact endpoint unreachable: %w", err) }
conn.Close()

if err := stage(...); err != nil {
  if strings.Contains(err.Error(), "in ") && strings.Contains(err.Error(), "attempts") {
    // all retries exhausted: check joined failure causes, fix network/binary, resubmit
  }
}

When it happens

Trigger: stageFiles returns an error on every retry (attempt limit exceeded), typically because the staging gRPC stream to the artifact service fails — unreachable endpoint, missing binary file, or the artifact service rejecting the staging token.

Common situations: JobService endpoint misconfigured (artifact staging endpoint unreachable), worker binary path wrong or not built, network/firewall blocking the artifact port, or artifact service not supporting the reverse retrieval stream.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• chunk send failed
• failed to receive header
• failed to send chunks for
• failed to send staging token
• chunk send failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6faf4a8a69b3ccc7. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/universal/runnerlib/stage.go:68

	return StageViaLegacyAPI(ctx, cc, binary, st)
}

// StageViaPortableAPI is a beam internal function for uploading artifacts to the staging service
// via the portable API.
//
// It will be unexported at a later time.
func StageViaPortableAPI(ctx context.Context, cc *grpc.ClientConn, binary, st string) (retErr error) {
	const attempts = 3
	var failures []string
	for {
		err := stageFiles(ctx, cc, binary, st)
		if err == nil {
			return nil // success!
		}
		failures = append(failures, err.Error())
		if len(failures) > attempts {
			return errors.Errorf("failed to stage artifacts for token %v in %v attempts: %v", st, attempts, strings.Join(failures, ";\n"))
		}
	}
}

func stageFiles(ctx context.Context, cc *grpc.ClientConn, binary, st string) error {
	client := jobpb.NewArtifactStagingServiceClient(cc)
	stream, err := client.ReverseArtifactRetrievalService(ctx)
	if err != nil {
		return err
	}
	defer func() {
		if err := stream.CloseSend(); err != nil {
			log.Error(ctx, "StageViaPortableApi CloseSend error: ", err)
		}
	}()

	if err := stream.Send(&jobpb.ArtifactResponseWrapper{StagingToken: st}); err != nil {
		return errors.Wrapf(err, "failed to send staging token")

View on GitHub (pinned to 12126d8942)