What causes "failed to stage %v in %v attempts: %v" in apache/beam?

Inspect the joined sub-errors in the message to identify the root cause (send error vs open error vs token rejection).

failed to stage %v in %v attempts: %v

Error message

failed to stage %v in %v attempts: %v

What it means

MultiStage sets this error (via errorx.GuardedError) when a file fails to stage after more than 3 attempts; the individual failure messages are joined with ';' into the aggregate message. It means the artifact could not be uploaded to the legacy artifact staging service despite retries with jittered backoff.

Source

Thrown at sdks/go/pkg/beam/artifact/stage.go:105

				if permErr.Error() != nil {
					continue
				}

				const attempts = 3

				var failures []string
				for {
					a, err := Stage(ctx, client, f.Key, f.Filename, st)
					if err == nil {
						ret <- a
						break
					}
					if permErr.Error() != nil {
						break // give up
					}
					failures = append(failures, err.Error())
					if len(failures) > attempts {
						permErr.TrySetError(errors.Errorf("failed to stage %v in %v attempts: %v", f.Filename, attempts, strings.Join(failures, "; ")))
						break // give up
					}
					time.Sleep(time.Duration(rand.Intn(5)+1) * time.Second)
				}
			}
		}()
	}
	wg.Wait()
	close(ret)

	return queue2slice(ret), permErr.Error()
}

// Stage stages a local file as an artifact with the given key. It computes
// the SHA256 and returns the full artifact metadata.
func Stage(ctx context.Context, client jobpb.LegacyArtifactStagingServiceClient, key, filename, st string) (*jobpb.ArtifactMetadata, error) {
	stat, err := os.Stat(filename)
	if err != nil {

View on GitHub (pinned to 12126d8942)

Solutions

Inspect the joined sub-errors in the message to identify the root cause (send error vs open error vs token rejection).
Verify the staging token is current and the staging service endpoint is reachable (curl/openssl the endpoint).
Ensure all staged files exist and are readable for the whole retry window.
Retry submission; persistent failure indicates a systemic endpoint/auth issue, not a transient one.

Defensive patterns

Strategy: retry

Validate before calling

for _, f := range files {
    if _, err := os.Stat(f.Filename); err != nil {
        return fmt.Errorf("file to stage missing: %w", err)
    }
}
conn, err := grpc.DialContext(ctx, endpoint, grpc.WithBlock(), grpc.WithTimeout(5*time.Second))
if err != nil {
    return fmt.Errorf("staging endpoint unreachable: %w", err)
}
conn.Close()

Try / catch

_, err := artifact.MultiStage(ctx, client, cpus, files, st)
if err != nil {
    var backoff = time.Second
    for i := 0; i < 2 && err != nil; i++ {
        time.Sleep(backoff)
        backoff *= 2
        _, err = artifact.MultiStage(ctx, client, cpus, files, st)
    }
}

Prevention

Keep staged files present and readable for the duration of staging
Use valid, unexpired staging session tokens
Verify network path and TLS config to the staging endpoint beforehand
Prefer MultiStage (built-in retries with backoff) over raw Stage calls

When it happens

Trigger: MultiStage -> Stage(ctx, client, key, filename, st) fails on attempt 4+: PutArtifact stream send errors, staging session token rejected, or the file vanished/became unreadable between attempts.

Common situations: Staging service endpoint unreachable from the submitting machine; invalid/expired staging session token; source file deleted or permission-revoked mid-retry; proxy blocking long gRPC uploads; TLS misconfiguration against the runner endpoint.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2e8cc9bfbef90458. Report an issue: GitHub.