apache/beam · error

failed to send header for %v

Error message

failed to send header for %v

What it means

Stage sends the PutArtifact header (metadata message) as the first stream message; if stream.Send fails, the stream is closed and the error wrapped with this message. It means the very first message of the artifact upload was rejected by the gRPC layer, before any file content was transmitted.

Source

Thrown at sdks/go/pkg/beam/artifact/stage.go:158

	fd, err := os.Open(filename)
	if err != nil {
		return nil, err
	}
	defer fd.Close()

	stream, err := client.PutArtifact(ctx)
	if err != nil {
		return nil, err
	}

	header := &jobpb.PutArtifactRequest{
		Content: &jobpb.PutArtifactRequest_Metadata{
			Metadata: pmd,
		},
	}
	if err := stream.Send(header); err != nil {
		stream.CloseAndRecv() // ignore error
		return nil, errors.Wrapf(err, "failed to send header for %v", filename)
	}
	stagedHash, err := stageChunks(stream, fd)
	if err != nil {
		_, errClose := stream.CloseAndRecv()
		return nil, errors.Wrapf(err, "failed to send chunks for %v; close error: %v", filename, errClose)
	}
	if resp, err := stream.CloseAndRecv(); err != nil && err != io.EOF {
		return nil, errors.Wrapf(err, "failed to close stream for %v; response: %v", filename, resp)
	}
	if hash != stagedHash {
		return nil, errors.Errorf("unexpected SHA256 for sent chunks for %v: %v, want %v", filename, stagedHash, hash)
	}
	return md, nil
}

func stageChunks(stream jobpb.LegacyArtifactStagingService_PutArtifactClient, r io.Reader) (string, error) {
	sha256W := sha256.New()
	data := make([]byte, 1<<20)

View on GitHub (pinned to 12126d8942)

Solutions

1. Read the wrapped gRPC status (Unavailable, Canceled, Unknown) to pinpoint transport vs server rejection.
2. Verify the staging session token is valid and not expired before staging.
3. Check connectivity to the artifact staging endpoint (port open, TLS certs correct).
4. Retry — MultiStage retries up to 3 times with jittered backoff; use it rather than single Stage calls.

Example fix

// before
md, err := artifact.Stage(ctx, client, key, filename, st)
// after — use the retrying wrapper
artifacts, err := artifact.MultiStage(ctx, client, 4, []artifact.FileArtifact{{Key: key, Filename: filename}}, st)
if err != nil {
    return err
}

conn, err := grpc.DialContext(ctx, endpoint, grpc.WithBlock(), grpc.WithTimeout(5*time.Second))
if err != nil {
    return fmt.Errorf("cannot reach staging endpoint: %w", err)
}
if _, err := os.Stat(filename); err != nil {
    conn.Close()
    return fmt.Errorf("cannot stage missing file: %w", err)
}
conn.Close()

if _, err := artifact.Stage(ctx, client, key, filename, st); err != nil {
    if strings.Contains(err.Error(), "failed to send header") && st, ok := status.FromError(errors.Unwrap(err)); ok && st.Code() == codes.Unavailable {
        time.Sleep(3 * time.Second)
        _, err = artifact.Stage(ctx, client, key, filename, st)
    }
    return err
}

When it happens

Trigger: Calling artifact.Stage (directly or via MultiStage) where the initial stream.Send(header) returns an error: connection broken immediately after PutArtifact was opened, context canceled/deadline exceeded, or the server closed the stream (e.g. invalid staging token handled via status instead of Send acceptance).

Common situations: Runner endpoint went down between connection setup and header send; submitting context times out on slow links; invalid staging session token causing the server to abort the stream; firewall/proxy killing the gRPC connection.

Related errors

• Logging stream terminated unexpectedly before it was closed
• chunk send failed
• failed to receive header
• failed to retrieve chunk for %v
• failed to retrieve %v in %v attempts: %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d6d508660fbd99b7. Report an issue: GitHub.