apache/beam · error

multiple worker ids in metadata

Error message

multiple worker ids in metadata: %v

What it means

ReadWorkerID extracts the worker ID from incoming gRPC metadata under the idKey. The Apache Beam Go SDK writes exactly one worker ID per request, so finding more than one value means the metadata is malformed and cannot be safely interpreted.

Solutions

1. Inspect the outgoing client code and ensure WriteWorkerID is applied only once per request context.
2. Check any gRPC proxy/interceptor for duplicate metadata key injection and deduplicate the worker-id key.
3. Fix test/setup code that constructs metadata manually to include the id key exactly once (use a []string of length 1).

Example fix

// before
md := metadata.Pairs("beam:worker:id", "a", "beam:worker:id", "b")
// after
md := metadata.Pairs("beam:worker:id", "a")

md, _ := metadata.FromIncomingContext(ctx)
ids := md["beam:worker:id"] // idKey
if len(ids) != 1 { /* fix metadata before calling ReadWorkerID */ }

func hasSingleWorkerID(md metadata.MD, key string) bool { return len(md.Get(key)) == 1 }

When it happens

Trigger: Calling ReadWorkerID (directly or via workerFromMetadataCtx) on a context whose incoming gRPC metadata contains 2+ values for the worker-id key, e.g. md[idKey] has len > 1.

Common situations: A proxy or intermediary merging metadata and duplicating the worker-id key; a client calling WriteWorkerID twice on the same context incorrectly; hand-crafted metadata in tests or custom transports appending the key multiple times.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• All workers communicate through gRPC should have worker_id…
• failed to find worker id in metadata
• failed to read metadata from context
• Header metadata already has a worker_id.
• Already bound to %r

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4218e61509a04e62. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/util/grpcx/metadata.go:39

	"github.com/apache/beam/sdks/v2/go/pkg/beam/internal/errors"
	"google.golang.org/grpc/metadata"
)

const idKey = "worker_id"

// ReadWorkerID reads the worker ID from an incoming gRPC request context.
func ReadWorkerID(ctx context.Context) (string, error) {
	md, ok := metadata.FromIncomingContext(ctx)
	if !ok {
		return "", errors.New("failed to read metadata from context")
	}
	id, ok := md[idKey]
	if !ok || len(id) < 1 {
		return "", errors.Errorf("failed to find worker id in metadata %v", md)
	}
	if len(id) > 1 {
		return "", errors.Errorf("multiple worker ids in metadata: %v", id)
	}
	return id[0], nil
}

// WriteWorkerID write the worker ID to an outgoing gRPC request context. It
// merges the information with any existing gRPC metadata.
func WriteWorkerID(ctx context.Context, id string) context.Context {
	md := metadata.New(map[string]string{
		idKey: id,
	})
	if old, ok := metadata.FromOutgoingContext(ctx); ok {
		md = metadata.Join(md, old)
	}
	return metadata.NewOutgoingContext(ctx, md)
}

View on GitHub (pinned to 12126d8942)