apache/beam · error

failed to find worker id in metadata %v

Error message

failed to find worker id in metadata %v

What it means

grpcx.ReadWorkerID finds the worker ID under the metadata key idKey. This error is returned when the incoming context HAS metadata but no entry under that key (or an empty slice), and includes the full metadata map for debugging. It distinguishes 'no metadata at all' (5378) from 'metadata present but worker id key missing'.

Source

Thrown at sdks/go/pkg/beam/util/grpcx/metadata.go:36

import (
	"context"

	"github.com/apache/beam/sdks/v2/go/pkg/beam/internal/errors"
	"google.golang.org/grpc/metadata"
)

const idKey = "worker_id"

// ReadWorkerID reads the worker ID from an incoming gRPC request context.
func ReadWorkerID(ctx context.Context) (string, error) {
	md, ok := metadata.FromIncomingContext(ctx)
	if !ok {
		return "", errors.New("failed to read metadata from context")
	}
	id, ok := md[idKey]
	if !ok || len(id) < 1 {
		return "", errors.Errorf("failed to find worker id in metadata %v", md)
	}
	if len(id) > 1 {
		return "", errors.Errorf("multiple worker ids in metadata: %v", id)
	}
	return id[0], nil
}

// WriteWorkerID write the worker ID to an outgoing gRPC request context. It
// merges the information with any existing gRPC metadata.
func WriteWorkerID(ctx context.Context, id string) context.Context {
	md := metadata.New(map[string]string{
		idKey: id,
	})
	if old, ok := metadata.FromOutgoingContext(ctx); ok {
		md = metadata.Join(md, old)
	}
	return metadata.NewOutgoingContext(ctx, md)
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Set the worker id under the expected key (use grpcx.AddWorkerID so the key always matches)
2. Compare the keys printed in the error message with what the client sends
3. Ensure only one non-empty worker id value is present (avoids the related 'multiple worker ids' error)

Example fix

// before
ctx = metadata.AppendToOutgoingContext(ctx, "id", workerID) // wrong key
// after
ctx = metadata.AppendToOutgoingContext(ctx, "beam:worker:id:1", workerID)

md, _ := metadata.FromIncomingContext(ctx)
if len(md[idKey]) == 0 {
	return errors.New("worker id key missing in metadata")
}

func hasWorkerID(ctx context.Context) bool {
	md, ok := metadata.FromIncomingContext(ctx)
	return ok && len(md[idKey]) > 0
}

id, err := grpcx.ReadWorkerID(ctx)
if err != nil {
	if strings.Contains(err.Error(), "failed to find worker id") {
		return status.Error(codes.InvalidArgument, "worker id key missing; use grpcx.AddWorkerID")
	}
	return err
}

When it happens

Trigger: A gRPC request reaches ReadWorkerID with metadata that lacks the worker-id key — the client set other metadata or used the wrong key name, or the id slice is empty.

Common situations: Custom Beam worker clients using a metadata key name that doesn't match the server's idKey; partial metadata propagation through proxies/interceptors; version mismatch where the key naming convention changed.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Header metadata already has a worker_id.
• failed to read metadata from context
• Error starting background log thread
• input is already set.
• All workers communicate through gRPC should have worker_id.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2aa363a2568c1632. Report an issue: GitHub.