apache/beam · error

failed to read metadata from context

Error message

failed to read metadata from context

What it means

grpcx.ReadWorkerID extracts the 'worker_id' key from a gRPC incoming metadata context; the FnAPI/JobAPI servers use it to identify which worker sent a request. This error means the incoming gRPC context carries no metadata at all, so the worker ID cannot be read. Callers typically abort or ignore the request since attribution is impossible.

Source

Thrown at sdks/go/pkg/beam/util/grpcx/metadata.go:32

// limitations under the License.

// Package grpcx contains utilities for working with gRPC.
package grpcx

import (
	"context"

	"github.com/apache/beam/sdks/v2/go/pkg/beam/internal/errors"
	"google.golang.org/grpc/metadata"
)

const idKey = "worker_id"

// ReadWorkerID reads the worker ID from an incoming gRPC request context.
func ReadWorkerID(ctx context.Context) (string, error) {
	md, ok := metadata.FromIncomingContext(ctx)
	if !ok {
		return "", errors.New("failed to read metadata from context")
	}
	id, ok := md[idKey]
	if !ok || len(id) < 1 {
		return "", errors.Errorf("failed to find worker id in metadata %v", md)
	}
	if len(id) > 1 {
		return "", errors.Errorf("multiple worker ids in metadata: %v", id)
	}
	return id[0], nil
}

// WriteWorkerID write the worker ID to an outgoing gRPC request context. It
// merges the information with any existing gRPC metadata.
func WriteWorkerID(ctx context.Context, id string) context.Context {
	md := metadata.New(map[string]string{
		idKey: id,
	})
	if old, ok := metadata.FromOutgoingContext(ctx); ok {

View on GitHub (pinned to 12126d8942)

Solutions

1. Ensure every worker-side client writes the ID before dialing: wrap the context with grpcx.WriteWorkerID(ctx, workerID) and pass it to grpc.DialContext.
2. In tests, construct the incoming context with metadata.NewIncomingContext(ctx, metadata.Pairs("worker_id", "test-worker")).
3. Treat the returned error as 'unknown caller' and reject or log-and-continue depending on the endpoint (e.g. skip attribution for probes).
4. Check for middleware that strips incoming metadata, and for version skew where an old worker binary predates worker_id stamping.

Example fix

// before
conn, _ := grpc.DialContext(ctx, addr, ...) // no worker id metadata

// after
ctx = grpcx.WriteWorkerID(ctx, "worker-1")
conn, _ := grpc.DialContext(ctx, addr, grpc.WithContextDialer(dialerWith(ctx)), ...)

if _, ok := metadata.FromIncomingContext(ctx); !ok {
    // no metadata: handle unknown-caller path before calling ReadWorkerID
}

func hasIncomingMetadata(ctx context.Context) bool {
    _, ok := metadata.FromIncomingContext(ctx)
    return ok
}

id, err := grpcx.ReadWorkerID(ctx)
if err != nil {
    log.Printf("request without worker_id metadata: %v", err)
    return status.Error(codes.InvalidArgument, "missing worker id metadata")
}

When it happens

Trigger: Calling grpcx.ReadWorkerID(ctx) on a context where metadata.FromIncomingContext(ctx) returns ok==false — i.e. the request arrived without gRPC metadata (e.g. a client built without grpcx.WriteWorkerID, a raw test connection, a health check, or a non-Beam client hitting the worker endpoint).

Common situations: Unit tests calling the Beam worker/harness service handlers with plain context.Background(); load balancers or probes dialing the gRPC port; custom clients or older SDK workers not stamping worker_id metadata; multiplexed servers where non-Beam RPCs land on the same listener.

Related errors

• Header metadata already has a worker_id.
• failed to connect to data service at %v
• failed to create data client on %v
• failed to find worker id in metadata %v
• empty port

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/eccbf06e8a08ab0c. Report an issue: GitHub.