apache/beam · critical

failed to create data client on %v

Error message

failed to create data client on %v

What it means

The gRPC connection to the data service was established, but creating the BeamFnData streaming client (NewBeamFnDataClient(cc).Data(ctx)) failed. This means the server did not accept or open the bidirectional data stream. Unlike 5221 the dial succeeded, so this points to a protocol or server-side rejection of the Data RPC.

Source

Thrown at sdks/go/pkg/beam/core/runtime/harness/datamgr.go:268

		if !ec.Closed() {
			atomic.StoreUint32(&ec.closed, 1)
			close(ec.ch)
		}
	}
}

func newDataChannel(ctx context.Context, port exec.Port) (*DataChannel, error) {
	ctx, cancelFn := context.WithCancel(ctx)
	cc, err := dial(ctx, port.URL, "data", 15*time.Second)
	if err != nil {
		cancelFn()
		return nil, errors.Wrapf(err, "failed to connect to data service at %v", port.URL)
	}
	client, err := fnpb.NewBeamFnDataClient(cc).Data(ctx)
	if err != nil {
		cc.Close()
		cancelFn()
		return nil, errors.Wrapf(err, "failed to create data client on %v", port.URL)
	}
	return makeDataChannel(ctx, port.URL, client, func() {
		cc.Close()
		cancelFn()
	}), nil
}

func makeDataChannel(ctx context.Context, id string, client dataClient, cancelFn context.CancelFunc) *DataChannel {
	ret := &DataChannel{
		id:                id,
		client:            client,
		writers:           make(map[instructionID]map[string]*dataWriter),
		timerWriters:      make(map[instructionID]map[timerKey]*timerWriter),
		channels:          make(map[instructionID]*elementsChan),
		endedInstructions: make(map[instructionID]struct{}),
		cancelFn:          cancelFn,
	}
	go ret.read(ctx)

View on GitHub (pinned to 12126d8942)

Solutions

1. Confirm port.URL points to the actual BeamFnData service endpoint and not another port
2. Align runner and Go SDK versions (same Beam release line) to avoid gRPC schema skew
3. Retry the bundle; a server under load or draining can transiently refuse stream creation
4. Inspect server logs for the Data RPC rejection to distinguish auth, capacity, or protocol issues

Example fix

// before
client, err := fnpb.NewBeamFnDataClient(cc).Data(ctx)
if err != nil { cc.Close(); cancelFn(); return nil, errors.Wrapf(err, "failed to create data client on %v", port.URL) }
// after
client, err := fnpb.NewBeamFnDataClient(cc).Data(ctx)
if err != nil {
  cc.Close(); cancelFn()
  return nil, retryable(errors.Wrapf(err, "failed to create data client on %v", port.URL))
}

resp, err := grpc_health_v1.NewHealthClient(cc).Check(ctx, &grpc_health_v1.HealthCheckRequest{})
if err != nil || resp.Status != grpc_health_v1.HealthCheckResponse_SERVING {
  return fmt.Errorf("data service not serving streams: %v", err)
}

client, err := fnpb.NewBeamFnDataClient(cc).Data(ctx)
if err != nil {
  if status, ok := status.FromError(err); ok && status.Code() == codes.Unavailable {
    return retryWithBackoff(ctx, redialAndOpen, 3)
  }
  return err
}

When it happens

Trigger: The remote endpoint at port.URL does not actually implement the BeamFnData gRPC service; gRPC server is shutting down or over capacity when the stream is requested; context canceled while establishing the stream; gRPC version/schema mismatch between SDK and runner.

Common situations: Pointing the harness at the wrong service port (e.g. logging port instead of data port); runner draining/restarting mid-bundle; version skew between Beam runner and Go SDK where the Data service definition changed; envoy/proxy intercepting gRPC and rejecting stream creation.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• failed to read metadata from context
• decodeStream value decode failed on close
• decodeMultiChunkStream chunk size decoding failed
• broken stream
• failed to connect to data service at %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b96e222027cf5b14. Report an issue: GitHub.