apache/beam · critical

failed to connect to control service

Error message

failed to connect to control service

What it means

After dialing the control server, the harness opens the bidirectional gRPC Control stream via client.Control(ctx). If the stream RPC itself fails (server unavailable, RPC unimplemented, deadline exceeded, stream refused), the error is wrapped as "failed to connect to control service" and MainWithOptions returns. Unlike a dial failure, the TCP/TLS connection existed but the Control RPC could not be established.

Source

Thrown at sdks/go/pkg/beam/core/runtime/harness/harness.go:122

		log.Debugf(ctx, "Failed to parse element_processing_timeout: %v, there will be no timeout for processing an element in a PTransform operation", err)
	}

	// Connect to FnAPI control server. Receive and execute work.
	conn, err := dial(ctx, controlEndpoint, "control", 60*time.Second)
	if err != nil {
		return errors.Wrap(err, "failed to connect")
	}
	defer conn.Close()

	client := fnpb.NewBeamFnControlClient(conn)

	lookupDesc := func(id bundleDescriptorID) (*fnpb.ProcessBundleDescriptor, error) {
		return client.GetProcessBundleDescriptor(ctx, &fnpb.GetProcessBundleDescriptorRequest{ProcessBundleDescriptorId: string(id)})
	}

	stub, err := client.Control(ctx)
	if err != nil {
		return errors.Wrapf(err, "failed to connect to control service")
	}

	log.Debugf(ctx, "Successfully connected to control @ %v", controlEndpoint)

	// Each ProcessBundle is a sub-graph of the original one.

	var wg sync.WaitGroup
	respc := make(chan *fnpb.InstructionResponse, 100)

	wg.Add(1)

	// gRPC requires all writers to a stream be the same goroutine, so this is the
	// goroutine for managing responses back to the control service.
	go func() {
		defer wg.Done()
		for resp := range respc {
			// TODO(lostluck): 2023/03/29 fix debug level logging to be flagged.
			// log.Debugf(ctx, "RESP: %v", proto.MarshalTextString(resp))

View on GitHub (pinned to 12126d8942)

Solutions

1. Check the wrapped gRPC status code: Unimplemented -> version mismatch, DeadlineExceeded -> raise timeout/resources, Unavailable -> retry or fix startup ordering.
2. Align Beam runner and Go SDK harness versions so the BeamFnControl service contract matches.
3. Confirm the control server process is healthy and not crash-looping at the moment of stream setup.
4. Retry the worker; transient Unavailable during startup is common in container environments.
5. Inspect runner logs alongside harness logs for the server-side reason the stream was refused.

Example fix

// before: no retry around stream setup
stub, err := client.Control(ctx)
if err != nil {
    return errors.Wrapf(err, "failed to connect to control service")
}
// after: tolerate transient unavailability at startup
var stub fnpb.BeamFnControl_ControlClient
backoff := 1 * time.Second
for attempt := 0; attempt < 5; attempt++ {
    stub, err = client.Control(ctx)
    if err == nil {
        break
    }
    if status.Code(err) == codes.Unimplemented {
        return errors.Wrapf(err, "failed to connect to control service")
    }
    time.Sleep(backoff)
    backoff *= 2
}
if err != nil {
    return errors.Wrapf(err, "failed to connect to control service")
}

if controlEndpoint == "" {
    return errors.New("control endpoint must be set before opening the Control stream")
}
// dial with grpc.WithBlock() + grpc.WaitForReady(true) so the channel is
// confirmed ready before attempting to open the Control stream

stub, err := client.Control(ctx)
if err != nil {
    switch status.Code(err) {
    case codes.Unimplemented:
        // version mismatch between runner and SDK: fix versions, do not retry
    case codes.Unavailable, codes.DeadlineExceeded:
        // transient: retry with backoff
    default:
        return errors.Wrapf(err, "failed to connect to control service")
    }
}

When it happens

Trigger: client.Control(ctx) returns an error: the control server does not expose/accept the Control stream, the context deadline expires, the server rejects the RPC (unimplemented/permission denied), or the connection drops during stream setup.

Common situations: Runner and SDK version mismatch causing an incompatible BeamFnControl service; control server crashed between dial and Control; per-RPC deadline exceeded under load; mTLS/auth misconfiguration on the stream.

Related errors

• failed to connect
• control.Recv failed
• Logging stream terminated unexpectedly before it was closed
• Failed to call Rate Limit Service
• Failed to get response from Rate Limit Service

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9a5b73265af3b31c. Report an issue: GitHub.