apache/beam · critical

failed to connect to data service at %v

Error message

failed to connect to data service at %v

What it means

newDataChannel failed to establish a gRPC connection to the Beam data service at the URL carried in the execution port, within a 15 second dial timeout. This is a transport-level failure: the logging/data gRPC endpoint was unreachable or refused the connection. The library wraps the underlying dial error with the target URL for diagnosis.

Source

Thrown at sdks/go/pkg/beam/core/runtime/harness/datamgr.go:262

// If permitted, PTransformDone closes the channel.
func (ec *elementsChan) PTransformDone() {
	ec.mu.Lock()
	defer ec.mu.Unlock()
	ec.got++
	if ec.want > 0 && ec.want == ec.got {
		if !ec.Closed() {
			atomic.StoreUint32(&ec.closed, 1)
			close(ec.ch)
		}
	}
}

func newDataChannel(ctx context.Context, port exec.Port) (*DataChannel, error) {
	ctx, cancelFn := context.WithCancel(ctx)
	cc, err := dial(ctx, port.URL, "data", 15*time.Second)
	if err != nil {
		cancelFn()
		return nil, errors.Wrapf(err, "failed to connect to data service at %v", port.URL)
	}
	client, err := fnpb.NewBeamFnDataClient(cc).Data(ctx)
	if err != nil {
		cc.Close()
		cancelFn()
		return nil, errors.Wrapf(err, "failed to create data client on %v", port.URL)
	}
	return makeDataChannel(ctx, port.URL, client, func() {
		cc.Close()
		cancelFn()
	}), nil
}

func makeDataChannel(ctx context.Context, id string, client dataClient, cancelFn context.CancelFunc) *DataChannel {
	ret := &DataChannel{
		id:                id,
		client:            client,
		writers:           make(map[instructionID]map[string]*dataWriter),

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify the port.URL host/port is reachable from the worker (nc/host dig) and that the data service is up
2. Fix worker networking: run containers on a shared network, open firewall/NetworkPolicy rules
3. Increase or inspect the 15s dial window; if the service starts slowly, ensure the runner is ready before workers launch
4. Check proxy env vars (HTTP_PROXY/HTTPS_PROXY) are not hijacking gRPC traffic; use NO_PROXY for internal hosts

Example fix

// before
cc, err := dial(ctx, port.URL, "data", 15*time.Second)
if err != nil { return nil, errors.Wrapf(err, "failed to connect to data service at %v", port.URL) }
// after
if err := waitForEndpoint(port.URL, 30*time.Second); err != nil {
  return nil, errors.Wrapf(err, "data service %v not reachable", port.URL)
}
cc, err := dial(ctx, port.URL, "data", 30*time.Second)

conn, err := net.DialTimeout("tcp", hostFromURL(port.URL), 5*time.Second)
if err != nil {
  return fmt.Errorf("data service %s unreachable before harness start: %w", port.URL, err)
}
conn.Close()

err := startWorker(ctx)
if err != nil && isDialFailure(err) {
  // exponential backoff retry, then fail with clear connectivity message
  return retryWithBackoff(ctx, startWorker, 3)
}

When it happens

Trigger: The data service host:port in port.URL is unreachable; the FnHarness was started with a stale or wrong artifact/log service endpoint; networking between SDK worker container and the runner's data service is blocked; the data service is not yet listening when the worker dials (15s timeout exceeded).

Common situations: Running Beam Go on Flink/Spark/Dataflow with misconfigured worker networking or missing ingress rules; docker-network isolation where the worker cannot resolve the runner hostname; Kubernetes NetworkPolicy blocking worker-to-runner traffic; local DirectRunner port conflicts.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

• failed to read metadata from context
• error connecting to job server at %v: %v
• failed to retrieve chunk for %v
• failed to get manifest
• failed to send chunks for %v; close error: %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/573daf93d1f6a9dd. Report an issue: GitHub.