apache/beam · error · TimeoutException

Timeout waiting for the service {endpoint.getUrl()} to start

Error message

Timeout waiting for the service {endpoint.getUrl()} to startup after {System.currentTimeMillis() - start} milliseconds.

What it means

waitForAllServicesToBeReady retries a socket connection to each expansion service endpoint with an increasing backoff (duration *= 1.2) until SERVICE_CHECK_TIMEOUT_MILLIS elapses. If a socket connection never succeeds within that window, it throws this TimeoutException naming the endpoint and elapsed time. The service either is not listening or is unreachable.

Source

Thrown at sdks/java/transform-service/src/main/java/org/apache/beam/sdk/transformservice/ExpansionService.java:97

          if (portIndex <= 0) {
            throw new RuntimeException(
                "Expected the endpoint to be of the form <host>:<port> but received " + url);
          }
          int port = Integer.parseInt(url.substring(portIndex + 1));
          String host = url.substring(0, portIndex);
          new Socket(host, port).close();
          // Current service is up. Checking the next one.
          continue outer;
        } catch (IOException exn) {
          try {
            Thread.sleep(duration);
          } catch (InterruptedException e) {
            // Ignore
          }
          duration = (long) (duration * 1.2);
        }
      }
      throw new TimeoutException(
          "Timeout waiting for the service "
              + endpoint.getUrl()
              + " to startup after "
              + (System.currentTimeMillis() - start)
              + " milliseconds.");
    }
  }

  @VisibleForTesting
  void disableServiceCheck() {
    disableServiceCheck = true;
  }

  @Override
  public void expand(
      ExpansionApi.ExpansionRequest request,
      StreamObserver<ExpansionApi.ExpansionResponse> responseObserver) {
    if (!checkedAllServices) {

View on GitHub (pinned to 12126d8942)

Solutions

Confirm the service is running and listening: docker compose ps / service ps, then test the port manually (nc -zv host port).
Verify the endpoint host/port pipeline options match the actually started service.
Fix the underlying service startup failure (check its logs) and restart with the launcher's up command.
Check firewall/network policy allows TCP access to the port from your client.

Example fix

// before: endpoint points at a service that never started
--transformServiceEndpoint=localhost:50051  // nothing listening

// after: start the service first, then connect
service.start(); service.waitTillUp(300_000);

Defensive patterns

Strategy: retry

Validate before calling

// verify reachability before expanding
try (Socket s = new Socket()) {
  s.connect(new InetSocketAddress(host, port), 2000); // throws if refused
}

Try / catch

try {
  expansionService.expand(transform, payload);
} catch (java.util.concurrent.TimeoutException e) {
  LOG.error("Expansion service {} unreachable: {}", endpointUrl, e.getMessage());
  // restart the service, then retry once
  launcherService.shutdown();
  launcherService.start();
  launcherService.waitTillUp(300_000);
}

Prevention

Confirm service uptime (ps command) before submitting pipelines.
Use reachable hostnames (localhost from host, not container-internal names).
Check container logs for silent crashes; keep the check window generous.

When it happens

Trigger: Calling expand() or discoverSchemaTransform() while the target expansion service endpoint refuses connections for the entire check window: the service process died, wrong host/port configured, firewall blocking, or the service is still starting after the check timeout.

Common situations: Transform service container exited right after launch (port conflict, image error); connecting to the wrong host (e.g. container-internal hostname from the host machine); network policy blocking the port; service slow to boot on a loaded machine.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8b29050467141916. Report an issue: GitHub.