apache/beam · error · TimeoutException

Timeout waiting for the service

Error message

Timeout waiting for the service {endpoint.getUrl()} to startup after {System.currentTimeMillis() - start} milliseconds.

What it means

waitForAllServicesToBeReady retries a socket connection to each expansion service endpoint with an increasing backoff (duration *= 1.2) until SERVICE_CHECK_TIMEOUT_MILLIS elapses. If a socket connection never succeeds within that window, it throws this TimeoutException naming the endpoint and elapsed time. The service either is not listening or is unreachable.

Solutions

Confirm the service is running and listening: docker compose ps / service ps, then test the port manually (nc -zv host port).
Verify the endpoint host/port pipeline options match the actually started service.
Fix the underlying service startup failure (check its logs) and restart with the launcher's up command.
Check firewall/network policy allows TCP access to the port from your client.

Example fix

// before: endpoint points at a service that never started
--transformServiceEndpoint=localhost:50051  // nothing listening

// after: start the service first, then connect
service.start(); service.waitTillUp(300_000);

Defensive patterns

Strategy: retry

Validate before calling

// verify reachability before expanding
try (Socket s = new Socket()) {
  s.connect(new InetSocketAddress(host, port), 2000); // throws if refused
}

Try / catch

try {
  expansionService.expand(transform, payload);
} catch (java.util.concurrent.TimeoutException e) {
  LOG.error("Expansion service {} unreachable: {}", endpointUrl, e.getMessage());
  // restart the service, then retry once
  launcherService.shutdown();
  launcherService.start();
  launcherService.waitTillUp(300_000);
}

Prevention

Confirm service uptime (ps command) before submitting pipelines.
Use reachable hostnames (localhost from host, not container-internal names).
Check container logs for silent crashes; keep the check window generous.

When it happens

Trigger: Calling expand() or discoverSchemaTransform() while the target expansion service endpoint refuses connections for the entire check window: the service process died, wrong host/port configured, firewall blocking, or the service is still starting after the check timeout.

Common situations: Transform service container exited right after launch (port conflict, image error); connecting to the wrong host (e.g. container-internal hostname from the host machine); network policy blocking the port; service slow to boot on a loaded machine.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8b29050467141916. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/transform-service/src/main/java/org/apache/beam/sdk/transformservice/ExpansionService.java:97

          if (portIndex <= 0) {
            throw new RuntimeException(
                "Expected the endpoint to be of the form <host>:<port> but received " + url);
          }
          int port = Integer.parseInt(url.substring(portIndex + 1));
          String host = url.substring(0, portIndex);
          new Socket(host, port).close();
          // Current service is up. Checking the next one.
          continue outer;
        } catch (IOException exn) {
          try {
            Thread.sleep(duration);
          } catch (InterruptedException e) {
            // Ignore
          }
          duration = (long) (duration * 1.2);
        }
      }
      throw new TimeoutException(
          "Timeout waiting for the service "
              + endpoint.getUrl()
              + " to startup after "
              + (System.currentTimeMillis() - start)
              + " milliseconds.");
    }
  }

  @VisibleForTesting
  void disableServiceCheck() {
    disableServiceCheck = true;
  }

  @Override
  public void expand(
      ExpansionApi.ExpansionRequest request,
      StreamObserver<ExpansionApi.ExpansionResponse> responseObserver) {
    if (!checkedAllServices) {

View on GitHub (pinned to 12126d8942)