apache/hadoop · error · IOException

Request execution with deadline failed

Error message

Request execution with deadline failed

What it means

The catch-all branch of AbfsApacheHttpClient.executeRequest wraps every non-timeout failure of the underlying HttpClient execution (connection acquisition errors, SSL failures, socket resets, InterruptedException from executor shutdown) into IOException('Request execution with deadline failed') with the real cause chained. The message itself is only a wrapper — the actionable diagnosis is always in the cause chain.

Source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsApacheHttpClient.java:215

        .setSocketTimeout(readTimeout);
    httpRequest.setConfig(requestConfigBuilder.build());
    ExecutorService executor = Executors.newSingleThreadExecutor();
    Future<HttpResponse> future = executor.submit(() ->
        httpClient.execute(httpRequest, abfsHttpClientContext)
    );

    try {
      return future.get(deadlineMillis, TimeUnit.MILLISECONDS);
    } catch (TimeoutException e) {
      /* Deadline exceeded, abort the request.
       * This will also kill the underlying socket exception in the HttpClient.
       * Connection will be marked stale and won't be returned back to KAC for reuse.
       */
      httpRequest.abort();
      throw new TailLatencyRequestTimeoutException(e);
    } catch (Exception e) {
      // Any other exception from execution should be thrown as IOException.
      throw new IOException("Request execution with deadline failed", e);
    } finally {
      executor.shutdownNow();
    }
  }

  /**
   * Creates the socket factory registry for HTTP and HTTPS.
   *
   * @param sslSocketFactory SSL socket factory.
   * @return Socket factory registry.
   */
  private Registry<ConnectionSocketFactory> createSocketFactoryRegistry(
      ConnectionSocketFactory sslSocketFactory) {
    if (sslSocketFactory == null) {
      return RegistryBuilder.<ConnectionSocketFactory>create()
          .register(HTTP_SCHEME,
              PlainConnectionSocketFactory.getSocketFactory())
          .build();

View on GitHub (pinned to 2add963021)

Solutions

1. Unwrap the chain (e.getCause(), often twice) and fix the underlying connectivity error identified there
2. For DNS/proxy/TLS causes, correct the endpoint URL, proxy settings, or truststore rather than retrying
3. For pool or fd exhaustion, tune fs.azure.io.* concurrency settings or reduce parallelism
4. Retry idempotent reads — connection-level failures are frequently transient

try {
  abfsClient.executeRequest(...);
} catch (IOException e) {
  Throwable root = e;
  while (root.getCause() != null) root = root.getCause();
  // diagnose root: UnknownHost / SSL / Connect / pool exhaustion, then act or retry
}

When it happens

Trigger: httpClient.execute(...) throwing anything other than TimeoutException on the worker thread: UnknownHostException, SSLException, ConnectException, connection-pool exhaustion, or the executor being interrupted (shutdownNow). Distinct from 4802, which fires on deadline expiry only.

Common situations: Misconfigured proxy or missing SSL truststore entries; keep-alive connections reset by middleboxes; socket/handle exhaustion under very high concurrency; races between job close and in-flight requests.

Related errors

• Request Duration Exceeded Tail Latency Threshold.
• Got EOF but currentPos = ${currentPos} < filelength = ${file
• Got invalid response code {rc} from {url}: {responseMessage}
• WASB Driver using wasb(s) schema is no longer supported. Ins
• "%s" must be set for user-bound SAS auth type.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e9c0556f1d10e4b9. Report an issue: GitHub.