apache/hadoop · error · IOException

Request execution with deadline failed

Error message

Request execution with deadline failed

What it means

The catch-all branch of AbfsApacheHttpClient.executeRequest wraps every non-timeout failure of the underlying HttpClient execution (connection acquisition errors, SSL failures, socket resets, InterruptedException from executor shutdown) into IOException('Request execution with deadline failed') with the real cause chained. The message itself is only a wrapper — the actionable diagnosis is always in the cause chain.

Solutions

1. Unwrap the chain (e.getCause(), often twice) and fix the underlying connectivity error identified there
2. For DNS/proxy/TLS causes, correct the endpoint URL, proxy settings, or truststore rather than retrying
3. For pool or fd exhaustion, tune fs.azure.io.* concurrency settings or reduce parallelism
4. Retry idempotent reads — connection-level failures are frequently transient

try {
  abfsClient.executeRequest(...);
} catch (IOException e) {
  Throwable root = e;
  while (root.getCause() != null) root = root.getCause();
  // diagnose root: UnknownHost / SSL / Connect / pool exhaustion, then act or retry
}

When it happens

Trigger: httpClient.execute(...) throwing anything other than TimeoutException on the worker thread: UnknownHostException, SSLException, ConnectException, connection-pool exhaustion, or the executor being interrupted (shutdownNow). Distinct from 4802, which fires on deadline expiry only.

Common situations: Misconfigured proxy or missing SSL truststore entries; keep-alive connections reset by middleboxes; socket/handle exhaustion under very high concurrency; races between job close and in-flight requests.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• Request Duration Exceeded Tail Latency Threshold.
• ABFS endpoint is not set correctly
• Both global key and encryption context are set, only one…
• Cannot create file because parent folder does not exist.
• Cannot remove user, group or other entry from access ACL.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e9c0556f1d10e4b9. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsApacheHttpClient.java:215

        .setSocketTimeout(readTimeout);
    httpRequest.setConfig(requestConfigBuilder.build());
    ExecutorService executor = Executors.newSingleThreadExecutor();
    Future<HttpResponse> future = executor.submit(() ->
        httpClient.execute(httpRequest, abfsHttpClientContext)
    );

    try {
      return future.get(deadlineMillis, TimeUnit.MILLISECONDS);
    } catch (TimeoutException e) {
      /* Deadline exceeded, abort the request.
       * This will also kill the underlying socket exception in the HttpClient.
       * Connection will be marked stale and won't be returned back to KAC for reuse.
       */
      httpRequest.abort();
      throw new TailLatencyRequestTimeoutException(e);
    } catch (Exception e) {
      // Any other exception from execution should be thrown as IOException.
      throw new IOException("Request execution with deadline failed", e);
    } finally {
      executor.shutdownNow();
    }
  }

  /**
   * Creates the socket factory registry for HTTP and HTTPS.
   *
   * @param sslSocketFactory SSL socket factory.
   * @return Socket factory registry.
   */
  private Registry<ConnectionSocketFactory> createSocketFactoryRegistry(
      ConnectionSocketFactory sslSocketFactory) {
    if (sslSocketFactory == null) {
      return RegistryBuilder.<ConnectionSocketFactory>create()
          .register(HTTP_SCHEME,
              PlainConnectionSocketFactory.getSocketFactory())
          .build();

View on GitHub (pinned to 2add963021)