apache/hadoop · error · IOException

host file read operation timed out

Error message

host file read operation timed out

What it means

CombinedHostsFileReader parses the JSON-based combined hosts file (dfs.hosts when the NameNode uses the CombinedHostsFileProvider) on a background FutureTask bounded by a read timeout, so a hung read cannot block NameNode refresh. When futureTask.get(readTimeout, MILLISECONDS) times out, the task is cancelled and IOException 'host file read operation timed out' propagates through CombinedHostFileManager.refresh(). The timeout comes from dfs.hosts.timeout (milliseconds; default 0 means no timeout wrapper).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/CombinedHostsFileReader.java:155

  public static DatanodeAdminProperties[]
      readFileWithTimeout(final String hostsFile, final int readTimeout) throws IOException {
    FutureTask<DatanodeAdminProperties[]> futureTask = new FutureTask<>(
        new Callable<DatanodeAdminProperties[]>() {
          @Override
          public DatanodeAdminProperties[] call() throws Exception {
            return readFile(hostsFile);
        }
      });

    Thread thread = new Thread(futureTask);
    thread.start();

    try {
      return futureTask.get(readTimeout, TimeUnit.MILLISECONDS);
    } catch (TimeoutException e) {
      futureTask.cancel(true);
      LOG.error("refresh File read operation timed out");
      throw new IOException("host file read operation timed out");
    } catch (InterruptedException | ExecutionException e) {
      LOG.error("File read operation interrupted : " + e.getMessage());
      throw new IOException("host file read operation timed out");
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Raise dfs.hosts.timeout (milliseconds) on the NameNode to comfortably exceed real read+parse time, or set it back to 0 to disable the timeout wrapper
2. Move dfs.hosts / dfs.hosts.exclude to local disk on the NameNode host instead of network storage
3. Fix the storage stall: check mount health (stat/df on the file), review NFS options (soft, timeo), remount if hung
4. Trim very large hosts files; the parse is single-threaded and CPU-heavy on huge JSON arrays

Example fix

<!-- before -->
<property><name>dfs.hosts.timeout</name><value>1000</value></property>

<!-- after -->
<property><name>dfs.hosts.timeout</name><value>30000</value></property>

File f = new File(conf.get("dfs.hosts", ""));
if (!f.exists() || !f.canRead()) {
  // fix before refreshing; a stalled mount also often fails canRead quickly
}

try {
  hostConfigManager.refresh(); // or CombinedHostsFileReader.readFileWithTimeout(file, timeoutMs)
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("host file read operation timed out")) {
    // schedule a delayed retry with backoff; alert if repeated (storage stall)
  } else { throw e; }
}

When it happens

Trigger: NameNode hosts refresh (startup, -refreshNodes, or scheduled refresh) with dfs.hosts.timeout > 0 while reading+parsing the JSON hosts file exceeds it: a stalled NFS mount, an enormous hosts file, or slow storage.

Common situations: dfs.hosts kept on NFS that hangs or is slow; hosts files with tens of thousands of entries; storage hiccups during scheduled refresh; dfs.hosts.timeout set too small.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Unknown nameservice: {}
• Configuration has multiple addresses that match local node's
• Configuration dfs.namenode.rpc-address must be suffixed with
• Unexpected configuration parameters: dfs.namenode.replicatio
• Unexpected configuration parameters: dfs.replication.max = {

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4673e9327df68c15. Report an issue: GitHub.