apache/hadoop · error · IOException

Cannot fetch records for {clazz}

Error message

Cannot fetch records for {clazz}

What it means

StateStoreFileBaseImpl backs the file-based State Store (local disk or HDFS): fetching all records for a record class lists the store directory and reads each child record file, often via per-record callables. Any exception during listing/reading — including the RuntimeException('Failed to retrieve record using file operations.') from a corrupted record — is counted as a metrics failure and rethrown as IOException('Cannot fetch records for <Class>') with the original cause attached.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/impl/StateStoreFileBaseImpl.java:270

        }
      } else {
        // Read records serially
        callables.forEach(e -> {
          try {
            e.call();
          } catch (Exception ex) {
            LOG.error("Failed to retrieve record using file operations.", ex);
            throw new RuntimeException(ex);
          }
        });
      }
    } catch (Exception e) {
      if (metrics != null) {
        metrics.addFailure(monotonicNow() - start);
      }
      String msg = "Cannot fetch records for " + clazz.getSimpleName();
      LOG.error(msg, e);
      throw new IOException(msg, e);
    }

    if (metrics != null) {
      metrics.addRead(monotonicNow() - start);
    }
    return new QueryResult<>(result, getTime());
  }

  /**
   * Get the state store record from the given path (path/child) and add the record to the
   * result list.
   *
   * @param clazz Class of the record.
   * @param result The list of results record. The records would be added to it unless the given
   * path represents old temp file.
   * @param path The parent path.
   * @param child The child path under the parent path. Both path and child completes the file
   * location for the given record.

View on GitHub (pinned to 2add963021)

Solutions

1. Read the logged cause — LOG.error(msg, e) prints the underlying exception identifying the file and I/O problem.
2. Fix filesystem access: permissions/ownership on the state store directory, mount the volume, or restore the backing HDFS.
3. If one corrupted record file is identified, quarantine/remove it; records such as membership regenerate from heartbeats, and mounts can be re-added from config.
4. Restart the router after the filesystem is healthy so the driver re-initializes.

// Pre-flight the file store directory before router start
Path storeDir = new Path(conf.get("dfs.federation.router.file-store.path"));
if (!fs.exists(storeDir) || !fs.getFileStatus(storeDir).getPermission()
    .getUserAction().implies(FsAction.READWRITE)) {
  throw new IllegalStateException("State store dir missing or not rw: " + storeDir);
}

try {
  QueryResult<T> r = recordStore.fetchAll();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Cannot fetch records")) {
    Throwable cause = e.getCause(); // real IO / corruption reason
    // if cause names one corrupted record file: quarantine it and retry,
    // else fix filesystem permissions/mount and retry
  } else { throw e; }
}

When it happens

Trigger: The state store directory is unreadable (permissions, missing mount) or its underlying HDFS NN is down for an HDFS-backed store; a single record file is corrupted or truncated (partial write); disk full or I/O errors during listing; file removed between list and read.

Common situations: Permissions changed on the state store directory after setup; HDFS-backed state store during an NN outage; unclean shutdown leaving a half-written record file; disk failures on a local-disk store.

Related errors

• Cannot serialize field {} into JSON
• Mount table state store is not available.
• Disabled Nameservice state store is not available.
• No namenodes to invoke {methodName} with params {params} fro
• Cached State Store not initialized, {recordClass} records no

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9ac4d9524656f69f. Report an issue: GitHub.