{"record":{"id":"9ac4d9524656f69f","repo":"apache/hadoop","slug":"cannot-fetch-records-for-clazz","errorCode":null,"errorMessage":"Cannot fetch records for {clazz}","messagePattern":"Cannot fetch records for (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/impl/StateStoreFileBaseImpl.java","lineNumber":270,"sourceCode":"        }\n      } else {\n        // Read records serially\n        callables.forEach(e -> {\n          try {\n            e.call();\n          } catch (Exception ex) {\n            LOG.error(\"Failed to retrieve record using file operations.\", ex);\n            throw new RuntimeException(ex);\n          }\n        });\n      }\n    } catch (Exception e) {\n      if (metrics != null) {\n        metrics.addFailure(monotonicNow() - start);\n      }\n      String msg = \"Cannot fetch records for \" + clazz.getSimpleName();\n      LOG.error(msg, e);\n      throw new IOException(msg, e);\n    }\n\n    if (metrics != null) {\n      metrics.addRead(monotonicNow() - start);\n    }\n    return new QueryResult<>(result, getTime());\n  }\n\n  /**\n   * Get the state store record from the given path (path/child) and add the record to the\n   * result list.\n   *\n   * @param clazz Class of the record.\n   * @param result The list of results record. The records would be added to it unless the given\n   * path represents old temp file.\n   * @param path The parent path.\n   * @param child The child path under the parent path. Both path and child completes the file\n   * location for the given record.","sourceCodeStart":252,"sourceCodeEnd":288,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/impl/StateStoreFileBaseImpl.java#L252-L288","documentation":"StateStoreFileBaseImpl backs the file-based State Store (local disk or HDFS): fetching all records for a record class lists the store directory and reads each child record file, often via per-record callables. Any exception during listing/reading — including the RuntimeException('Failed to retrieve record using file operations.') from a corrupted record — is counted as a metrics failure and rethrown as IOException('Cannot fetch records for <Class>') with the original cause attached.","triggerScenarios":"The state store directory is unreadable (permissions, missing mount) or its underlying HDFS NN is down for an HDFS-backed store; a single record file is corrupted or truncated (partial write); disk full or I/O errors during listing; file removed between list and read.","commonSituations":"Permissions changed on the state store directory after setup; HDFS-backed state store during an NN outage; unclean shutdown leaving a half-written record file; disk failures on a local-disk store.","solutions":["Read the logged cause — LOG.error(msg, e) prints the underlying exception identifying the file and I/O problem.","Fix filesystem access: permissions/ownership on the state store directory, mount the volume, or restore the backing HDFS.","If one corrupted record file is identified, quarantine/remove it; records such as membership regenerate from heartbeats, and mounts can be re-added from config.","Restart the router after the filesystem is healthy so the driver re-initializes."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Pre-flight the file store directory before router start\nPath storeDir = new Path(conf.get(\"dfs.federation.router.file-store.path\"));\nif (!fs.exists(storeDir) || !fs.getFileStatus(storeDir).getPermission()\n    .getUserAction().implies(FsAction.READWRITE)) {\n  throw new IllegalStateException(\"State store dir missing or not rw: \" + storeDir);\n}","typeGuard":null,"tryCatchPattern":"try {\n  QueryResult<T> r = recordStore.fetchAll();\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Cannot fetch records\")) {\n    Throwable cause = e.getCause(); // real IO / corruption reason\n    // if cause names one corrupted record file: quarantine it and retry,\n    // else fix filesystem permissions/mount and retry\n  } else { throw e; }\n}","preventionTips":["Pin permissions and ownership of the state store directory in provisioning (ansible/chef) so drift is impossible.","For HDFS-backed stores, monitor NN health; the file driver cannot fetch during NN outages.","Back up the state dir; corrupted single-record files can be dropped and regenerated from heartbeats/config."],"tags":["hdfs","router-based-federation","state-store","file-io","corruption"],"backgroundTag":"state-store-read-failure","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}