{"record":{"id":"7c9fa0851103f2dd","repo":"apache/hadoop","slug":"found-more-than-one-meta-files-matches","errorCode":null,"errorMessage":"Found more than one meta files: {matches}","messagePattern":"Found more than one meta files: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetUtil.java","lineNumber":106,"sourceCode":"\n  /** Find the corresponding meta data file from a given block file */\n  public static File findMetaFile(final File blockFile) throws IOException {\n    final String prefix = blockFile.getName() + \"_\";\n    final File parent = blockFile.getParentFile();\n    final File[] matches = parent.listFiles(new FilenameFilter() {\n      @Override\n      public boolean accept(File dir, String name) {\n        return dir.equals(parent) && name.startsWith(prefix)\n            && name.endsWith(Block.METADATA_EXTENSION);\n      }\n    });\n\n    if (matches == null || matches.length == 0) {\n      throw new FileNotFoundException(\n          \"Meta file not found, blockFile=\" + blockFile);\n    }\n    if (matches.length > 1) {\n      throw new IOException(\"Found more than one meta files: \" \n          + Arrays.asList(matches));\n    }\n    return matches[0];\n  }\n\n  public static FileDescriptor openAndSeek(File file, long offset)\n      throws IOException {\n    RandomAccessFile raf = null;\n    try {\n      raf = new RandomAccessFile(file, \"r\");\n      if (offset > 0) {\n        raf.seek(offset);\n      }\n      return raf.getFD();\n    } catch(IOException ioe) {\n      IOUtils.cleanupWithLogger(null, raf);\n      throw ioe;\n    }","sourceCodeStart":88,"sourceCodeEnd":124,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetUtil.java#L88-L124","documentation":"findMetaFile found two or more '<block>_<gs>.meta' files for one block file. HDFS allows exactly one; duplicates typically survive a DN crash between creating a new-generation-stamp meta and deleting the old one, leaving both stamps on disk.","triggerScenarios":"Crash during finalize or generation-stamp bump (old .meta not yet deleted); interrupted append or truncate recovery; the same block restored from backup alongside a newer meta.","commonSituations":"Unclean shutdowns; kill -9 testing; restored volume snapshots; releases before duplicate-meta reconciliation in DirectoryScanner.","solutions":["List the matches from the exception: keep the .meta whose generation stamp equals the NN's committed stamp (usually the highest) and delete the stale one.","Restart the DN or trigger the DirectoryScanner after cleanup so the volume map re-syncs.","Verify block health with 'hdfs fsck' afterwards.","If duplicates recur, find the crash source (OOM, kill -9 during finalize) since clean shutdowns delete atomically."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"File[] metas = blockFile.getParentFile().listFiles(\n    (d, n) -> n.startsWith(blockFile.getName() + \"_\") && n.endsWith(\".meta\"));\nif (metas != null && metas.length > 1) {\n  // keep the highest-generation-stamp meta, delete the rest before scanning\n}","typeGuard":null,"tryCatchPattern":"try {\n  File meta = FsDatasetUtil.findMetaFile(blockFile);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"more than one meta\")) {\n    // duplicate metas after a crash: keep the one matching the committed GS\n  }\n}","preventionTips":["Avoid kill -9 during finalize; clean shutdowns delete old metas atomically.","After crashes, check finalized dirs for duplicate '<block>_<gs>.meta' files.","Run the DirectoryScanner after unclean restarts to reconcile volume state."],"tags":["hdfs","datanode","metadata","generation-stamp","crash-recovery"],"backgroundTag":"hdfs-duplicate-block-metadata","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}