apache/hadoop · error · IOException

Found more than one meta files: {matches}

Error message

Found more than one meta files: {matches}

What it means

findMetaFile found two or more '<block>_<gs>.meta' files for one block file. HDFS allows exactly one; duplicates typically survive a DN crash between creating a new-generation-stamp meta and deleting the old one, leaving both stamps on disk.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetUtil.java:106

  /** Find the corresponding meta data file from a given block file */
  public static File findMetaFile(final File blockFile) throws IOException {
    final String prefix = blockFile.getName() + "_";
    final File parent = blockFile.getParentFile();
    final File[] matches = parent.listFiles(new FilenameFilter() {
      @Override
      public boolean accept(File dir, String name) {
        return dir.equals(parent) && name.startsWith(prefix)
            && name.endsWith(Block.METADATA_EXTENSION);
      }
    });

    if (matches == null || matches.length == 0) {
      throw new FileNotFoundException(
          "Meta file not found, blockFile=" + blockFile);
    }
    if (matches.length > 1) {
      throw new IOException("Found more than one meta files: " 
          + Arrays.asList(matches));
    }
    return matches[0];
  }

  public static FileDescriptor openAndSeek(File file, long offset)
      throws IOException {
    RandomAccessFile raf = null;
    try {
      raf = new RandomAccessFile(file, "r");
      if (offset > 0) {
        raf.seek(offset);
      }
      return raf.getFD();
    } catch(IOException ioe) {
      IOUtils.cleanupWithLogger(null, raf);
      throw ioe;
    }

View on GitHub (pinned to 2add963021)

Solutions

1. List the matches from the exception: keep the .meta whose generation stamp equals the NN's committed stamp (usually the highest) and delete the stale one.
2. Restart the DN or trigger the DirectoryScanner after cleanup so the volume map re-syncs.
3. Verify block health with 'hdfs fsck' afterwards.
4. If duplicates recur, find the crash source (OOM, kill -9 during finalize) since clean shutdowns delete atomically.

File[] metas = blockFile.getParentFile().listFiles(
    (d, n) -> n.startsWith(blockFile.getName() + "_") && n.endsWith(".meta"));
if (metas != null && metas.length > 1) {
  // keep the highest-generation-stamp meta, delete the rest before scanning
}

try {
  File meta = FsDatasetUtil.findMetaFile(blockFile);
} catch (IOException e) {
  if (e.getMessage().contains("more than one meta")) {
    // duplicate metas after a crash: keep the one matching the committed GS
  }
}

When it happens

Trigger: Crash during finalize or generation-stamp bump (old .meta not yet deleted); interrupted append or truncate recovery; the same block restored from backup alongside a newer meta.

Common situations: Unclean shutdowns; kill -9 testing; restored volume snapshots; releases before duplicate-meta reconciliation in DirectoryScanner.

Related errors

• Failed to parse generation stamp: blockFile={blockFile}, met
• Replica gen stamp < block genstamp, block={block}, replica={
• Meta-data not found for {block}
• The meta file length {metaIn.getLength()} is less than the e
• Failed to create temporary file for {}. File {} should not

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7c9fa0851103f2dd. Report an issue: GitHub.