{"record":{"id":"71cea84b2af2435d","repo":"apache/hadoop","slug":"meta-data-not-found-for-block","errorCode":null,"errorMessage":"Meta-data not found for {block}","messagePattern":"Meta-data not found for (.+?)","errorType":"exception","errorClass":"FileNotFoundException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java","lineNumber":323,"sourceCode":"\n      /* \n       * (corruptChecksumOK, meta_file_exist): operation\n       * True,   True: will verify checksum  \n       * True,  False: No verify, e.g., need to read data from a corrupted file \n       * False,  True: will verify checksum\n       * False, False: throws IOException file not found\n       */\n      DataChecksum csum = null;\n      if (verifyChecksum || sendChecksum) {\n        LengthInputStream metaIn = null;\n        boolean keepMetaInOpen = false;\n        try {\n          DataNodeFaultInjector.get().throwTooManyOpenFiles();\n          metaIn = datanode.data.getMetaDataInputStream(block);\n          if (!corruptChecksumOk || metaIn != null) {\n            if (metaIn == null) {\n              //need checksum but meta-data not found\n              throw new FileNotFoundException(\"Meta-data not found for \" +\n                  block);\n            }\n\n            // The meta file will contain only the header if the NULL checksum\n            // type was used, or if the replica was written to transient storage.\n            // Also, when only header portion of a data packet was transferred\n            // and then pipeline breaks, the meta file can contain only the\n            // header and 0 byte in the block data file.\n            // Checksum verification is not performed for replicas on transient\n            // storage.  The header is important for determining the checksum\n            // type later when lazy persistence copies the block to non-transient\n            // storage and computes the checksum.\n            int expectedHeaderSize = BlockMetadataHeader.getHeaderSize();\n            if (!replica.isOnTransientStorage() &&\n                metaIn.getLength() >= expectedHeaderSize) {\n              checksumIn = new DataInputStream(new BufferedInputStream(\n                  metaIn, IO_FILE_BUFFER_SIZE));\n","sourceCodeStart":305,"sourceCodeEnd":341,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java#L305-L341","documentation":"BlockSender needs the block's meta file to verify or send checksums (verifyChecksum || sendChecksum and corruptChecksumOk false). When datanode.data.getMetaDataInputStream(block) returns null, there is no checksum file alongside the block file, so a FileNotFoundException is thrown and (per the catch block) the datanode invalidates the missing block with the NameNode so it can be re-replicated.","triggerScenarios":"The block file exists but its .meta companion was deleted or never written: interrupted finalize, meta file lost in a disk incident, an improperly copied/moved block directory, or a transient/lazy-persist replica that legitimately has no checksums being read with checksum verification required.","commonSituations":"Manual block-dir copying or disk repair that dropped .meta files; filesystem corruption hitting only small meta files; reading transient (RAM-disk lazy-persist) replicas through a path that demands checksums.","solutions":["Check whether the .meta file exists in the block's finalizado directory on disk (blk_<id>_<gen>.meta next to blk_<id>).","If lost, the DN already reports the replica missing/invalid to the NN — let re-replication restore a complete replica from another datanode.","Audit for the root cause: failed disks, improper rsync of storage dirs, or operators deleting meta files.","Never reconstruct block dirs by hand without their .meta files; use re-replication or backups instead."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":"static boolean isMissingMeta(IOException e) {\n  return e instanceof FileNotFoundException &&\n      e.getMessage() != null && e.getMessage().startsWith(\"Meta-data not found for\");\n}","tryCatchPattern":"try {\n  sender = new BlockSender(block, startOffset, length, /*corruptChecksumOk*/false, ...);\n} catch (FileNotFoundException e) {\n  if (isMissingMeta(e)) {\n    reportReplicaCorrupt(block);      // DN invalidates it, NN re-replicates\n    reader = readFromOtherReplica();  // fall back to a complete replica\n  } else {\n    throw e;\n  }\n}","preventionTips":["Never copy or restore block directories without their blk_*.meta companions.","Monitor datanode 'invalidateMissingBlock' traffic — spikes indicate disappearing meta files.","Schedule volume health checks; lost meta files often precede full disk failure."],"tags":["hdfs","datanode","read-path","metadata","checksum"],"backgroundTag":"missing-block-metadata","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}