{"record":{"id":"07715735bad2981b","repo":"apache/hadoop","slug":"fail-to-get-checksum-since-file-is-under-const","errorCode":null,"errorMessage":"Fail to get checksum, since file {} is under construction.","messagePattern":"Fail to get checksum, since file (.+?) is under construction\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java","lineNumber":1965,"sourceCode":"   * @return The checksum\n   * @see DistributedFileSystem#getFileChecksum(Path)\n   */\n  public MD5MD5CRC32FileChecksum getFileChecksum(String src, long length)\n      throws IOException {\n    return (MD5MD5CRC32FileChecksum) getFileChecksumInternal(\n        src, length, ChecksumCombineMode.MD5MD5CRC);\n  }\n\n  protected LocatedBlocks getBlockLocations(String src,\n                                            long length) throws IOException {\n    //get block locations for the file range\n    LocatedBlocks blockLocations = callGetBlockLocations(namenode,\n        src, 0, length);\n    if (null == blockLocations) {\n      throw new FileNotFoundException(\"File does not exist: \" + src);\n    }\n    if (blockLocations.isUnderConstruction()) {\n      throw new IOException(\"Fail to get checksum, since file \" + src\n          + \" is under construction.\");\n    }\n\n    return blockLocations;\n  }\n\n  protected IOStreamPair connectToDN(DatanodeInfo dn, int timeout,\n                                     Token<BlockTokenIdentifier> blockToken)\n      throws IOException {\n    return DFSUtilClient.connectToDN(dn, timeout, conf, saslClient,\n        socketFactory, getConf().isConnectToDnViaHostname(), this, blockToken);\n  }\n\n  /**\n   * Infer the checksum type for a replica by sending an OP_READ_BLOCK\n   * for the first byte of that replica. This is used for compatibility\n   * with older HDFS versions which did not include the checksum type in\n   * OpBlockChecksumResponseProto.","sourceCodeStart":1947,"sourceCodeEnd":1983,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java#L1947-L1983","documentation":"getFileChecksum requires finalized blocks with stable CRCs. If the file's last block is still under construction — some writer holds the file open for create/append — getBlockLocations reports isUnderConstruction and the client aborts with IOException('Fail to get checksum, since file ... is under construction.') rather than returning a partial checksum.","triggerScenarios":"Checksumming a file that the same process or another client is still writing: a leaked FSDataOutputStream (an exception path skipped close), verification steps running before the writer's committer closed files, or append-mode log files audited while open.","commonSituations":"Output verification racing job commit; long-lived append writers (log aggregation) checked by auditors; close() failures swallowed earlier leaving the lease and UC state behind.","solutions":["Fix the writer: guarantee close via try-with-resources on every FSDataOutputStream so no file is left under construction.","Sequence the pipeline: writers emit a completion marker (_SUCCESS or final atomic rename) after close; verifiers only run past the marker.","If a writer may legitimately still be active, bounded-retry the checksum until the file is no longer under construction.","If the lease is orphaned (writer died), wait for lease recovery or trigger it, then retry."],"exampleFix":"// before\nFileChecksum cs = fs.getFileChecksum(path);\n// IOException: Fail to get checksum, since file <path> is under construction.\n\n// after: bounded wait for the writer to finish\nFileChecksum cs = null;\nfor (int i = 0; i < 30; i++) {\n  try {\n    cs = fs.getFileChecksum(path);\n    break;\n  } catch (IOException e) {\n    if (!e.getMessage().contains(\"under construction\")) { throw e; }\n    Thread.sleep(1000);\n  }\n}","handlingStrategy":"retry","validationCode":"// @InterfaceAudience.Private but public: check UC state before checksumming\nHdfsFileStatus st = ((DistributedFileSystem) fs).getClient().getFileInfo(path);\nif (st != null && st.isUnderConstruction()) {\n  // a writer still holds the file: wait for close before getFileChecksum\n}","typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  if (e.getMessage().contains(\"under construction\")) {\n    // bounded wait/retry until the writer closes, then retry the checksum\n  } else { throw e; }\n}","preventionTips":["Use try-with-resources for every FSDataOutputStream; audit exception paths for skipped close().","Gate verification on completion markers (_SUCCESS / final rename) emitted after writers close.","For orphaned leases, trigger or await lease recovery before verifying."],"tags":["hdfs","file-checksum","under-construction","stream-lifecycle","lease"],"backgroundTag":"file-under-construction","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}