{"record":{"id":"7d44c01d1693c233","repo":"apache/druid","slug":"negative-size-footer-corrupt-or-truncated-file-s","errorCode":null,"errorMessage":"Negative-size footer. Corrupt or truncated file[%s]?","messagePattern":"Negative-size footer\\. Corrupt or truncated file\\[(.+?)\\]\\?","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/frame/file/FrameFile.java","lineNumber":216,"sourceCode":"        throw new IOE(\"File[%s] is too short (size[%,d])\", file, fileLength);\n      }\n\n      // Verify magic.\n      final byte[] buf = new byte[FrameFileWriter.TRAILER_LENGTH /* Larger than FrameFileWriter.MAGIC */];\n      final Memory bufMemory = Memory.wrap(buf, ByteOrder.LITTLE_ENDIAN);\n      randomAccessFile.readFully(buf, 0, FrameFileWriter.MAGIC.length);\n\n      if (!bufMemory.equalTo(0, Memory.wrap(FrameFileWriter.MAGIC), 0, FrameFileWriter.MAGIC.length)) {\n        throw new IOE(\"File[%s] is not a frame file\", file);\n      }\n\n      // Read number of frames and partitions.\n      randomAccessFile.seek(fileLength - FrameFileWriter.TRAILER_LENGTH);\n      randomAccessFile.readFully(buf, 0, FrameFileWriter.TRAILER_LENGTH);\n\n      final int footerLength = bufMemory.getInt(Integer.BYTES * 2L);\n      if (footerLength < 0) {\n        throw new ISE(\"Negative-size footer. Corrupt or truncated file[%s]?\", file);\n      } else if (footerLength > fileLength) {\n        throw new ISE(\"Oversize footer. Corrupt or truncated file[%s]?\", file);\n      }\n\n      final Memory wholeFileMemory;\n      final Memory footerMemory;\n\n      if (fileLength <= maxMmapSize) {\n        // Map entire file, use region for footer.\n        final MappedByteBufferHandler mapHandle = FileUtils.map(randomAccessFile, 0, fileLength);\n        sharedMapCloser = mapHandle;\n        wholeFileMemory = Memory.wrap(mapHandle.get(), ByteOrder.LITTLE_ENDIAN);\n\n        if (wholeFileMemory.getCapacity() != fileLength) {\n          // Check that the mapped file is the expected length. May differ if the file was updated while we're trying\n          // to map it.\n          throw new ISE(\"Memory map size does not match file size\");\n        }","sourceCodeStart":198,"sourceCodeEnd":234,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/frame/file/FrameFile.java#L198-L234","documentation":"FrameFile.open reads the trailer's footerLength field; a negative value is impossible in a well-formed file, so it indicates corruption or truncation. The check catches files whose trailer bytes were read from the wrong offset or were garbage (e.g. a truncated or externally modified file).","triggerScenarios":"Opening a frame file whose last TRAILER_LENGTH bytes do not encode a valid non-negative footerLength — typically a file truncated mid-write, padded, or written by an incompatible writer version.","commonSituations":"A task failed midway and the frame file was never finalized; the file was appended to or edited; transferring files in text/ASCII mode corrupted binary content; reading a file still being written by another process.","solutions":["Confirm the producing job completed successfully and finalized the file; re-run the writer job to regenerate the file.","Check whether the file is still being written — only read frame files after the writing task completes.","Verify the file was transferred in binary mode (no FTP/text-mode or encoding transformations).","Compare file size against the expected output of the producing job to detect truncation."],"exampleFix":"// before\nFrameFile.open(partialFile, maxMmapSize); // throws on truncated download\n// after\nif (isTaskComplete(jobHandle)) {\n  FrameFile.open(partialFile, maxMmapSize);\n} else {\n  // wait for writer completion or re-run the job\n}","handlingStrategy":"validation","validationCode":"if (file.length() != expectedSizeFromProducer) {\n  throw new IllegalStateException(\"Frame file truncated: \" + file);\n}","typeGuard":null,"tryCatchPattern":"try {\n  FrameFile.open(file, maxMmapSize);\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"Corrupt or truncated\")) {\n    regenerateFrameFile(file); // re-run producer task\n  } else {\n    throw e;\n  }\n}","preventionTips":["Only consume frame files after the writing task reports success/completion.","Verify file size against the producer's report before opening.","Avoid text-mode or encoding-transforming file transfers for binary frame files.","Monitor producer jobs for crashes that leave unfinalized files."],"tags":["io","corruption","truncation","druid"],"backgroundTag":"checksum-mismatch","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}