{"record":{"id":"f3fdd89881e03006","repo":"apache/hadoop","slug":"invalid-size-for-file-metadata-object","errorCode":null,"errorMessage":"Invalid size: {} for file metadata object","messagePattern":"Invalid size: (.+?) for file metadata object","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java","lineNumber":774,"sourceCode":"      return new TreeMap<Text, Text>(this.theMetadata);\n    }\n    \n    @Override\n    public void write(DataOutput out) throws IOException {\n      out.writeInt(this.theMetadata.size());\n      Iterator<Map.Entry<Text, Text>> iter =\n        this.theMetadata.entrySet().iterator();\n      while (iter.hasNext()) {\n        Map.Entry<Text, Text> en = iter.next();\n        en.getKey().write(out);\n        en.getValue().write(out);\n      }\n    }\n\n    @Override\n    public void readFields(DataInput in) throws IOException {\n      int sz = in.readInt();\n      if (sz < 0) throw new IOException(\"Invalid size: \" + sz + \" for file metadata object\");\n      this.theMetadata = new TreeMap<Text, Text>();\n      for (int i = 0; i < sz; i++) {\n        Text key = new Text();\n        Text val = new Text();\n        key.readFields(in);\n        val.readFields(in);\n        this.theMetadata.put(key, val);\n      }    \n    }\n\n    @Override\n    public boolean equals(Object other) {\n      if (other == null) {\n        return false;\n      }\n      if (other.getClass() != this.getClass()) {\n        return false;\n      } else {","sourceCodeStart":756,"sourceCodeEnd":792,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java#L756-L792","documentation":"SequenceFile.Metadata.readFields throws this IOException when the metadata entry-count field read from the file header is negative. The writer always writes a non-negative count, so a negative value means the bytes being parsed are not a valid sequence-file metadata block: the file is truncated, corrupted, or not a SequenceFile at all. It typically escapes from the SequenceFile.Reader constructor while the header is parsed.","triggerScenarios":"Constructing SequenceFile.Reader on a file whose metadata section is cut off (truncated transfer), that is 0 bytes or garbage, that was written by an incompatible tool/version, or whose header bytes were damaged.","commonSituations":"Interrupted copyToLocal/putFromLocal leaving partial files; a job pointed at a text/Avro file where a .seq path was expected; bad disks or bit rot on archival data; reading files produced by an old Hadoop version with a different header layout.","solutions":["Confirm it is a sequence file: hadoop fs -cat /path | head -c 4 must show the 'SEQ' magic bytes","Verify completeness against the source (compare sizes, hadoop fs -checksum) and re-copy if it differs","Delete or regenerate the corrupt/truncated file from its source data","Pin writer and reader jobs to the same Hadoop version to avoid header-layout skew"],"exampleFix":"// before\nSequenceFile.Reader r = new SequenceFile.Reader(conf, Reader.file(p)); // Invalid size IOException\n\n// after: reject non-sequence files before the Reader parses the header\ntry (FSDataInputStream in = p.getFileSystem(conf).open(p)) {\n  byte[] magic = new byte[3];\n  in.readFully(magic);\n  if (!new String(magic).equals(\"SEQ\")) throw new IOException(p + \" is not a SequenceFile\");\n}\nSequenceFile.Reader r = new SequenceFile.Reader(conf, Reader.file(p));","handlingStrategy":"try-catch","validationCode":"try (FSDataInputStream in = fs.open(p)) {\n  byte[] magic = new byte[3];\n  in.readFully(magic);\n  if (!\"SEQ\".equals(new String(magic))) {\n    throw new IOException(p + \" is not a sequence file (bad magic)\");\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  reader = new SequenceFile.Reader(conf, Reader.file(p));\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Invalid size\")) {\n    // header corruption: quarantine the file, do not retry the same bytes\n    quarantine(p);\n  } else { throw e; }\n}","preventionTips":["Validate the SEQ magic and file size before handing inputs to SequenceFile.Reader","Verify transfers with checksums (hadoop fs -checksum) so truncated files never enter the pipeline","Keep writer and reader jobs on the same Hadoop version to avoid header-layout mismatches"],"tags":["hadoop","sequence-file","corruption","file-format"],"backgroundTag":"corrupt-sequence-file","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}