{"record":{"id":"41efef990452a984","repo":"apache/hadoop","slug":"corrupted-vlong-encoding","errorCode":null,"errorMessage":"Corrupted VLong encoding","messagePattern":"Corrupted VLong encoding","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Utils.java","lineNumber":243,"sourceCode":"      case 1:\n        return ((firstByte + 112) << 24) | (in.readUnsignedShort() << 8)\n            | in.readUnsignedByte();\n      case 0:\n        int len = firstByte + 129;\n        switch (len) {\n          case 4:\n            return in.readInt();\n          case 5:\n            return ((long) in.readInt()) << 8 | in.readUnsignedByte();\n          case 6:\n            return ((long) in.readInt()) << 16 | in.readUnsignedShort();\n          case 7:\n            return ((long) in.readInt()) << 24 | (in.readUnsignedShort() << 8)\n                | in.readUnsignedByte();\n          case 8:\n            return in.readLong();\n          default:\n            throw new IOException(\"Corrupted VLong encoding\");\n        }\n      default:\n        throw new RuntimeException(\"Internal error\");\n    }\n  }\n\n  /**\n   * Write a String as a VInt n, followed by n Bytes as in Text format.\n   * \n   * @param out out.\n   * @param s s.\n   * @throws IOException raised on errors performing I/O.\n   */\n  public static void writeString(DataOutput out, String s) throws IOException {\n    if (s != null) {\n      Text text = new Text(s);\n      byte[] buffer = text.getBytes();\n      int len = text.getLength();","sourceCodeStart":225,"sourceCodeEnd":261,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/Utils.java#L225-L261","documentation":"IOException from Utils.readVLong(DataInput): the varint's first byte lands in the undefined part of the encoding space. Per the decoder, first bytes in [-128, -126] compute len = firstByte + 129 in {1,2,3}, which no case handles, so the default throws 'Corrupted VLong encoding'. Valid multi-byte forms start at firstByte -125 (len 4) up to -120 (len 8); shorter extended forms don't exist in this format.","triggerScenarios":"Decoding a VLong/VInt prefix whose first byte is 0x80..0x82 (-128..-126): random bytes where a length was expected, mid-stream desynchronization, or truncated/garbled TFile regions. Because TFile prefixes every record and index element with varints, corruption surfaces here quickly.","commonSituations":"Bit rot and truncated files, reading a stream at the wrong offset, or interop code that assumed Hadoop Writable varint semantics (which differ: WritableUtils accepts different leading bytes) and wrote TFiles with the wrong encoder.","solutions":["Restore or regenerate the file — the byte stream is not decodable varint data.","If you produce TFiles with custom tooling, write length prefixes with Utils.writeVInt/writeVLong (not WritableUtils) so the encodings match.","Catch IOException around full-file parsing in batch jobs and quarantine the file."],"exampleFix":"// before (writing prefixes with the wrong varint encoder)\norg.apache.hadoop.io.WritableUtils.writeVInt(out, len);\n// after\norg.apache.hadoop.io.file.tfile.Utils.writeVInt(out, len);","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  TFile.Reader r = new TFile.Reader(fsdis, fileLength, conf);\n} catch (IOException e) {\n  if (\"Corrupted VLong encoding\".equals(e.getMessage())) { // bytes are not TFile varints\n    quarantine(path);\n  } else throw e;\n}","preventionTips":["If you generate TFile-format bytes yourself, encode prefixes with org.apache.hadoop.io.file.tfile.Utils, not WritableUtils (encodings differ).","Verify stream offsets before reading; misalignment is the most common cause of varint garbage.","Catch IOException per file in batch pipelines and continue with the rest."],"tags":["tfile","hadoop-common","utils","varint","encoding","data-corruption"],"backgroundTag":"malformed-file-format","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}