apache/hadoop · error · IOException

Error deserializing buffer.

Error message

Error deserializing buffer.

What it means

Utils.fromCSVBuffer() expects the CSV wire form of a Buffer to start with '#' followed by pairs of hex digits (as written by toCSVBuffer). If the first character is not '#', it throws IOException 'Error deserializing buffer.' — the value being read is not a CSV-serialized buffer at all.

Source

Thrown at hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java:237

   * @return
   */
  static String toCSVBuffer(Buffer buf) {
    StringBuilder sb = new StringBuilder("#");
    sb.append(buf.toString());
    return sb.toString();
  }
  
  /**
   * Converts a CSV-serialized representation of buffer to a new
   * Buffer
   * @param s CSV-serialized representation of buffer
   * @throws java.io.IOException
   * @return Deserialized Buffer
   */
  static Buffer fromCSVBuffer(String s)
    throws IOException {
    if (s.charAt(0) != '#') {
      throw new IOException("Error deserializing buffer.");
    }
    if (s.length() == 1) { return new Buffer(); }
    int blen = (s.length()-1)/2;
    byte[] barr = new byte[blen];
    for (int idx = 0; idx < blen; idx++) {
      char c1 = s.charAt(2*idx+1);
      char c2 = s.charAt(2*idx+2);
      barr[idx] = (byte)Integer.parseInt(""+c1+c2, 16);
    }
    return new Buffer(barr);
  }
  
  private static int utf8LenForCodePoint(final int cpt) throws IOException {
    if (cpt >=0 && cpt <= 0x7F) {
      return 1;
    }
    if (cpt >= 0x80 && cpt <= 0x07FF) {
      return 2;

View on GitHub (pinned to 2add963021)

Solutions

1. Confirm writer and reader field order/types match — the '#' sentinel is written only by toCSVBuffer()
2. Log and hex-dump the failing field to identify what was actually supplied
3. Regenerate the serialized data with the matching org.apache.hadoop.record version
4. Replace record/CSV serialization with Avro, which has an explicit schema

static boolean isCsvEncodedBuffer(String s) {
  if (s == null || s.isEmpty() || s.charAt(0) != '#') return false;
  String hex = s.substring(1);
  return hex.isEmpty() || hex.matches("([0-9a-fA-F]{2})+");
}

catch IOException from readBuffer/fromCSVBuffer; report field position and raw value, treat as schema mismatch (string vs buffer).

When it happens

Trigger: CsvRecordInput.readBuffer() (or direct fromCSVBuffer calls) receiving a plain string, a hex blob without the '#' sentinel, or a value shifted from another field/column of the record.

Common situations: Schema drift between writer and reader: the writer wrote a string where the reader expects a buffer, records truncated in transit, or hand-built CSV payloads missing the '#' prefix.

Related errors

• Error deserializing string.
• Error serializing {}
• Invalid UTF-8 representation.
• Invalid UTF-8 byte {} at offset {} in length of {}
• Illegal Unicode Codepoint {} in stream.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/00fe16af60de6909. Report an issue: GitHub.