apache/hadoop · error · IOException

Error deserializing string.

Error message

Error deserializing string.

What it means

Utils.fromCSVString() decodes the CSV-escaping produced by Utils.toCSVString(), which wraps strings in single quotes ('...'). If the input's first character is not a quote character, the input is not a CSV-encoded record string and the method throws IOException 'Error deserializing string.'.

Source

Thrown at hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/record/Utils.java:156

      case '%':
        sb.append("%25");
        break;
      default:
        sb.append(c);
      }
    }
    return sb.toString();
  }
  
  /**
   *
   * @param s
   * @throws java.io.IOException
   * @return
   */
  static String fromCSVString(String s) throws IOException {
    if (s.charAt(0) != '\'') {
      throw new IOException("Error deserializing string.");
    }
    int len = s.length();
    StringBuilder sb = new StringBuilder(len-1);
    for (int i = 1; i < len; i++) {
      char c = s.charAt(i);
      if (c == '%') {
        char ch1 = s.charAt(i+1);
        char ch2 = s.charAt(i+2);
        i += 2;
        if (ch1 == '0' && ch2 == '0') {
          sb.append('\0');
        } else if (ch1 == '0' && ch2 == 'A') {
          sb.append('\n');
        } else if (ch1 == '0' && ch2 == 'D') {
          sb.append('\r');
        } else if (ch1 == '2' && ch2 == 'C') {
          sb.append(',');
        } else if (ch1 == '7' && ch2 == 'D') {

View on GitHub (pinned to 2add963021)

Solutions

1. Only feed fromCSVString with values produced by Utils.toCSVString() on the same Hadoop version
2. Log the offending value hex-dumped when the exception fires to find where the framing was lost
3. Check that the record being parsed is aligned (not off-by-one field) and not truncated
4. Replace org.apache.hadoop.record usage with Avro/Writable, which have unambiguous framing

// guard before decoding
static boolean isCsvEncodedString(String s) {
  return s != null && s.length() >= 2 && s.charAt(0) == '\'' && s.charAt(s.length() - 1) == '\'';
}
if (!isCsvEncodedString(field)) throw new IllegalArgumentException("Not a record-CSV string: " + field);

catch IOException around CsvRecordInput.readString()/Utils.fromCSVString and reject the whole record (log the raw field), since partial CSV state is unrecoverable.

When it happens

Trigger: Calling fromCSVString (directly or via CsvRecordInput.readString) with a raw unquoted string, the wrong field/column from a CSV record, a truncated record, or data produced by a different serializer or Hadoop version with a different framing convention.

Common situations: Interoperability bugs where one side writes with toCSVString and the other side reads plain text, hand-crafted CSV test data missing the leading quote, or record streams corrupted by encoding/transfer so the framing character is lost.

Related errors

• Error deserializing buffer.
• Error serializing {}
• Invalid UTF-8 representation.
• Invalid UTF-8 byte {} at offset {} in length of {}
• Illegal Unicode Codepoint {} in stream.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0285ed33c602ddbf. Report an issue: GitHub.