apache/hadoop · error · IOException

Too many bytes before delimiter: " + bytesConsumed

Error message

Too many bytes before delimiter: " + bytesConsumed

What it means

The custom-delimiter path readCustomLine() scans for recordDelimiterBytes. Partial delimiter matches at buffer boundaries are deferred via ambiguousByteCount, so a delimiter that never fully materializes keeps the loop consuming; when bytesConsumed exceeds Integer.MAX_VALUE without a complete match it throws IOException("Too many bytes before delimiter: N") — the same 2 GiB guard as the newline path.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java:368

        // since it is now certain that the split did not split a delimiter we
        // should not read the next record: clear the flag otherwise duplicate
        // records could be generated
        unsetNeedAdditionalRecordAfterSplit();
      }
      if (appendLength > 0) {
        str.append(buffer, startPosn, appendLength);
        txtLength += appendLength;
      }
      if (bufferPosn >= bufferLength) {
        if (delPosn > 0 && delPosn < recordDelimiterBytes.length) {
          ambiguousByteCount = delPosn;
          bytesConsumed -= ambiguousByteCount; //to be consumed in next
        }
      }
    } while (delPosn < recordDelimiterBytes.length 
        && bytesConsumed < maxBytesToConsume);
    if (bytesConsumed > Integer.MAX_VALUE) {
      throw new IOException("Too many bytes before delimiter: " + bytesConsumed);
    }
    return (int) bytesConsumed; 
  }

  /**
   * Read from the InputStream into the given Text.
   * @param str the object to store the given line
   * @param maxLineLength the maximum number of bytes to store into str.
   * @return the number of bytes read including the newline
   * @throws IOException if the underlying stream throws
   */
  public int readLine(Text str, int maxLineLength) throws IOException {
    return readLine(str, maxLineLength, Integer.MAX_VALUE);
  }

  /**
   * Read from the InputStream into the given Text.
   * @param str the object to store the given line

View on GitHub (pinned to 2add963021)

Solutions

1. Hexdump a record boundary (od -c) and rebuild the delimiter byte[] to match the file exactly
2. Fix the delimiter configuration (e.g. textinputformat.record.delimiter) to the actual separator
3. Choose a shorter delimiter guaranteed to appear per record, or pre-split oversized records
4. If the data itself is corrupt, restore it from source

Example fix

// before: guessed delimiter never occurs in the file
LineReader r = new LineReader(in, "\r\n".getBytes(StandardCharsets.UTF_8));

// after: delimiter verified against actual data (LF-only producer)
LineReader r = new LineReader(in, "\n".getBytes(StandardCharsets.UTF_8));

// verify the delimiter bytes actually occur in the file before processing
byte[] probe = Files.readAllBytes(Path.of(file)); // or stream first MB
byte[] d = "\u0001".getBytes(StandardCharsets.UTF_8);
boolean found = indexOf(probe, d) >= 0;
if (!found) throw new IOException("record delimiter not present in input");

try { n = reader.readLine(text, maxLen); } catch (IOException e) { if (e.getMessage() != null && e.getMessage().startsWith("Too many bytes before delimiter")) { throw new IOException("Delimiter mismatch: configured bytes never terminate a record", e); } throw e; }

When it happens

Trigger: LineReader configured with delimiter bytes that never fully occur in the data ("\r\n" on an LF-only file, a tab or sentinel string that is absent); binary data repeatedly producing partial delimiter-prefix matches; a legitimate single record over 2 GiB.

Common situations: textinputformat.record.delimiter (or a hand-built LineReader) set to the wrong byte sequence after a producer changed format; sentinel-delimited exports where the sentinel gained escaping; job upgrades that dropped the delimiter setting.

Related errors

• Too many bytes before newline: " + bytesConsumed
• Mark not supported
• Mark not set
• %s: Stream is closed!
• Wrong key length. Required ${options.getBitLength()}, but go

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1e142ba6b64b1539. Report an issue: GitHub.