prestodb/presto · error · PrestoException

HIVE_BAD_DATA

HIVE_BAD_DATA

Error message

Line too long in text file: %s

What it means

A text-format file contains a line longer than the maximum allowed by the LineRecordReader (default 10MB per line). HiveUtil.createRecordReader converts the TextLineLengthLimitExceededException from the underlying reader into HIVE_BAD_DATA with the file path, signaling corrupt or malformed source data rather than a connector bug.

Source

Thrown at presto-hive/src/main/java/com/facebook/presto/hive/HiveUtil.java:320

        try {
            RecordReader<WritableComparable, Writable> recordReader = (RecordReader<WritableComparable, Writable>) inputFormat.getRecordReader(fileSplit, jobConf, Reporter.NULL);

            int headerCount = getHeaderCount(schema);
            //  Only skip header rows when the split is at the beginning of the file
            if (start == 0 && headerCount > 0) {
                Utilities.skipHeader(recordReader, headerCount, recordReader.createKey(), recordReader.createValue());
            }

            int footerCount = getFooterCount(schema);
            if (footerCount > 0) {
                recordReader = new FooterAwareRecordReader<>(recordReader, footerCount, jobConf);
            }

            return recordReader;
        }
        catch (IOException e) {
            if (e instanceof TextLineLengthLimitExceededException) {
                throw new PrestoException(HIVE_BAD_DATA, "Line too long in text file: " + path, e);
            }

            throw new PrestoException(HIVE_CANNOT_OPEN_SPLIT, format("Error opening Hive split %s (offset=%s, length=%s) using %s: %s",
                    path,
                    start,
                    length,
                    getInputFormatName(schema),
                    firstNonNull(e.getMessage(), e.getClass().getName())),
                    e);
        }
    }

    public static void setReadColumns(Configuration configuration, List<Integer> readHiveColumnIndexes)
    {
        configuration.set(READ_COLUMN_IDS_CONF_STR, Joiner.on(',').join(readHiveColumnIndexes));
        configuration.setBoolean(READ_ALL_COLUMNS, false);
    }

View on GitHub (pinned to 55bb57d202)

Solutions

  1. Repair or re-write the source file with proper line breaks and correct format
  2. Increase the per-line limit via jobconf if genuinely long lines are expected (mapreduce.input.linerecordreader.line.maxlength)
  3. Change the table/storage format to one matching the actual data (e.g. sequence/parquet for binary data)

Example fix

-- before: binary data declared as TEXTFILE
CREATE TABLE t WITH (format = 'TEXTFILE') ...
-- after
CREATE TABLE t WITH (format = 'PARQUET') ... -- match the real file format
Defensive patterns

Strategy: try-catch

Validate before calling

long maxLen = maxLineLengthBytes("hdfs://path/file"); // precompute longest line, compare to record reader limit
if (maxLen > 10L * 1024 * 1024) throw new IllegalStateException("file has lines exceeding text record reader limit");

Try / catch

try { readSplit(split); } catch (PrestoException e) { if (e.getErrorCode().getCode() == StandardErrorCode.HIVE_BAD_DATA.getCode()) { log.error("Oversized line in {}", path); /* quarantine file or use a format with no line limit */ } else throw e; }

Prevention

When it happens

Trigger: Reading a Hive text/textfile split whose single line exceeds the record reader's max line length — typically a file missing newline terminators, a corrupted text file, or binary data (e.g. gzip-less avoro/sequence bytes) written as text.

Common situations: Unterminated final line in a text file produced by a broken writer; log files with one giant line; accidentally registering non-text data as TEXTFILE format.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/6aa078c4673b969d. Report an issue: GitHub.