What causes "Line too long in text file: " in prestodb/presto?

Fix or regenerate the source file so lines are within the limit and properly newline-terminated

prestodb/presto · error · PrestoException

HIVE_BAD_DATA

HIVE_BAD_DATA

Error message

Line too long in text file:

What it means

While reading a Hive text-file table, GenericHiveRecordCursor.advanceNextPosition catches TextLineLengthLimitExceededException from the record reader and rethrows it as HIVE_BAD_DATA 'Line too long in text file: <path>'. The underlying TextInputFormat enforces a maximum line length (default 100MB-ish buffer limit); lines longer than this indicate corrupted or non-text data.

Source

Thrown at presto-hive/src/main/java/com/facebook/presto/hive/GenericHiveRecordCursor.java:253

    {
        try {
            if (closed || !recordReader.next(key, value)) {
                close();
                return false;
            }

            // reset loaded flags
            Arrays.fill(loaded, false);

            // decode value
            rowData = deserializer.deserialize(value);

            return true;
        }
        catch (IOException | SerDeException | RuntimeException e) {
            closeWithSuppression(this, e);
            if (e instanceof TextLineLengthLimitExceededException) {
                throw new PrestoException(HIVE_BAD_DATA, "Line too long in text file: " + path, e);
            }
            throw new PrestoException(HIVE_CURSOR_ERROR, e);
        }
    }

    @Override
    public boolean getBoolean(int fieldId)
    {
        checkState(!closed, "Cursor is closed");

        validateType(fieldId, boolean.class);
        if (!loaded[fieldId]) {
            parseBooleanColumn(fieldId);
        }
        return booleans[fieldId];
    }

    private void parseBooleanColumn(int column)

View on GitHub (pinned to 55bb57d202)

Solutions

Fix or regenerate the source file so lines are within the limit and properly newline-terminated
Inspect the file (e.g. awk to find lines longer than the limit) and split or repair it
Load the data as a correctly compressed/typed format (ORC/Parquet) instead of raw text
If the limit is genuinely too small, use a record reader/format with a larger max line length configuration

Example fix

// before: one giant line, no newline
writer.write("a,b,c"); // ... never writes '\n'
// after
writer.write("a,b,c\n");

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-flight: detect overlong lines before reading via Hive
long maxLen = Files.lines(Paths.get(path)).mapToLong(String::length).max().orElse(0);
if (maxLen > MAX_SUPPORTED_LINE_LENGTH) {
    throw new IllegalStateException("File has lines exceeding limit: " + path);
}

Try / catch

try {
    cursor.advanceNextPosition();
} catch (PrestoException e) {
    if ("HIVE_BAD_DATA".equals(e.getErrorCode().getName()) && e.getMessage().startsWith("Line too long in text file:")) {
        // quarantine/skip the file, report path to the data pipeline owner
    }
    throw e;
}

Prevention

Ensure writers always emit newline terminators
Validate text files (max line length) before loading into Hive tables
Store large blobs in a binary/object format, not text tables
Use ORC/Parquet for data without natural line boundaries

When it happens

Trigger: Scanning a text/CSV Hive table whose file contains a single line exceeding the record reader's line-length limit — usually due to missing newlines, corrupted files, or binary data written as text.

Common situations: A CSV without proper line breaks (unterminated quoted field spanning everything), a log file that never flushes newlines, accidentally loading a binary/compressed-as-text file into a text table.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/201059119806da43. Report an issue: GitHub.