apache/seatunnel · error · IOException

File [%s] has fewer lines than expected to skip.

Error message

File [%s] has fewer lines than expected to skip.

What it means

CsvReadStrategy.readProcess skips skip_header_number lines before parsing. If the reader hits EOF before all requested header lines are skipped (the file has fewer lines than skip_header_number), a plain IOException is thrown stating the file lacks enough lines. Split-range reads bypass this skip logic entirely.

Source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/CsvReadStrategy.java:116

            Map<String, String> partitionsMap,
            String currentFileName)
            throws IOException {
        log.info(
                "Start reading CSV file: {}, split start: {}, split length: {}",
                currentFileName,
                split.getStart(),
                split.getLength());
        final boolean useSplitRead = isSplitReadEnabled(split);
        try (BufferedReader reader =
                        createBomAwareBufferedReader(
                                wrapInputStream(inputStream, split), encoding);
                CSVParser csvParser = new CSVParser(reader, getCSVFormat(split))) {
            // skip lines
            // if split range is used, no need to skip
            if (!useSplitRead) {
                for (int i = 0; i < skipHeaderNumber; i++) {
                    if (reader.readLine() == null) {
                        throw new IOException(
                                String.format(
                                        "File [%s] has fewer lines than expected to skip.",
                                        currentFileName));
                    }
                }
            }
            // read header lines
            List<String> headers = getHeaders(csvParser, split);
            // Clean up BOM characters (\uFEFF) in the header to solve occasional BOM residue
            // issues
            List<String> cleanedHeaders =
                    headers.stream()
                            .map(header -> header.replace("\uFEFF", ""))
                            .collect(Collectors.toList());
            for (CSVRecord csvRecord : csvParser) {
                HashMap<Integer, String> fieldIdValueMap = new HashMap<>();
                for (int i = 0; i < cleanedHeaders.size(); i++) {
                    // the user input schema may not contain all the columns in the csv header

View on GitHub (pinned to cf67b549a7)

Solutions

  1. Lower skip_header_number to at most (file line count).
  2. Remove or fix skip_header_number if files have no header (use 0).
  3. Verify the source files are not truncated/empty and check currentFileName in the message.

Example fix

// before
skip_header_number = 5
// after
skip_header_number = 1
Defensive patterns

Strategy: validation

Validate before calling

// Java, before reading: ensure file has at least skipHeaderNumber lines
long lines = countLines(file);
if (skipHeaderNumber > lines) {
    throw new IllegalArgumentException("skip_header_number=" + skipHeaderNumber + " exceeds file lines=" + lines);
}

Try / catch

try {
    rows = readProcess(inputStream, partitionsMap, fileName);
} catch (IOException e) {
    if (e.getMessage().contains("fewer lines than expected to skip")) {
        log.error("Header skip exceeds file length for {}", fileName);
    }
    throw e;
}

Prevention

When it happens

Trigger: Reading a CSV with skip_header_number = N where the split's file contains fewer than N lines, and useSplitRead is false (no split-range read).

Common situations: Header count copied from another file; empty or truncated CSV files; miscomputed skip value such as 3 when the file has only a 1-line header; reading many small files where some are headerless.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/dce4c6afa17e5f77. Report an issue: GitHub.