apache/flink · error · RuntimeException

Encountered exception when reading from avro format file

Error message

Encountered exception when reading from avro format file

What it means

AvroBlockIterator.next() calls the underlying Avro DataFileReader to materialize the next record; an IOException there (corrupt block, truncated file, codec failure) is wrapped in RuntimeException. Because hasNext() only counts remaining records declared for the block, I/O errors surface here rather than during hasNext.

Source

Thrown at flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AbstractAvroBulkFormat.java:218

            this.reader = reader;
            this.reuse = reuse;
            this.converter = converter;
        }

        @Override
        public boolean hasNext() {
            return numRecordsRemaining > 0;
        }

        @Override
        public T next() {
            try {
                numRecordsRemaining--;
                // reader.next merely deserialize bytes in memory to java objects
                // and will not read from file
                return converter.apply(reader.next(reuse));
            } catch (IOException e) {
                throw new RuntimeException(
                        "Encountered exception when reading from avro format file", e);
            }
        }
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

Inspect the nested IOException: codec errors point to missing codec jars, 'Invalid sync' points to truncation/corruption.
Verify file integrity: avro-tools jar tocat / getmeta on a copy; compare with the producer's output.
Ensure required Avro codec dependencies (e.g. xz, snappy, zstd native libs) are on the TaskManager classpath.
For truncated files, re-generate or exclude the file; enable exactly-once end-to-end so producers cannot publish partial files.

Defensive patterns

Strategy: try-catch

Validate before calling

// verify file readability before handing it to the source
try (DataFileReader<Object> r =
        (DataFileReader<Object>) DataFileReader.openReader(
                new SeekableInputFile(path), new GenericDatumReader<>())) {
    // header and first block parse OK
}

Try / catch

try {
    while (it.hasNext()) { emit(it.next()); }
} catch (RuntimeException e) {
    if (e.getCause() instanceof IOException) {
        // corrupt block/truncated file — quarantine the file
        quarantine(path);
    }
    throw e;
}

Prevention

Commit files atomically (rename/_SUCCESS pattern) so partial files are never read.
Ship required codec natives (snappy/zstd/xz) on TaskManagers.
Validate Avro files with avro-tools before ingesting external data.

When it happens

Trigger: Iterating a batch whose block bytes are unreadable: truncated Avro file (incomplete upload), corrupt data from a bad transfer, an unsupported/missing codec implementation for the block, or a schema-incompatible reader trying to decode records.

Common situations: Reading an Avro file that was incompletely written to S3/HDFS (crashed producer); files compressed with a codec whose native library is missing on the TaskManager; schema evolution where the file schema no longer resolves against the reader schema; bit rot on long-lived storage.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/ad637a35db0df994. Report an issue: GitHub.