{"record":{"id":"ad637a35db0df994","repo":"apache/flink","slug":"encountered-exception-when-reading-from-avro-forma","errorCode":null,"errorMessage":"Encountered exception when reading from avro format file","messagePattern":"Encountered exception when reading from avro format file","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AbstractAvroBulkFormat.java","lineNumber":218,"sourceCode":"            this.reader = reader;\n            this.reuse = reuse;\n            this.converter = converter;\n        }\n\n        @Override\n        public boolean hasNext() {\n            return numRecordsRemaining > 0;\n        }\n\n        @Override\n        public T next() {\n            try {\n                numRecordsRemaining--;\n                // reader.next merely deserialize bytes in memory to java objects\n                // and will not read from file\n                return converter.apply(reader.next(reuse));\n            } catch (IOException e) {\n                throw new RuntimeException(\n                        \"Encountered exception when reading from avro format file\", e);\n            }\n        }\n    }\n}\n","sourceCodeStart":200,"sourceCodeEnd":224,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AbstractAvroBulkFormat.java#L200-L224","documentation":"AvroBlockIterator.next() calls the underlying Avro DataFileReader to materialize the next record; an IOException there (corrupt block, truncated file, codec failure) is wrapped in RuntimeException. Because hasNext() only counts remaining records declared for the block, I/O errors surface here rather than during hasNext.","triggerScenarios":"Iterating a batch whose block bytes are unreadable: truncated Avro file (incomplete upload), corrupt data from a bad transfer, an unsupported/missing codec implementation for the block, or a schema-incompatible reader trying to decode records.","commonSituations":"Reading an Avro file that was incompletely written to S3/HDFS (crashed producer); files compressed with a codec whose native library is missing on the TaskManager; schema evolution where the file schema no longer resolves against the reader schema; bit rot on long-lived storage.","solutions":["Inspect the nested IOException: codec errors point to missing codec jars, 'Invalid sync' points to truncation/corruption.","Verify file integrity: avro-tools jar tocat / getmeta on a copy; compare with the producer's output.","Ensure required Avro codec dependencies (e.g. xz, snappy, zstd native libs) are on the TaskManager classpath.","For truncated files, re-generate or exclude the file; enable exactly-once end-to-end so producers cannot publish partial files."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// verify file readability before handing it to the source\ntry (DataFileReader<Object> r =\n        (DataFileReader<Object>) DataFileReader.openReader(\n                new SeekableInputFile(path), new GenericDatumReader<>())) {\n    // header and first block parse OK\n}","typeGuard":null,"tryCatchPattern":"try {\n    while (it.hasNext()) { emit(it.next()); }\n} catch (RuntimeException e) {\n    if (e.getCause() instanceof IOException) {\n        // corrupt block/truncated file — quarantine the file\n        quarantine(path);\n    }\n    throw e;\n}","preventionTips":["Commit files atomically (rename/_SUCCESS pattern) so partial files are never read.","Ship required codec natives (snappy/zstd/xz) on TaskManagers.","Validate Avro files with avro-tools before ingesting external data."],"tags":["avro","format","data-corruption","io","codec"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}