{"record":{"id":"2192f6f831995b47","repo":"prestodb/presto","slug":"bigquery-error-end-of-avro-buffer","errorCode":"BIGQUERY_ERROR_END_OF_AVRO_BUFFER","errorMessage":"Error determining the end of Avro buffer","messagePattern":"Error determining the end of Avro buffer","errorType":"error_code","errorClass":"BigQueryException","httpStatus":null,"severity":"error","filePath":"presto-bigquery/src/main/java/com/facebook/presto/plugin/bigquery/BigQueryResultPageSource.java","lineNumber":321,"sourceCode":"            implements Iterator<GenericRecord>\n    {\n        GenericDatumReader<GenericRecord> reader;\n        BinaryDecoder decode;\n\n        AvroBinaryIterator(Schema avroSchema, byte[] buffer)\n        {\n            this.reader = new GenericDatumReader<>(avroSchema);\n            this.decode = new DecoderFactory().binaryDecoder(buffer, null);\n        }\n\n        @Override\n        public boolean hasNext()\n        {\n            try {\n                return !decode.isEnd();\n            }\n            catch (IOException e) {\n                throw new BigQueryException(BIGQUERY_ERROR_END_OF_AVRO_BUFFER, \"Error determining the end of Avro buffer\", e);\n            }\n        }\n\n        @Override\n        public GenericRecord next()\n        {\n            try {\n                return reader.read(null, decode);\n            }\n            catch (IOException e) {\n                throw new BigQueryException(BIGQUERY_ERROR_READING_NEXT_AVRO_RECORD, \"Error reading next Avro Record\", e);\n            }\n        }\n    }\n\n    static class AvroDecimalConverter\n    {\n        private static final Conversions.DecimalConversion AVRO_DECIMAL_CONVERSION = new Conversions.DecimalConversion();","sourceCodeStart":303,"sourceCodeEnd":339,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-bigquery/src/main/java/com/facebook/presto/plugin/bigquery/BigQueryResultPageSource.java#L303-L339","documentation":"The Avro record iterator's hasNext calls decode.isEnd() to check whether more records remain in the Avro buffer returned by BigQuery Storage; an IOException there is wrapped in BIGQUERY_ERROR_END_OF_AVRO_BUFFER. It means the underlying Avro stream/decoder failed while checking for more data, typically a transport/read failure mid-stream.","triggerScenarios":"Reading rows from a BigQuery Storage read session when the underlying gRPC-to-Avro stream is interrupted or the decoder hits a corrupt/truncated buffer during isEnd().","commonSituations":"Network interruptions to Google APIs; read session expiring mid-scan; truncated responses; transient Google backend errors.","solutions":["Retry the query/split — the failure is often transient","Re-create the BigQuery read session and re-run the scan","Check network connectivity and any proxy/firewall between Presto and Google APIs","Check Google Cloud status for BigQuery Storage API incidents"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// preflight: confirm the read session/stream is reachable before scanning\nReadSession session = bigQueryClient.createReadSession(readSessionRequest);\nif (session.getStreamsCount() == 0) {\n    throw new IllegalStateException(\"No streams returned for read session\");\n}","typeGuard":null,"tryCatchPattern":"int attempts = 0;\nwhile (attempts < 3) {\n    try {\n        return pageSource.getNextPage(...);\n    } catch (BigQueryException e) {\n        if (BIGQUERY_ERROR_END_OF_AVRO_BUFFER.getCode() == e.getErrorCode().getCode() && ++attempts < 3) {\n            backoff(attempts); continue; // transient stream error\n        }\n        throw e;\n    }\n}","preventionTips":["Enable retry/backoff on splits that read Avro streams","Monitor network stability between cluster and Google APIs","Keep read sessions short-lived; re-create on long scans","Watch Google Cloud status for BigQuery Storage incidents"],"tags":["bigquery","avro","io","stream"],"backgroundTag":"avro-stream-read-failure","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}