{"record":{"id":"526ce761419d58f2","repo":"apache/seatunnel","slug":"commit-failed","errorCode":"COMMIT_FAILED","errorMessage":"FlushRows did not reach expected offset. stream=%s, expected=%d, actual=%d","messagePattern":"FlushRows did not reach expected offset\\. stream=(.+?), expected=(.+?), actual=(.+?)","errorType":"error_code","errorClass":"BigQueryConnectorException","httpStatus":null,"severity":"critical","filePath":"seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/committer/BigQueryCommitter.java","lineNumber":77,"sourceCode":"                        .collect(Collectors.toList());\n\n        if (bufferedCommitInfos.isEmpty()) {\n            return Collections.emptyList();\n        }\n\n        try (BigQueryWriteClient client = BigQueryClientFactory.getWriteClient(config)) {\n            for (BigQueryCommitInfo info : bufferedCommitInfos) {\n                FlushRowsRequest request =\n                        FlushRowsRequest.newBuilder()\n                                .setWriteStream(info.getStreamName())\n                                .setOffset(Int64Value.of(info.getFlushOffset()))\n                                .build();\n\n                FlushRowsResponse response = client.flushRows(request);\n\n                long flushedOffset = response.getOffset();\n                if (flushedOffset < info.getFlushOffset()) {\n                    throw new BigQueryConnectorException(\n                            BigQueryConnectorErrorCode.COMMIT_FAILED,\n                            String.format(\n                                    \"FlushRows did not reach expected offset. stream=%s, expected=%d, actual=%d\",\n                                    info.getStreamName(), info.getFlushOffset(), flushedOffset));\n                }\n\n                log.info(\n                        \"Successfully flushed BigQuery buffered stream {} to offset {}\",\n                        info.getStreamName(),\n                        info.getFlushOffset());\n            }\n        } catch (Exception e) {\n            throw new BigQueryConnectorException(BigQueryConnectorErrorCode.COMMIT_FAILED, e);\n        }\n\n        return Collections.emptyList();\n    }\n","sourceCodeStart":59,"sourceCodeEnd":95,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-connectors-v2/connector-bigquery/src/main/java/org/apache/seatunnel/connectors/bigquery/sink/committer/BigQueryCommitter.java#L59-L95","documentation":"Thrown by BigQueryCommitter.commit when the BigQuery FlushRows RPC returns an offset lower than the offset the writer recorded as needing to be flushed. It means the streaming buffer did not durably commit all rows the sink believed it had written, so committing the checkpoint would silently lose data. SeaTunnel aborts the commit rather than ack inconsistent offsets.","triggerScenarios":"Raised inside commit() after client.flushRows(request) when response.getOffset() < info.getFlushOffset(). Happens when BigQuery drops or partially applies rows from the streaming buffer, or when state offsets are inconsistent across a restart/restore.","commonSituations":"BigQuery streaming buffer throttling or transient backend errors; job restored from an old checkpoint whose flush offset is ahead of what BigQuery actually persisted; table concurrently modified/deleted so flush applies to a stale stream.","solutions":["Check the BigQuery table's streaming buffer status (bq show / INFORMATION_SCHEMA) and retry the job; transient flush shortfalls often succeed on retry.","Verify the checkpoint from which the job was restored is the latest completed one, not an older savepoint with a stale flushOffset.","Confirm network/proxy stability between the cluster and BigQuery; retried RPCs returning stale offsets indicate dropped requests.","If persistent, reduce sink batch/flush frequency and enable retries in BigQuery client options, then re-run the pipeline."],"exampleFix":"// before\nlong flushedOffset = response.getOffset();\nif (flushedOffset < info.getFlushOffset()) {\n    throw new BigQueryConnectorException(...COMMIT_FAILED...);\n}\n// after\nlong flushedOffset = response.getOffset();\nif (flushedOffset < info.getFlushOffset()) {\n    // retry flush a bounded number of times before failing the commit\n    for (int i = 0; i < 3 && flushedOffset < info.getFlushOffset(); i++) {\n        response = client.flushRows(request);\n        flushedOffset = response.getOffset();\n    }\n    if (flushedOffset < info.getFlushOffset()) {\n        throw new BigQueryConnectorException(...COMMIT_FAILED...);\n    }\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// catch COMMIT_FAILED and retry the job from the last completed checkpoint\ntry {\n    job.submit(cfg);\n} catch (BigQueryConnectorException e) {\n    if (BigQueryConnectorErrorCode.COMMIT_FAILED.equals(e.getErrorCode())) {\n        resumeFromLatestCheckpoint();\n    } else { throw e; }\n}","preventionTips":["Always restore from the latest completed checkpoint, not older savepoints.","Monitor BigQuery streaming buffer stats and quota/throttle errors.","Keep stable connectivity between the cluster and BigQuery (avoid flaky proxies/NAT)."],"tags":["bigquery","checkpoint","data-loss","flush"],"backgroundTag":"unexpected-response-shape","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}