apache/beam · error · IOException

Insert failed:

Error message

Insert failed: 

What it means

Thrown during streaming insertAll result processing when BigQuery returns per-row InsertErrors and one error row has a null index. A null index means the error applies to the whole request rather than a specific row (e.g. table-level failure), so the code cannot attribute it; it throws an IOException listing the error and all other collected errors.

Solutions

1. Verify the destination table exists and its schema matches the rows being written
2. Check the allErrors content in the message for the underlying request-level cause (notFound, invalid schema, access denied)
3. Recreate or re-point the table and re-run; avoid concurrent schema mutations during writes
4. Wrap the write with BigQueryIO's retry policy / DLQ for request-level failures

Example fix

// before
bq.tables().delete(project, dataset, table).execute(); // while pipeline streams into it
// after
// stop the pipeline or write to a new table before deleting the old one

// before streaming, verify table exists and schema matches
Table table = datasetService.getTable(ref);
if (table == null || !schemaMatches(table.getDefinition().getSchema(), rowSchema)) {
  throw new IllegalStateException("Destination table missing or schema mismatch");
}

try {
  writeResult = rows.apply(BigQueryIO.writeTableRows()...);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Insert failed:")) {
    // parse request-level error; recreate/verify table, then re-run
  }
}

When it happens

Trigger: BigQuery returns TableDataInsertAllResponse.InsertErrors with getIndex()==null — typically when the entire insert request failed: table does not exist (or was deleted mid-run), schema mismatch, or access/quota error at the request level.

Common situations: Destination table deleted or recreated while a streaming write is in flight; streaming buffer/schema conflicts after a schema change; wrong table reference in dynamic destinations.

Related errors

• Cannot start an export job since table
• Append to stream failed with invalid offset of
• Append to stream failed with Status Code . The stream may…
• BigQuery source must be split before being read
• BigQuery table " + tableReference + " not found. If you…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ada13e563a2c0a84. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java:1293

                      rows,
                      maxThrottlingMsec,
                      sleeper,
                      streamingInsertsResults)));
          strideIndices.add(strideIndex);
          retTotalDataSize += dataSize;
          rows = new ArrayList<>();
        }

        try {
          for (int i = 0; i < futures.size(); i++) {
            List<TableDataInsertAllResponse.InsertErrors> errors = futures.get(i).get();
            if (errors == null) {
              continue;
            }

            for (TableDataInsertAllResponse.InsertErrors error : errors) {
              if (error.getIndex() == null) {
                throw new IOException("Insert failed: " + error + ", other errors: " + allErrors);
              }
              int errorIndex = error.getIndex().intValue() + strideIndices.get(i);
              failedIndices.add(errorIndex);
              if (retryPolicy.shouldRetry(new InsertRetryPolicy.Context(error))) {
                allErrors.add(error);
                retryRows.add(rowsToPublish.get(errorIndex));
                // TODO (https://github.com/apache/beam/issues/20891): Select the retry rows(using
                // errorIndex) from the batch of rows which attempted insertion in this call.
                // Not the entire set of rows in rowsToPublish.
                if (retryIds != null) {
                  // retryIds is non-null exactly when idsToPublish is non-null; see where both are
                  // initialized above.
                  retryIds.add(checkStateNotNull(idsToPublish).get(errorIndex));
                }
              } else {
                numFailedRows += 1;
                errorContainer.add(failedInserts, error, ref, rowsToPublish.get(errorIndex));
              }

View on GitHub (pinned to 12126d8942)