apache/beam · error · IOException

Interrupted while waiting before retrying insertAll

Error message

Interrupted while waiting before retrying insertAll

What it means

Thrown inside DatasetServiceImpl.insertAll when the thread sleeping between streaming-insert retries (throttling backoff before the next insertAll attempt) is interrupted. The InterruptedException is converted to an IOException with this message so it can propagate through the Beam sink's checked-exception API.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java:1092

                    .withOperation("insert_all")
                    .withFullResourceName(BigQueryHelpers.toTableFullResourceName(ref))
                    .create()) {
              LOG.info(
                  "BigQuery insertAll error, retrying: {}",
                  ApiErrorExtractor.INSTANCE.getErrorMessage(e));
            }
            try {
              long nextBackOffMillis = backoff1.nextBackOffMillis();
              if (nextBackOffMillis == BackOff.STOP) {
                throw e;
              }
              sleeper.sleep(nextBackOffMillis);
              totalBackoffMillis += nextBackOffMillis;
              final long totalBackoffMillisSoFar = totalBackoffMillis;
              maxThrottlingMsec.getAndUpdate(current -> Math.max(current, totalBackoffMillisSoFar));
              result.updateRetriedRowsWithStatus(errorReason, rows.size());
            } catch (InterruptedException interrupted) {
              throw new IOException("Interrupted while waiting before retrying insertAll");
            }
          }
        }
      }
    }

    @VisibleForTesting
    <T> long insertAll(
        TableReference ref,
        List<FailsafeValueInSingleWindow<TableRow, TableRow>> rowList,
        @Nullable List<String> insertIdList,
        BackOff backoff,
        FluentBackoff rateLimitBackoffFactory,
        final Sleeper sleeper,
        InsertRetryPolicy retryPolicy,
        List<ValueInSingleWindow<T>> failedInserts,
        ErrorContainer<T> errorContainer,
        boolean skipInvalidRows,

View on GitHub (pinned to 12126d8942)

Solutions

Let the pipeline finish or cancel gracefully instead of hard-killing workers
Check for user code that interrupts executor threads doing BigQuery writes
Reduce retryable error volume (smaller batches, backoff tuning) so the code spends less time in the sleep/retry loop
Catch and inspect the IOException message to confirm interruption rather than a real insert failure

Example fix

// before
Thread.interrupt(); // somewhere in user DoFn teardown while insert in flight
// after
// allow in-flight writes to complete; cancel via pipeline.cancel() or await completion

Defensive patterns

Strategy: try-catch

Try / catch

try {
  pipeline.run().waitUntilFinish();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Interrupted while waiting before retrying insertAll")) {
    // thread was interrupted during job shutdown; treat as cancellation, not data loss
  }
}

Prevention

Don't interrupt worker threads executing BigQuery writes
Cancel jobs via pipeline.cancel() and let stages drain
Reduce throttling retries by staying within BigQuery streaming quotas
Use deterministic row IDs so re-running after interruption is idempotent

When it happens

Trigger: BigQuery streaming insert receives retryable errors (e.g. 503 rate limiting), the code sleeps via sleeper.sleep(nextBackOffMillis) before retrying, and the executing thread is interrupted (pipeline cancel, worker shutdown, or another thread interrupting).

Common situations: Cancelling a running Dataflow/Flink job mid-insert; worker teardown during autoscaling; user code interrupting threads around a blocking BigQuery write.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/13b1bda82c86c1a7. Report an issue: GitHub.