apache/beam · critical · RuntimeException

Failed to write to ClickHouse after retries

Error message

Failed to write to ClickHouse after retries

What it means

ClickHouseIO's write DoFn buffers rows and flushes them to ClickHouse, retrying failures with a configured backoff. If every retry attempt is exhausted and BackOffUtils.next reports no more attempts, flush wraps the last exception in a RuntimeException("Failed to write to ClickHouse after retries"), failing the bundle.

Source

Thrown at sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIO.java:642

                      table(), new java.io.ByteArrayInputStream(data), ClickHouseFormat.RowBinary)
                  .get();

          if (response != null) {
            LOG.debug(
                "Successfully inserted {} rows out of {} into table {}. total size written {} bytes",
                response.getWrittenRows(),
                buffer.size(),
                table(),
                response.getWrittenBytes());
          } else {
            LOG.debug("Successfully inserted {} rows into table {}", buffer.size(), table());
          }

          buffer.clear();
          break;
        } catch (Exception e) {
          if (!BackOffUtils.next(Sleeper.DEFAULT, backOff)) {
            throw new RuntimeException("Failed to write to ClickHouse after retries", e);
          } else {
            retries.inc();
            LOG.warn(RETRY_ATTEMPT_LOG, attempt, e);
            attempt++;
          }
        }
      }
    }

    @AutoValue.Builder
    abstract static class Builder<T> {

      public abstract Builder<T> clickHouseUrl(String clickHouseUrl);

      public abstract Builder<T> database(String database);

      public abstract Builder<T> table(String table);

View on GitHub (pinned to 12126d8942)

Solutions

1. Check the chained cause exception for the root failure (auth, network, or server-side rejection)
2. Verify ClickHouse availability and connectivity from the Beam worker (host, port, credentials)
3. Increase withMaxRetries/backoff duration to tolerate longer outages
4. Reduce batch size (withBatchSize) if inserts are rejected for size/too-many-parts reasons

Example fix

// before
ClickHouseIO.<Row>write(url, table).withMaxRetries(3)
// after
ClickHouseIO.<Row>write(url, table).withMaxRetries(10) // plus fix the root cause in the chained exception

// pre-flight: verify ClickHouse reachability before running the pipeline
try (Connection c = DriverManager.getConnection(jdbcUrl)) { /* OK */ }

try {
  pipeline.run().waitUntilFinish();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("Failed to write to ClickHouse after retries")) {
    Throwable root = e.getCause(); // inspect the real failure
    LOG.error("ClickHouse write exhausted retries; root cause:", root);
  }
  throw e;
}

When it happens

Trigger: Sustained ClickHouse unavailability during flush: network partitions, ClickHouse server restarts, auth failures, table locks/too many parts, or oversized inserts that fail on every attempt until the retry budget is spent.

Common situations: ClickHouse cluster down or OOM during a Beam batch; DNS/network issues between the Beam worker and ClickHouse; rejected inserts due to max_insert_block_size or storage policy limits; transient errors that outlast the backoff window.

Related errors

• Properties cannot be null
• Property conflict: '%s' is already set to '%s' (likely from
• Failed to get table schema for table:
• Illegal access to pipeline after visitor traversal was compl
• Pipeline update will not be possible because the following t

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/fb969ffd19231217. Report an issue: GitHub.