apache/beam · error

Error writing to ClickHouse. Retry attempt

Error message

Error writing to ClickHouse. Retry attempt[{}]

What it means

WARN logged in ClickHouseIO's flush retry loop: an attempt to insert the buffered rows failed and the connector is retrying with exponential backoff (BackOffUtils). If backoff is exhausted the loop rethrows as a runtime exception; this message specifically marks a transient, retryable failure during processElement/finishBundle flushing.

Solutions

1. Let the built-in retry with backoff handle transient ClickHouse overload
2. If retries exhaust, check ClickHouse server health/capacity and tune the backoff policy
3. Batch larger writes to reduce insert pressure

When it happens

Trigger: Thrown at sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIO.java:645 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3dc02027d0355f69. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIO.java:645

          if (response != null) {
            LOG.debug(
                "Successfully inserted {} rows out of {} into table {}. total size written {} bytes",
                response.getWrittenRows(),
                buffer.size(),
                table(),
                response.getWrittenBytes());
          } else {
            LOG.debug("Successfully inserted {} rows into table {}", buffer.size(), table());
          }

          buffer.clear();
          break;
        } catch (Exception e) {
          if (!BackOffUtils.next(Sleeper.DEFAULT, backOff)) {
            throw new RuntimeException("Failed to write to ClickHouse after retries", e);
          } else {
            retries.inc();
            LOG.warn(RETRY_ATTEMPT_LOG, attempt, e);
            attempt++;
          }
        }
      }
    }

    @AutoValue.Builder
    abstract static class Builder<T> {

      public abstract Builder<T> clickHouseUrl(String clickHouseUrl);

      public abstract Builder<T> database(String database);

      public abstract Builder<T> table(String table);

      public abstract Builder<T> maxInsertBlockSize(long maxInsertBlockSize);

      public abstract Builder<T> schema(TableSchema schema);

View on GitHub (pinned to 12126d8942)