{"record":{"id":"fb969ffd19231217","repo":"apache/beam","slug":"failed-to-write-to-clickhouse-after-retries","errorCode":null,"errorMessage":"Failed to write to ClickHouse after retries","messagePattern":"Failed to write to ClickHouse after retries","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIO.java","lineNumber":642,"sourceCode":"                      table(), new java.io.ByteArrayInputStream(data), ClickHouseFormat.RowBinary)\n                  .get();\n\n          if (response != null) {\n            LOG.debug(\n                \"Successfully inserted {} rows out of {} into table {}. total size written {} bytes\",\n                response.getWrittenRows(),\n                buffer.size(),\n                table(),\n                response.getWrittenBytes());\n          } else {\n            LOG.debug(\"Successfully inserted {} rows into table {}\", buffer.size(), table());\n          }\n\n          buffer.clear();\n          break;\n        } catch (Exception e) {\n          if (!BackOffUtils.next(Sleeper.DEFAULT, backOff)) {\n            throw new RuntimeException(\"Failed to write to ClickHouse after retries\", e);\n          } else {\n            retries.inc();\n            LOG.warn(RETRY_ATTEMPT_LOG, attempt, e);\n            attempt++;\n          }\n        }\n      }\n    }\n\n    @AutoValue.Builder\n    abstract static class Builder<T> {\n\n      public abstract Builder<T> clickHouseUrl(String clickHouseUrl);\n\n      public abstract Builder<T> database(String database);\n\n      public abstract Builder<T> table(String table);\n","sourceCodeStart":624,"sourceCodeEnd":660,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIO.java#L624-L660","documentation":"ClickHouseIO's write DoFn buffers rows and flushes them to ClickHouse, retrying failures with a configured backoff. If every retry attempt is exhausted and BackOffUtils.next reports no more attempts, flush wraps the last exception in a RuntimeException(\"Failed to write to ClickHouse after retries\"), failing the bundle.","triggerScenarios":"Sustained ClickHouse unavailability during flush: network partitions, ClickHouse server restarts, auth failures, table locks/too many parts, or oversized inserts that fail on every attempt until the retry budget is spent.","commonSituations":"ClickHouse cluster down or OOM during a Beam batch; DNS/network issues between the Beam worker and ClickHouse; rejected inserts due to max_insert_block_size or storage policy limits; transient errors that outlast the backoff window.","solutions":["Check the chained cause exception for the root failure (auth, network, or server-side rejection)","Verify ClickHouse availability and connectivity from the Beam worker (host, port, credentials)","Increase withMaxRetries/backoff duration to tolerate longer outages","Reduce batch size (withBatchSize) if inserts are rejected for size/too-many-parts reasons"],"exampleFix":"// before\nClickHouseIO.<Row>write(url, table).withMaxRetries(3)\n// after\nClickHouseIO.<Row>write(url, table).withMaxRetries(10) // plus fix the root cause in the chained exception","handlingStrategy":"retry","validationCode":"// pre-flight: verify ClickHouse reachability before running the pipeline\ntry (Connection c = DriverManager.getConnection(jdbcUrl)) { /* OK */ }","typeGuard":null,"tryCatchPattern":"try {\n  pipeline.run().waitUntilFinish();\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Failed to write to ClickHouse after retries\")) {\n    Throwable root = e.getCause(); // inspect the real failure\n    LOG.error(\"ClickHouse write exhausted retries; root cause:\", root);\n  }\n  throw e;\n}","preventionTips":["Increase withMaxRetries and backoff to cover transient ClickHouse outages","Size batches with withBatchSize to avoid too-many-parts rejections","Monitor ClickHouse health from Beam workers and alert before retry budgets exhaust","Always inspect the chained cause — the message is generic, the cause is specific"],"tags":["java","clickhouse","beam","write-failure","retry-exhausted"],"backgroundTag":"database-write-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}