{"record":{"id":"bf87009d69e24531","repo":"apache/iceberg","slug":"is-true-but-rewrite-commits-failed-check-th","errorCode":null,"errorMessage":"{} is true but {} rewrite commits failed. Check the logs to determine why the individual commits failed. If this is persistent it may help to increase {} which will split the rewrite operation into smaller commits.","messagePattern":"(.+?) is true but (.+?) rewrite commits failed\\. Check the logs to determine why the individual commits failed\\. If this is persistent it may help to increase (.+?) which will split the rewrite operation into smaller commits\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java","lineNumber":358,"sourceCode":"        .onFailure(\n            (fileGroup, exception) -> {\n              LOG.error(\"Failure during rewrite group {}\", fileGroup.info(), exception);\n              rewriteFailures.add(\n                  ImmutableRewriteDataFiles.FileGroupFailureResult.builder()\n                      .info(fileGroup.info())\n                      .dataFilesCount(fileGroup.inputFileNum())\n                      .build());\n            })\n        .run(fileGroup -> commitService.offer(rewriteFiles(plan, fileGroup)));\n    rewriteService.shutdown();\n\n    // stop commit service\n    commitService.close();\n\n    int totalCommits = Math.min(plan.totalGroupCount(), maxCommits);\n    int failedCommits = totalCommits - commitService.succeededCommits();\n    if (failedCommits > 0 && failedCommits <= maxFailedCommits) {\n      LOG.warn(\n          \"{} is true but {} rewrite commits failed. Check the logs to determine why the individual \"\n              + \"commits failed. If this is persistent it may help to increase {} which will split the rewrite operation \"\n              + \"into smaller commits.\",\n          PARTIAL_PROGRESS_ENABLED,\n          failedCommits,\n          PARTIAL_PROGRESS_MAX_COMMITS);\n    } else if (failedCommits > maxFailedCommits) {\n      String errorMessage =\n          String.format(\n              Locale.ROOT,\n              \"%s is true but %d rewrite commits failed. This is more than the maximum allowed failures of %d. \"\n                  + \"Check the logs to determine why the individual commits failed. If this is persistent it may help to \"\n                  + \"increase %s which will split the rewrite operation into smaller commits.\",\n              PARTIAL_PROGRESS_ENABLED,\n              failedCommits,\n              maxFailedCommits,\n              PARTIAL_PROGRESS_MAX_COMMITS);\n      throw new RuntimeException(errorMessage);","sourceCodeStart":340,"sourceCodeEnd":376,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java#L340-L376","documentation":"With partial-progress enabled, RewriteDataFiles allows up to maxFailedCommits failed commits; when the number of failed commits falls within that allowance, the action logs this warning naming the partial-progress properties so users know some rewrite commits did not land but the job still succeeds.","triggerScenarios":"Running RewriteDataFiles with partial-progress.enabled=true where (totalGroups capped at max-commits) minus commitService.succeededCommits() is > 0 and <= maxFailedCommits — i.e. one or more group commits failed but stayed within tolerance.","commonSituations":"Concurrent writers causing commit conflicts (CommitStateUnknown / validation failures); too few max commits for large rewrites forcing oversized commits that conflict; metastore contention.","solutions":["Raise partial-progress.max-commits so each commit is smaller and less likely to conflict with concurrent writers.","Inspect earlier ERROR logs to find why individual commits failed (usually ValidationException from concurrent appends).","Reduce concurrent write load on the table or retry the rewrite when the table is quiet.","If failures exceed the allowance the job errors out — keeping partial-progress.enabled true limits wasted work either way."],"exampleFix":"// before: too few, too-large commits that fail under concurrency\nspark.sql(\"CALL cat.sys.rewrite_data_files(table => 'db.t', \" +\n  \"options => map('partial-progress.enabled','true'))\");\n// after: split into more, smaller commits\nspark.sql(\"CALL cat.sys.rewrite_data_files(table => 'db.t', \" +\n  \"options => map('partial-progress.enabled','true','partial-progress.max-commits','50'))\");","handlingStrategy":"retry","validationCode":"int maxCommits = Integer.parseInt(props.getOrDefault(\"partial-progress.max-commits\", \"10\"));\n// ensure maxCommits is high relative to expected group count","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set partial-progress.max-commits proportional to group count for large rewrites.","Avoid heavy concurrent writers during rewrite windows.","Retry the rewrite after transient commit conflicts; table state stays consistent.","Grep logs for ValidationException to identify conflicting operations."],"tags":["spark","rewrite-data-files","commit-conflict","partial-progress"],"backgroundTag":"conflicting-config-options","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}