{"record":{"id":"a8b7c949e555bb5d","repo":"apache/iceberg","slug":"failure-during-rewrite-process-for-group-a8b7c9","errorCode":null,"errorMessage":"Failure during rewrite process for group {}","messagePattern":"Failure during rewrite process for group (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java","lineNumber":280,"sourceCode":"    return new RewriteDataFilesCommitManager(\n        table, startingSnapshotId, useStartingSequenceNumber, commitSummary(), branch);\n  }\n\n  private Builder doExecute(\n      FileRewritePlan<FileGroupInfo, FileScanTask, DataFile, RewriteFileGroup> plan,\n      RewriteDataFilesCommitManager commitManager) {\n    ExecutorService rewriteService = rewriteService();\n\n    ConcurrentLinkedQueue<RewriteFileGroup> rewrittenGroups = new ConcurrentLinkedQueue<>();\n\n    Tasks.Builder<RewriteFileGroup> rewriteTaskBuilder =\n        Tasks.foreach(plan.groups())\n            .executeWith(rewriteService)\n            .stopOnFailure()\n            .noRetry()\n            .onFailure(\n                (fileGroup, exception) -> {\n                  LOG.warn(\n                      \"Failure during rewrite process for group {}\", fileGroup.info(), exception);\n                });\n\n    try {\n      rewriteTaskBuilder.run(\n          fileGroup -> {\n            rewrittenGroups.add(rewriteFiles(plan, fileGroup));\n          });\n    } catch (Exception e) {\n      // At least one rewrite group failed, clean up all completed rewrites\n      LOG.error(\n          \"Cannot complete rewrite, {} is not enabled and one of the file set groups failed to \"\n              + \"be rewritten. This error occurred during the writing of new files, not during the commit process. This \"\n              + \"indicates something is wrong that doesn't involve conflicts with other Iceberg operations. Enabling \"\n              + \"{} may help in this case but the root cause should be investigated. Cleaning up {} groups which finished \"\n              + \"being written.\",\n          PARTIAL_PROGRESS_ENABLED,\n          PARTIAL_PROGRESS_ENABLED,","sourceCodeStart":262,"sourceCodeEnd":298,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java#L262-L298","documentation":"RewriteDataFilesSparkAction.doExecute schedules one rewrite task per file group; the Tasks builder stops on the first failure and logs 'Failure during rewrite process for group <group>' with the exception. The affected group's files are not rewritten and the original files remain committed-untouched, so data is safe but compaction is incomplete.","triggerScenarios":"A rewrite group's task throws during doExecute — e.g. writer failures from schema issues, oversized groups exceeding memory, or read errors on the input files.","commonSituations":"Very large file groups causing OOM in executors; corrupted or concurrently-modified input files; partial-failure caching/AVRO schema evolution mismatches during rewrites.","solutions":["Read the chained exception for the failing group and fix the root cause (usually memory or file corruption)","Reduce max-file-group-size-bytes / partial-progress settings so groups are smaller and failures are isolated","Re-run the rewrite action after fixing — unmodified groups are simply retried"],"exampleFix":"// before\nactions.rewriteDataFiles(table).execute();\n// after\nactions.rewriteDataFiles(table)\n    .option(\"max-file-group-size-bytes\", \"1073741824\")\n    .option(\"partial-progress.enabled\", \"true\")\n    .execute();","handlingStrategy":"fallback","validationCode":"// keep groups small enough for executor memory\nlong groupBytes = /* max-file-group-size-bytes */ 0;\nif (groupBytes > executorMemory * 2) { /* lower the option */ }","typeGuard":null,"tryCatchPattern":"try {\n  RewriteDataFilesSparkAction rw = actions.rewriteDataFiles(table);\n  rw.option(\"partial-progress.enabled\", \"true\").execute();\n} catch (Exception e) {\n  // failing groups logged; fix cause and re-run remaining groups\n}","preventionTips":["Enable partial-progress so successful groups still commit","Size file groups relative to executor memory","Fix underlying file corruption before re-running rewrites"],"tags":["spark","compaction","rewrite","task-failure"],"backgroundTag":"invalid-state-transition","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}