{"record":{"id":"4b2314b31ba7511f","repo":"apache/beam","slug":"caught-retentionpolicynotmet-error-while-rewriting-to-a","errorCode":null,"errorMessage":"Caught retentionPolicyNotMet error while rewriting to a bucket with retention policy. Skipping because destination {} and source {} are considered identical because their MD5 Hashes are equal.","messagePattern":"Caught retentionPolicyNotMet error while rewriting to a bucket with retention policy\\. Skipping because destination (.+?) and source (.+?) are considered identical because their MD5 Hashes are equal\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java","lineNumber":1147,"sourceCode":"          // Treat a missing source as a successful rewrite.\n          readyToEnqueue = false;\n          lastError = null;\n        } else {\n          throw new FileNotFoundException(\n              String.format(\n                  \"Rewrite from %s to %s has failed. Either source or sink not found. \"\n                      + \"Failed with error: %s\",\n                  from.toString(), to.toString(), e.getMessage()));\n        }\n      } else if (e.getCode() == 403\n          && e.getErrors().size() == 1\n          && e.getErrors().get(0).getReason().equals(\"retentionPolicyNotMet\")) {\n        List<StorageObjectOrIOException> srcAndDestObjects = getObjects(Arrays.asList(from, to));\n        String srcHash = srcAndDestObjects.get(0).storageObject().getMd5Hash();\n        String destHash = srcAndDestObjects.get(1).storageObject().getMd5Hash();\n        if (srcHash != null && srcHash.equals(destHash)) {\n          // Source and destination are identical. Treat this as a successful rewrite\n          LOG.warn(\n              \"Caught retentionPolicyNotMet error while rewriting to a bucket with retention \"\n                  + \"policy. Skipping because destination {} and source {} are considered identical \"\n                  + \"because their MD5 Hashes are equal.\",\n              getFrom(),\n              getTo());\n\n          if (deleteSource) {\n            readyToEnqueue = true;\n            performDelete = true;\n          } else {\n            readyToEnqueue = false;\n          }\n          lastError = null;\n        } else {\n          // User is attempting to write to a file that hasn't met its retention policy yet.\n          // Not a transient error so likely will not be fixed by a retry\n          throw new IOException(e.getMessage());\n        }","sourceCodeStart":1129,"sourceCodeEnd":1165,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java#L1129-L1165","documentation":"GCS rewrite fails with a retentionPolicyNotMet error when copying into a bucket with a retention policy. GcsUtilV1 treats source and destination as identical when their MD5 hashes match, logs this warning, and skips the rewrite, treating it as success. It surfaces when retrying rewrites that were already effectively completed.","triggerScenarios":"Calling GCS rewrite/copy (e.g. via GcsUtil or Rename/Match recovery paths) where the destination object already exists with identical content, but a retention policy on the destination bucket prevents overwriting.","commonSituations":"Re-running a failed/staged pipeline that partially copied files into a retention-locked bucket; rewriting a file onto itself in a bucket with retention enabled; Dataflow file-system rename retries.","solutions":["Verify srcHash equals destHash as the log indicates — the file content is already in place and no action is needed.","If the rewrite must genuinely change the object, remove or wait out the retention policy on the destination bucket.","Write to a different destination object/bucket without an active retention lock.","Avoid rewriting an object onto itself when retention policies are enabled."],"exampleFix":"// before\nstorageRewrite(from, to); // fails with retentionPolicyNotMet on retry\n// after\nif (!from.equals(to)) { // skip no-op self-rewrites\n  storageRewrite(from, to);\n}","handlingStrategy":"validation","validationCode":"// Check MD5 identity before rewriting into a retention-policy bucket:\nString srcMd5 = storage.get(from).getMd5();\nString destMd5 = storage.get(to) == null ? null : storage.get(to).getMd5();\nboolean skipRewrite = srcMd5 != null && srcMd5.equals(destMd5);","typeGuard":null,"tryCatchPattern":"try {\n  storageRewrite(from, to);\n} catch (StorageException e) {\n  if (e.getCode() == 412 && \"retentionPolicyNotMet\".equals(e.getReason())) {\n    // verify hashes match, then treat as success / skip\n  }\n}","preventionTips":["Don't rewrite objects onto themselves in retention-enabled buckets","Check bucket retention policies before configuring destinations","Make pipelines idempotent for partially-completed copy stages","Use distinct destination paths per run to avoid retention collisions"],"tags":["gcs","retention-policy","md5","retry"],"backgroundTag":"http-error-response","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}