Caught retentionPolicyNotMet error while rewriting to a…

Error message

Caught retentionPolicyNotMet error while rewriting to a bucket with retention policy. Skipping because destination {} and source {} are considered identical because their MD5 Hashes are equal.

What it means

GCS rewrite fails with a retentionPolicyNotMet error when copying into a bucket with a retention policy. GcsUtilV1 treats source and destination as identical when their MD5 hashes match, logs this warning, and skips the rewrite, treating it as success. It surfaces when retrying rewrites that were already effectively completed.

Solutions

Verify srcHash equals destHash as the log indicates — the file content is already in place and no action is needed.
If the rewrite must genuinely change the object, remove or wait out the retention policy on the destination bucket.
Write to a different destination object/bucket without an active retention lock.
Avoid rewriting an object onto itself when retention policies are enabled.

Example fix

// before
storageRewrite(from, to); // fails with retentionPolicyNotMet on retry
// after
if (!from.equals(to)) { // skip no-op self-rewrites
  storageRewrite(from, to);
}

Defensive patterns

Strategy: validation

Validate before calling

// Check MD5 identity before rewriting into a retention-policy bucket:
String srcMd5 = storage.get(from).getMd5();
String destMd5 = storage.get(to) == null ? null : storage.get(to).getMd5();
boolean skipRewrite = srcMd5 != null && srcMd5.equals(destMd5);

Try / catch

try {
  storageRewrite(from, to);
} catch (StorageException e) {
  if (e.getCode() == 412 && "retentionPolicyNotMet".equals(e.getReason())) {
    // verify hashes match, then treat as success / skip
  }
}

Prevention

Don't rewrite objects onto themselves in retention-enabled buckets
Check bucket retention policies before configuring destinations
Make pipelines idempotent for partially-completed copy stages
Use distinct destination paths per run to avoid retention collisions

When it happens

Trigger: Calling GCS rewrite/copy (e.g. via GcsUtil or Rename/Match recovery paths) where the destination object already exists with identical content, but a retention policy on the destination bucket prevents overwriting.

Common situations: Re-running a failed/staged pipeline that partially copied files into a retention-locked bucket; rewriting a file onto itself in a bucket with retention enabled; Dataflow file-system rename retries.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4b2314b31ba7511f. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java:1147

          // Treat a missing source as a successful rewrite.
          readyToEnqueue = false;
          lastError = null;
        } else {
          throw new FileNotFoundException(
              String.format(
                  "Rewrite from %s to %s has failed. Either source or sink not found. "
                      + "Failed with error: %s",
                  from.toString(), to.toString(), e.getMessage()));
        }
      } else if (e.getCode() == 403
          && e.getErrors().size() == 1
          && e.getErrors().get(0).getReason().equals("retentionPolicyNotMet")) {
        List<StorageObjectOrIOException> srcAndDestObjects = getObjects(Arrays.asList(from, to));
        String srcHash = srcAndDestObjects.get(0).storageObject().getMd5Hash();
        String destHash = srcAndDestObjects.get(1).storageObject().getMd5Hash();
        if (srcHash != null && srcHash.equals(destHash)) {
          // Source and destination are identical. Treat this as a successful rewrite
          LOG.warn(
              "Caught retentionPolicyNotMet error while rewriting to a bucket with retention "
                  + "policy. Skipping because destination {} and source {} are considered identical "
                  + "because their MD5 Hashes are equal.",
              getFrom(),
              getTo());

          if (deleteSource) {
            readyToEnqueue = true;
            performDelete = true;
          } else {
            readyToEnqueue = false;
          }
          lastError = null;
        } else {
          // User is attempting to write to a file that hasn't met its retention policy yet.
          // Not a transient error so likely will not be fixed by a retry
          throw new IOException(e.getMessage());
        }

View on GitHub (pinned to 12126d8942)