apache/beam · error · IOException

%s

Error message

%s

What it means

During a GCS Rewrite, certain non-transient GoogleStorageExceptions indicate the destination object is under an unmet retention policy (or similar bucket policy condition). The library rethrows the raw exception message as an IOException because retrying will not fix it — the message itself is the error payload.

Solutions

1. Wait until the destination object's retention period expires before rewriting
2. Write to a different object name/bucket without an active retention hold
3. Remove or shorten the retention policy if policy allows (requires retention policy unlock permissions)
4. Inspect e.getMessage() (the original GoogleStorageException) for the exact policy violation

Example fix

// before
gcsUtil.copy(src, dest); // IOException: retention policy message
// after
try {
  gcsUtil.copy(src, dest);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("retention")) {
    throw new IllegalStateException("Destination under retention policy: " + e.getMessage(), e);
  }
  throw e;
}

boolean isRetentionBlock(IOException e) {
  return e.getMessage() != null && e.getMessage().toLowerCase().contains("retention");
}

try {
  gcsUtil.copy(src, dest);
} catch (IOException e) {
  if (isRetentionBlock(e)) {
    // not transient; do not retry — pick new destination or wait out retention
    throw new IllegalStateException("Destination under retention policy", e);
  }
  throw e;
}

When it happens

Trigger: copy/rewrite to a bucket whose objects have a retention policy that hasn't expired; attempt to overwrite/modify a retention-locked object before its retention period ends.

Common situations: Writing into buckets with retention/bucket-lock enabled (e.g. compliance buckets); copying data into a WORM-style bucket before retention allows mutation.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• Caught retentionPolicyNotMet error while rewriting to a…
• Invalid state for Rewrite, from=
• Rewrite from to has failed. Either source or sink not…
• source blob not found during copying
• Target object already exists and strategy is FAIL_IF_EXISTS

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/bbe23e88d1918be6. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java:1164

          // Source and destination are identical. Treat this as a successful rewrite
          LOG.warn(
              "Caught retentionPolicyNotMet error while rewriting to a bucket with retention "
                  + "policy. Skipping because destination {} and source {} are considered identical "
                  + "because their MD5 Hashes are equal.",
              getFrom(),
              getTo());

          if (deleteSource) {
            readyToEnqueue = true;
            performDelete = true;
          } else {
            readyToEnqueue = false;
          }
          lastError = null;
        } else {
          // User is attempting to write to a file that hasn't met its retention policy yet.
          // Not a transient error so likely will not be fixed by a retry
          throw new IOException(e.getMessage());
        }
      } else {
        lastError = e;
        readyToEnqueue = true;
      }
    }
  }

  public void copy(Iterable<String> srcFilenames, Iterable<String> destFilenames)
      throws IOException {
    rewriteHelper(
        srcFilenames,
        destFilenames,
        /* deleteSource= */ false,
        /* ignoreMissingSource= */ false,
        /* ignoreExistingDest= */ false);
  }

View on GitHub (pinned to 12126d8942)