apache/beam · error · FileNotFoundException

Rewrite from to has failed. Either source or sink not…

Error message

Rewrite from %s to %s has failed. Either source or sink not found. Failed with error: %s

What it means

During a GCS Rewrite operation, if the underlying API returns HTTP 404, the library throws this FileNotFoundException indicating that either the source object or destination bucket/object does not exist. If ignoreMissingSource is set, a missing source is treated as a successful no-op instead.

Solutions

Verify the source object exists (gcsUtil.objectExists / getObject) before rewriting
Verify the destination bucket exists and is writable
Set ignoreMissingSource=true if a vanished source should be tolerated
Re-check upstream pipeline logic for concurrent deletes

Example fix

// before
gcsUtil.copy(srcPaths, destPaths); // FileNotFoundException on missing source
// after
if (gcsUtil.objectExists(srcPath)) {
  gcsUtil.copy(srcPaths, destPaths);
} else {
  LOG.warn("Skipping missing source {}", srcPath);
}

Defensive patterns

Strategy: validation

Validate before calling

if (!gcsUtil.objectExists(srcPath)) {
  throw new FileNotFoundException("Source missing: " + srcPath);
}
if (!gcsUtil.bucketExists(GcsPath.fromUri("gs://" + destBucket))) {
  throw new FileNotFoundException("Dest bucket missing: " + destBucket);
}

Try / catch

try {
  gcsUtil.copy(src, dest);
} catch (FileNotFoundException e) {
  if (e.getMessage().startsWith("Rewrite from")) {
    LOG.warn("Rewrite source/dest missing: {}", e.getMessage());
  } else { throw e; }
}

Prevention

Set ignoreMissingSource=true for best-effort copies
Verify source objects and dest bucket before copy
Watch for concurrent deletes/moves of source objects

When it happens

Trigger: Calling copy/rewrite (e.g. GcsUtil.copy with Rewrite) where the source gs:// object was deleted, or the destination bucket does not exist, while ignoreMissingSource is false.

Common situations: Copying files that a concurrent job already deleted/moved; typos in source object names; destination bucket removed between planning and execution.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e0857243dc0eeb59. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java:1133

            from,
            to);
        rewriteRequest.setRewriteToken(rewriteResponse.getRewriteToken());
        readyToEnqueue = true;
        if (numRewriteTokensUsed != null) {
          numRewriteTokensUsed.incrementAndGet();
        }
      }
    }

    @Override
    public void onFailure(GoogleJsonError e, HttpHeaders responseHeaders) throws IOException {
      if (e.getCode() == HttpStatusCodes.STATUS_CODE_NOT_FOUND) {
        if (ignoreMissingSource) {
          // Treat a missing source as a successful rewrite.
          readyToEnqueue = false;
          lastError = null;
        } else {
          throw new FileNotFoundException(
              String.format(
                  "Rewrite from %s to %s has failed. Either source or sink not found. "
                      + "Failed with error: %s",
                  from.toString(), to.toString(), e.getMessage()));
        }
      } else if (e.getCode() == 403
          && e.getErrors().size() == 1
          && e.getErrors().get(0).getReason().equals("retentionPolicyNotMet")) {
        List<StorageObjectOrIOException> srcAndDestObjects = getObjects(Arrays.asList(from, to));
        String srcHash = srcAndDestObjects.get(0).storageObject().getMd5Hash();
        String destHash = srcAndDestObjects.get(1).storageObject().getMd5Hash();
        if (srcHash != null && srcHash.equals(destHash)) {
          // Source and destination are identical. Treat this as a successful rewrite
          LOG.warn(
              "Caught retentionPolicyNotMet error while rewriting to a bucket with retention "
                  + "policy. Skipping because destination {} and source {} are considered identical "
                  + "because their MD5 Hashes are equal.",
              getFrom(),

View on GitHub (pinned to 12126d8942)