{"record":{"id":"e0857243dc0eeb59","repo":"apache/beam","slug":"rewrite-from-s-to-s-has-failed-either-source-or-sink-not","errorCode":null,"errorMessage":"Rewrite from %s to %s has failed. Either source or sink not found. Failed with error: %s","messagePattern":"Rewrite from (.+?) to (.+?) has failed\\. Either source or sink not found\\. Failed with error: (.+?)","errorType":"exception","errorClass":"FileNotFoundException","httpStatus":404,"severity":"error","filePath":"sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java","lineNumber":1133,"sourceCode":"            from,\n            to);\n        rewriteRequest.setRewriteToken(rewriteResponse.getRewriteToken());\n        readyToEnqueue = true;\n        if (numRewriteTokensUsed != null) {\n          numRewriteTokensUsed.incrementAndGet();\n        }\n      }\n    }\n\n    @Override\n    public void onFailure(GoogleJsonError e, HttpHeaders responseHeaders) throws IOException {\n      if (e.getCode() == HttpStatusCodes.STATUS_CODE_NOT_FOUND) {\n        if (ignoreMissingSource) {\n          // Treat a missing source as a successful rewrite.\n          readyToEnqueue = false;\n          lastError = null;\n        } else {\n          throw new FileNotFoundException(\n              String.format(\n                  \"Rewrite from %s to %s has failed. Either source or sink not found. \"\n                      + \"Failed with error: %s\",\n                  from.toString(), to.toString(), e.getMessage()));\n        }\n      } else if (e.getCode() == 403\n          && e.getErrors().size() == 1\n          && e.getErrors().get(0).getReason().equals(\"retentionPolicyNotMet\")) {\n        List<StorageObjectOrIOException> srcAndDestObjects = getObjects(Arrays.asList(from, to));\n        String srcHash = srcAndDestObjects.get(0).storageObject().getMd5Hash();\n        String destHash = srcAndDestObjects.get(1).storageObject().getMd5Hash();\n        if (srcHash != null && srcHash.equals(destHash)) {\n          // Source and destination are identical. Treat this as a successful rewrite\n          LOG.warn(\n              \"Caught retentionPolicyNotMet error while rewriting to a bucket with retention \"\n                  + \"policy. Skipping because destination {} and source {} are considered identical \"\n                  + \"because their MD5 Hashes are equal.\",\n              getFrom(),","sourceCodeStart":1115,"sourceCodeEnd":1151,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV1.java#L1115-L1151","documentation":"During a GCS Rewrite operation, if the underlying API returns HTTP 404, the library throws this FileNotFoundException indicating that either the source object or destination bucket/object does not exist. If ignoreMissingSource is set, a missing source is treated as a successful no-op instead.","triggerScenarios":"Calling copy/rewrite (e.g. GcsUtil.copy with Rewrite) where the source gs:// object was deleted, or the destination bucket does not exist, while ignoreMissingSource is false.","commonSituations":"Copying files that a concurrent job already deleted/moved; typos in source object names; destination bucket removed between planning and execution.","solutions":["Verify the source object exists (gcsUtil.objectExists / getObject) before rewriting","Verify the destination bucket exists and is writable","Set ignoreMissingSource=true if a vanished source should be tolerated","Re-check upstream pipeline logic for concurrent deletes"],"exampleFix":"// before\ngcsUtil.copy(srcPaths, destPaths); // FileNotFoundException on missing source\n// after\nif (gcsUtil.objectExists(srcPath)) {\n  gcsUtil.copy(srcPaths, destPaths);\n} else {\n  LOG.warn(\"Skipping missing source {}\", srcPath);\n}","handlingStrategy":"validation","validationCode":"if (!gcsUtil.objectExists(srcPath)) {\n  throw new FileNotFoundException(\"Source missing: \" + srcPath);\n}\nif (!gcsUtil.bucketExists(GcsPath.fromUri(\"gs://\" + destBucket))) {\n  throw new FileNotFoundException(\"Dest bucket missing: \" + destBucket);\n}","typeGuard":null,"tryCatchPattern":"try {\n  gcsUtil.copy(src, dest);\n} catch (FileNotFoundException e) {\n  if (e.getMessage().startsWith(\"Rewrite from\")) {\n    LOG.warn(\"Rewrite source/dest missing: {}\", e.getMessage());\n  } else { throw e; }\n}","preventionTips":["Set ignoreMissingSource=true for best-effort copies","Verify source objects and dest bucket before copy","Watch for concurrent deletes/moves of source objects"],"tags":["gcs","copy","not-found"],"backgroundTag":"file-not-found","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}