apache/beam · error · RuntimeException

Output path does not exist or is not writeable: %s

Error message

Output path does not exist or is not writeable: %s

What it means

GcsPathValidator.verifyPathIsAccessible checks that a GCS path's bucket exists and is accessible via GcsUtil.verifyBucketAccessible. When that check throws an IOException (bucket missing, no permission, or path not writeable), the validator wraps it in a RuntimeException using the supplied message template, e.g. 'Output path does not exist or is not writeable: %s'. It is thrown before a job runs, during output file prefix validation.

Source

Thrown at sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsPathValidator.java:85

    GcsPath gcsPath = getGcsPath(path);
    checkArgument(gcsPath.isAbsolute(), "Must provide absolute paths for Dataflow");
    checkArgument(
        !gcsPath.getObject().isEmpty(),
        "Missing object or bucket in path: '%s', did you mean: 'gs://some-bucket/%s'?",
        gcsPath,
        gcsPath.getBucket());
    checkArgument(
        !gcsPath.getObject().contains("//"),
        "Dataflow Service does not allow objects with consecutive slashes");
    return gcsPath.toResourceName();
  }

  private void verifyPathIsAccessible(String path, String errorMessage) {
    GcsPath gcsPath = getGcsPath(path);
    try {
      gcpOptions.getGcsUtil().verifyBucketAccessible(gcsPath);
    } catch (IOException e) {
      throw new RuntimeException(String.format(errorMessage, path), e);
    }
  }

  private GcsPath getGcsPath(String path) {
    try {
      return GcsPath.fromUri(path);
    } catch (IllegalArgumentException e) {
      throw new IllegalArgumentException(
          String.format("Expected a valid 'gs://' path but was given '%s'", path), e);
    }
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

Verify the bucket exists: gsutil ls gs://<bucket>/
Confirm credentials/project: the pipeline options' project must match the bucket's project or have access
Grant the running service account roles/storage.admin (or at least storage.buckets.get) on the bucket
Check the output prefix path format: it must be a valid gs:// URI, not a local path

Example fix

// before
.apply("Write", TextIO.write().to("my-bucket/output/data"));
// after
.apply("Write", TextIO.write().to("gs://my-bucket/output/data"));

Defensive patterns

Strategy: validation

Validate before calling

String path = options.getOutputFilePrefix();
if (!path.startsWith("gs://")) throw new IllegalArgumentException("output must be a gs:// path: " + path);
String bucket = GcsPath.fromUri(path).getBucket();
// pre-check via gsutil or Storage client: bucket exists and credentials can access it

Try / catch

try { validator.validateOutputFilePrefixSupported(prefix); }
catch (RuntimeException e) { LOG.error("Output prefix rejected: " + e.getMessage(), e.getCause()); throw e; }

Prevention

Always use fully-qualified gs://bucket/prefix output paths
Run gsutil ls on the bucket before submitting the job
Grant the pipeline service account storage object/bucket access in the target project

When it happens

Trigger: Calling FileBasedSink/TextIO validate or pipeline validation with an output prefix whose bucket does not exist, whose bucket is in another project without access, or when GCS API calls fail (credentials, network).

Common situations: Typo in the gs:// output prefix or bucket name; bucket deleted between job submission; service account lacking storage.buckets.get; wrong project credentials on the Dataflow runner.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5aff38899da373b4. Report an issue: GitHub.