apache/beam · error · RuntimeException

The provided external bucket could not be matched to a known

Error message

The provided external bucket could not be matched to a known source.

What it means

replacePathWithLocal() rewrites an external path (e.g. gs://bucket/key) into a local worker path by locating the "://" scheme separator with lastIndexOf. If the path has no "://", it cannot be attributed to a known filesystem, and this RuntimeException is thrown early instead of producing a bogus local path.

Source

Thrown at sdks/java/extensions/kafka-factories/src/main/java/org/apache/beam/sdk/extensions/kafka/factories/FileAwareFactoryFn.java:215

  protected byte[] getSecretWithCache(String secretId) {
    return secretCache.computeIfAbsent(secretId, this::getSecret);
  }

  /**
   * A helper method to create a new string with the external paths replaced with their local path
   * and subdirectory based on the factory type in the /tmp directory. For example, the kerberos
   * factory type will replace the file paths with /tmp/kerberos/file.path
   *
   * @param externalPath
   * @return a string with all instances of external paths converted to the local paths where the
   *     files sit.
   */
  private String replacePathWithLocal(String externalPath) throws IOException {
    String externalBucketPrefixIdentifier = "://";
    int externalBucketPrefixIndex = externalPath.lastIndexOf(externalBucketPrefixIdentifier);
    if (externalBucketPrefixIndex == -1) {
      // if we don't find a known bucket prefix then we will error early.
      throw new RuntimeException(
          "The provided external bucket could not be matched to a known source.");
    }

    int prefixLength = externalBucketPrefixIndex + externalBucketPrefixIdentifier.length();
    return DIRECTORY_PREFIX + "/" + factoryType + "/" + externalPath.substring(prefixLength);
  }

  /**
   * A hook for subclasses to download and process specific files before the main configuration is
   * handled. For example, the kerberos factory can use this to download a krb5.conf and set a
   * system property.
   *
   * @throws IOException If downloading or processing the file fails.
   */
  protected void downloadAndProcessExtraFiles() throws IOException {
    // Default implementation should do nothing.
  }

View on GitHub (pinned to 12126d8942)

Solutions

1. Prefix the path with a supported scheme, e.g. "gs://bucket/path" instead of "bucket/path"
2. If the file is local, use an absolute file URI: "file:///local/path"
3. Validate all path-valued config keys contain "://" before constructing the KafkaIO read
4. Check for typos where the scheme was stripped (e.g. by templating)

Example fix

// before
config.put("ssl.keystore.location", "my-bucket/keystore.jks");
// after
config.put("ssl.keystore.location", "gs://my-bucket/keystore.jks");

java
private static void requireSchemedPath(String p) {
  if (p == null || !p.contains("://")) {
    throw new IllegalArgumentException("Path must include a scheme, e.g. gs://bucket/key: " + p);
  }
}

When it happens

Trigger: Passing a bucket-relative or local path without a scheme to a config key that replacePathWithLocal processes — e.g. "my-bucket/file.jks" or "/local/file.jks" instead of "gs://my-bucket/file.jks".

Common situations: Users copying local file paths into Kafka SSL/keystore config keys; docs examples showing bucket-relative paths; switching from local runner (where plain paths worked) to Dataflow where schemes are required.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

• Failed trying to process value for key %s.
• Unable to create the keytab file for the provided secret.
• Unknown watermark type: ${type}. Supported types are Process
• There is no Kafka read implementation that supports every co
• Unrecognized value for stable unique names:

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1058f57e13013e89. Report an issue: GitHub.