{"record":{"id":"1058f57e13013e89","repo":"apache/beam","slug":"the-provided-external-bucket-could-not-be-matched-to-a-known","errorCode":null,"errorMessage":"The provided external bucket could not be matched to a known source.","messagePattern":"The provided external bucket could not be matched to a known source\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/extensions/kafka-factories/src/main/java/org/apache/beam/sdk/extensions/kafka/factories/FileAwareFactoryFn.java","lineNumber":215,"sourceCode":"  protected byte[] getSecretWithCache(String secretId) {\n    return secretCache.computeIfAbsent(secretId, this::getSecret);\n  }\n\n  /**\n   * A helper method to create a new string with the external paths replaced with their local path\n   * and subdirectory based on the factory type in the /tmp directory. For example, the kerberos\n   * factory type will replace the file paths with /tmp/kerberos/file.path\n   *\n   * @param externalPath\n   * @return a string with all instances of external paths converted to the local paths where the\n   *     files sit.\n   */\n  private String replacePathWithLocal(String externalPath) throws IOException {\n    String externalBucketPrefixIdentifier = \"://\";\n    int externalBucketPrefixIndex = externalPath.lastIndexOf(externalBucketPrefixIdentifier);\n    if (externalBucketPrefixIndex == -1) {\n      // if we don't find a known bucket prefix then we will error early.\n      throw new RuntimeException(\n          \"The provided external bucket could not be matched to a known source.\");\n    }\n\n    int prefixLength = externalBucketPrefixIndex + externalBucketPrefixIdentifier.length();\n    return DIRECTORY_PREFIX + \"/\" + factoryType + \"/\" + externalPath.substring(prefixLength);\n  }\n\n  /**\n   * A hook for subclasses to download and process specific files before the main configuration is\n   * handled. For example, the kerberos factory can use this to download a krb5.conf and set a\n   * system property.\n   *\n   * @throws IOException If downloading or processing the file fails.\n   */\n  protected void downloadAndProcessExtraFiles() throws IOException {\n    // Default implementation should do nothing.\n  }\n","sourceCodeStart":197,"sourceCodeEnd":233,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/kafka-factories/src/main/java/org/apache/beam/sdk/extensions/kafka/factories/FileAwareFactoryFn.java#L197-L233","documentation":"replacePathWithLocal() rewrites an external path (e.g. gs://bucket/key) into a local worker path by locating the \"://\" scheme separator with lastIndexOf. If the path has no \"://\", it cannot be attributed to a known filesystem, and this RuntimeException is thrown early instead of producing a bogus local path.","triggerScenarios":"Passing a bucket-relative or local path without a scheme to a config key that replacePathWithLocal processes — e.g. \"my-bucket/file.jks\" or \"/local/file.jks\" instead of \"gs://my-bucket/file.jks\".","commonSituations":"Users copying local file paths into Kafka SSL/keystore config keys; docs examples showing bucket-relative paths; switching from local runner (where plain paths worked) to Dataflow where schemes are required.","solutions":["Prefix the path with a supported scheme, e.g. \"gs://bucket/path\" instead of \"bucket/path\"","If the file is local, use an absolute file URI: \"file:///local/path\"","Validate all path-valued config keys contain \"://\" before constructing the KafkaIO read","Check for typos where the scheme was stripped (e.g. by templating)"],"exampleFix":"// before\nconfig.put(\"ssl.keystore.location\", \"my-bucket/keystore.jks\");\n// after\nconfig.put(\"ssl.keystore.location\", \"gs://my-bucket/keystore.jks\");","handlingStrategy":"validation","validationCode":"java\nprivate static void requireSchemedPath(String p) {\n  if (p == null || !p.contains(\"://\")) {\n    throw new IllegalArgumentException(\"Path must include a scheme, e.g. gs://bucket/key: \" + p);\n  }\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always use fully qualified URIs (gs://, file:///) in file-path config keys","Add a config lint step rejecting scheme-less paths","Beware templating that strips scheme prefixes","Test config processing locally before production launch"],"tags":["java","kafka","path","configuration"],"backgroundTag":"invalid-url-format","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}