{"record":{"id":"72b204da889182f3","repo":"apache/beam","slug":"failed-to-download-file-s","errorCode":null,"errorMessage":"Failed to download file : %s","messagePattern":"Failed to download file : (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/extensions/kafka-factories/src/main/java/org/apache/beam/sdk/extensions/kafka/factories/FileAwareFactoryFn.java","lineNumber":130,"sourceCode":"          value = e.getValue();\n          if (value instanceof String) {\n            String originalValue = (String) value;\n            Matcher matcher = PATH_PATTERN.matcher(originalValue);\n            StringBuffer sb = new StringBuffer();\n\n            while (matcher.find()) {\n              String externalPath = matcher.group(1);\n              String secretValue = matcher.group(2);\n              String secretFile = matcher.group(3);\n\n              if (externalPath != null) {\n                try {\n                  String tmpPath = replacePathWithLocal(externalPath);\n                  String localPath = downloadExternalFile(externalPath, tmpPath);\n                  matcher.appendReplacement(sb, Matcher.quoteReplacement(localPath));\n                  LOG.info(\"Downloaded {} to {}\", externalPath, localPath);\n                } catch (IOException io) {\n                  throw new IOException(\"Failed to download file : \" + externalPath, io);\n                }\n              } else if (secretValue != null) {\n                try {\n                  String secretId = secretValue.substring(SECRET_VALUE_PREFIX.length());\n                  String processedSecret =\n                      processSecret(originalValue, secretId, getSecretWithCache(secretId));\n\n                  matcher.appendReplacement(sb, Matcher.quoteReplacement(processedSecret));\n                } catch (IllegalArgumentException ia) {\n                  throw new IllegalArgumentException(\"Failed to get secret.\", ia);\n                }\n              } else if (secretFile != null) {\n                throw new UnsupportedOperationException(\"Not yet implemented.\");\n              }\n            }\n            matcher.appendTail(sb);\n            String processedValue = sb.toString();\n            processedConfig.put(key, processedValue);","sourceCodeStart":112,"sourceCodeEnd":148,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/kafka-factories/src/main/java/org/apache/beam/sdk/extensions/kafka/factories/FileAwareFactoryFn.java#L112-L148","documentation":"FileAwareFactoryFn rewrites Kafka configuration values that reference external files (MRS://{...} style placeholders) by downloading them to a local temp path and substituting the local path. If the download fails with IOException, it is rethrown as 'Failed to download file : <externalPath>'.","triggerScenarios":"A config value contains an external-file placeholder whose URL is unreachable, credentials are missing, the bucket/path does not exist, or the local temp directory is not writable during apply().","commonSituations":"Kafka connectors configured with remote keystore/truststore files (e.g. in GCS) where the path was mistyped, permissions/credentials are absent, or the worker cannot reach the storage endpoint (VPC/no internet).","solutions":["Verify the external path is correct and the object exists and is readable by the job's credentials","Pre-download the file and reference a local path instead of the remote placeholder","Check network access from workers (firewall/VPC) and that the temp directory is writable"],"exampleFix":"// before\n\"ssl.keystore.location\": \"mrs://mybucket/keystore.jks\" // download fails on workers\n// after\n\"ssl.keystore.location\": \"/opt/certs/keystore.jks\" // pre-staged on workers","handlingStrategy":"retry","validationCode":"boolean reachable = false; try (InputStream in = new URL(stripScheme(externalPath)).openStream()) { reachable = true; } catch (IOException e) { LOG.error(\"External file unreachable: {}\", externalPath); }","typeGuard":null,"tryCatchPattern":"try { apply(configValue); } catch (IOException e) { if (e.getMessage().startsWith(\"Failed to download file\")) { /* check path/credentials/network, retry with backoff */ } else throw e; }","preventionTips":["Verify external file paths and read permissions before launching the job","Pre-stage files locally on workers to avoid runtime downloads","Ensure workers have network egress to the storage endpoint"],"tags":["kafka","io","download","network"],"backgroundTag":"file-download-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}