{"record":{"id":"f74aeff7936a99dd","repo":"apache/beam","slug":"failed-trying-to-process-extra-files","errorCode":null,"errorMessage":"Failed trying to process extra files.","messagePattern":"Failed trying to process extra files\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/extensions/kafka-factories/src/main/java/org/apache/beam/sdk/extensions/kafka/factories/FileAwareFactoryFn.java","lineNumber":155,"sourceCode":"\n                  matcher.appendReplacement(sb, Matcher.quoteReplacement(processedSecret));\n                } catch (IllegalArgumentException ia) {\n                  throw new IllegalArgumentException(\"Failed to get secret.\", ia);\n                }\n              } else if (secretFile != null) {\n                throw new UnsupportedOperationException(\"Not yet implemented.\");\n              }\n            }\n            matcher.appendTail(sb);\n            String processedValue = sb.toString();\n            processedConfig.put(key, processedValue);\n          }\n        } catch (IOException ex) {\n          throw new RuntimeException(\"Failed trying to process value for key \" + key + \".\", ex);\n        }\n      }\n    } catch (IOException e) {\n      throw new RuntimeException(\"Failed trying to process extra files.\", e);\n    }\n\n    return createObject(processedConfig);\n  }\n\n  /**\n   * A function to download files from their specified external storage path and copy them to the\n   * provided local filepath. The local filepath is provided by the replacePathWithLocal.\n   *\n   * @param externalFilePath\n   * @param outputFileString\n   * @return\n   * @throws IOException\n   */\n  protected static synchronized String downloadExternalFile(\n      String externalFilePath, String outputFileString) throws IOException {\n    // create the file only if it doesn't exist\n    if (new File(outputFileString).exists()) {","sourceCodeStart":137,"sourceCodeEnd":173,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/kafka-factories/src/main/java/org/apache/beam/sdk/extensions/kafka/factories/FileAwareFactoryFn.java#L137-L173","documentation":"The outermost IOException catch in FileAwareFactoryFn.apply(). It wraps any IOException raised while iterating/processing the extra files directory itself (as opposed to a single key's value). The original IOException is chained as the cause.","triggerScenarios":"IOException while opening or walking the extra-files directory that backs the factory config — e.g. the staged directory can't be opened or a contained file's stream fails outside the per-key handler.","commonSituations":"Worker image missing the staged extras directory; corrupted staged bundle; filesystem transient failure; misconfigured directory prefix for the factory type.","solutions":["Inspect the chained cause for the concrete path that failed","Confirm extra files were actually staged and unpacked on the worker (check --filesToStage / worker logs)","Verify DIRECTORY_PREFIX-derived staging directory exists and is readable for the factory type","Retry the pipeline; transient distributed-filesystem failures often resolve on rerun"],"exampleFix":"// before\n--filesToStage=old-extras/\n// after\n--filesToStage=/correct/path/extra-files/  # verified to contain the referenced files","handlingStrategy":"try-catch","validationCode":"java\n// verify extras directory exists and is readable on workers before pipeline start\nFileSystems.match(\"gs://my-bucket/extra-files/*\");","typeGuard":null,"tryCatchPattern":"java\ntry {\n  factoryFn.apply(config);\n} catch (RuntimeException ex) {\n  if (\"Failed trying to process extra files.\".equals(ex.getMessage())) {\n    log.severe(\"Extra files processing failed: \" + ex.getCause());\n  }\n}","preventionTips":["Confirm staged extras appear in worker logs at pipeline startup","Use --filesToStage consistently across runner configs","Treat transient distributed-FS IOExceptions with a pipeline retry","Pin the directory layout expected by DIRECTORY_PREFIX/factoryType"],"tags":["java","io","kafka","staging"],"backgroundTag":"file-read-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}