What causes "Failed to load user-provided jar(s)." in apache/beam?

Verify each jar path is correct and reachable from the worker (file exists locally or bucket/URL is accessible)

apache/beam · error · RuntimeException

Failed to load user-provided jar(s).

Error message

Failed to load user-provided jar(s).

What it means

Beam SQL loads user-provided UDF jars into a custom class loader before compiling the generated CalcFn Java code. If reading any of the jar paths throws IOException, the setup fails with this RuntimeException wrapping the cause. It means the SQL pipeline cannot access the jar files given in the UDF clause (e.g. via UDFProvider or 'jar_path' options).

Source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java:308

      this.outputSchema = outputSchema;
      this.verifyRowValues = verifyRowValues;
      this.jarPaths = jarPaths;
      this.fieldAccess = fieldAccess;
      this.collectErrors = collectErrors;

      // validate generated code
      compile(processElementBlock, jarPaths);
    }

    private static ScriptEvaluator compile(String processElementBlock, List<String> jarPaths) {
      ScriptEvaluator se = new ScriptEvaluator();
      if (!jarPaths.isEmpty()) {
        try {
          JavaUdfLoader udfLoader = new JavaUdfLoader();
          ClassLoader classLoader = udfLoader.createClassLoader(jarPaths);
          se.setParentClassLoader(classLoader);
        } catch (IOException e) {
          throw new RuntimeException("Failed to load user-provided jar(s).", e);
        }
      }
      se.setParameters(
          new String[] {rowParam.name, DataContext.ROOT.name},
          new Class[] {(Class) rowParam.getType(), (Class) DataContext.ROOT.getType()});
      se.setReturnType(Object[].class);
      try {
        se.cook(processElementBlock);
      } catch (CompileException e) {
        throw new UnsupportedOperationException(
            "Could not compile CalcFn: " + processElementBlock, e);
      }
      return se;
    }

    @Setup
    public void setup() {
      this.se = compile(processElementBlock, jarPaths);

View on GitHub (pinned to 12126d8942)

Solutions

Verify each jar path is correct and reachable from the worker (file exists locally or bucket/URL is accessible)
Check the wrapped IOException cause for the actual failing path or connection problem
If using remote storage, confirm worker credentials/permissions (e.g. GCS/ADLS access for the Dataflow worker)
Pre-download or stage jars with the pipeline (--filesToStage or dependency staging) instead of fetching at runtime

Example fix

// before
CREATE FUNCTION MYFN AS 'com.x.MyFn' USING JAR 'gs://wrong-bucket/udf.jar';
// after
CREATE FUNCTION MYFN AS 'com.x.MyFn' USING JAR 'gs://my-correct-bucket/udf.jar';

Defensive patterns

Strategy: validation

Validate before calling

for (String path : jarPaths) {
  if (!(new java.io.File(path).exists()) && !path.startsWith("gs://") && !path.startsWith("http")) {
    throw new IllegalArgumentException("Jar not found: " + path);
  }
}

Try / catch

try {
  ClassLoader cl = udfLoader.createClassLoader(jarPaths);
} catch (IOException e) {
  throw new IllegalStateException("Check jar paths/credentials: " + e.getMessage(), e);
}

Prevention

Validate jar paths at pipeline construction time
Stage UDF jars with the pipeline instead of remote fetch at Setup time
Verify worker service-account permissions for remote jar storage
Log the resolved jar paths before loading

When it happens

Trigger: Calling JavaUdfLoader.createClassLoader(jarPaths) during CalcFn.setup or compile with a jar path that is missing, unreadable, a bad URL, or a corrupt/nonexistent artifact (e.g. wrong gs:// or http path in USING/jar options).

Common situations: Typos in the jar path in CREATE FUNCTION ... USING JAR 'path'; remote jar not accessible (missing credentials, wrong bucket, network failure); jar deleted between pipeline submission and worker startup; malformed maven coordinate.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c8527434db371698. Report an issue: GitHub.