apache/beam · error · RuntimeException

Failed to load user-defined scalar function ${functionFullNa

Error message

Failed to load user-defined scalar function ${functionFullName} from ${jarPath}

What it means

JavaUdfLoader.loadScalarFunction() wraps any IOException from loadJar(jarPath) (jar open/read/classloading) into RuntimeException 'Failed to load user-defined scalar function %s from %s'. The jar could not be read at all — not that the function is missing — so the function lookup never happened.

Source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JavaUdfLoader.java:88

  public ScalarFn loadScalarFunction(List<String> functionPath, String jarPath) {
    String functionFullName = String.join(".", functionPath);
    try {
      FunctionDefinitions functionDefinitions = loadJar(jarPath);
      if (!functionDefinitions.scalarFunctions().containsKey(functionPath)) {
        throw new IllegalArgumentException(
            String.format(
                "No implementation of scalar function %s found in %s.%n"
                    + " 1. Create a class implementing %s and annotate it with @AutoService(%s.class).%n"
                    + " 2. Add function %s to the class's userDefinedScalarFunctions implementation.",
                functionFullName,
                jarPath,
                UdfProvider.class.getSimpleName(),
                UdfProvider.class.getSimpleName(),
                functionFullName));
      }
      return functionDefinitions.scalarFunctions().get(functionPath);
    } catch (IOException e) {
      throw new RuntimeException(
          String.format(
              "Failed to load user-defined scalar function %s from %s", functionFullName, jarPath),
          e);
    }
  }

  /** Load a user-defined aggregate function from the specified jar. */
  public AggregateFn loadAggregateFunction(List<String> functionPath, String jarPath) {
    String functionFullName = String.join(".", functionPath);
    try {
      FunctionDefinitions functionDefinitions = loadJar(jarPath);
      if (!functionDefinitions.aggregateFunctions().containsKey(functionPath)) {
        throw new IllegalArgumentException(
            String.format(
                "No implementation of aggregate function %s found in %s.%n"
                    + " 1. Create a class implementing %s and annotate it with @AutoService(%s.class).%n"
                    + " 2. Add function %s to the class's userDefinedAggregateFunctions implementation.",
                functionFullName,

View on GitHub (pinned to 12126d8942)

Solutions

Verify jarPath points to an existing, readable jar on the node executing the query.
Open the jar manually (jar tf <path>) to confirm validity and the UdfProvider service file.
Check file permissions and that the jar was staged/uploaded to the runner environment.
Inspect the cause IOException for the underlying read/load error.

Example fix

// before
new JavaUdfLoader().loadScalarFunction(ImmutableList.of("myfn"), "/wrong/path/udf.jar");
// after
new JavaUdfLoader().loadScalarFunction(ImmutableList.of("myfn"), "/correct/path/udf.jar");

Defensive patterns

Strategy: try-catch

Validate before calling

File jar = new File(jarPath);
if (!jar.isFile() || !jar.canRead()) {
  throw new IllegalArgumentException("UDF jar missing or unreadable: " + jarPath);
}

Try / catch

try {
  ScalarFn fn = udfLoader.loadScalarFunction(fnPath, jarPath);
} catch (RuntimeException e) {
  if (e.getMessage().startsWith("Failed to load user-defined scalar function")) {
    LOG.error("Could not read jar {}: {}", jarPath, e.getCause().getMessage());
  }
  throw e;
}

Prevention

Ensure jars are staged to every worker node's filesystem
Validate jar integrity (jar tf) after upload
Check read permissions in containerized runners
Resolve jar paths from a single configuration source to avoid typos

When it happens

Trigger: loadScalarFunction called with a jarPath that does not exist, is unreadable, is not a valid jar, or whose classes fail to load (URLClassLoader / loadJar throws IOException).

Common situations: Wrong file path or missing file on the runner's filesystem (the path is local to the worker); permission problems; corrupted or non-jar file; classloader conflicts inside a fat jar.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9a6a5915acb7ca09. Report an issue: GitHub.