apache/beam · error · java.lang.RuntimeException

Failed to get dependencies for spec %s

Error message

Failed to get dependencies for spec %s

What it means

ExpansionService.getDependencies() resolves a transform's artifact dependencies from its config; any exception during payload parsing or dependency resolution is wrapped in this RuntimeException. The expansion service needs these dependencies to stage files for cross-language execution.

Source

Thrown at sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java:353

                ExternalConfigurationPayload.parseFrom(spec.getPayload()), configClass));
      } catch (Exception e) {
        throw new RuntimeException(
            String.format("Failed to build transform from spec %s: %s", spec, e.getMessage()), e);
      }
    }

    @Override
    public List<String> getDependencies(RunnerApi.FunctionSpec spec, PipelineOptions options) {
      try {
        Class configClass = getConfigClass(transformBuilder);
        Optional<List<String>> dependencies =
            transformBuilder.getDependencies(
                payloadToConfig(
                    ExternalConfigurationPayload.parseFrom(spec.getPayload()), configClass),
                options);
        return dependencies.orElseGet(() -> TransformProvider.super.getDependencies(spec, options));
      } catch (Exception e) {
        throw new RuntimeException(
            String.format("Failed to get dependencies for spec %s", spec), e);
      }
    }

    @Override
    public boolean equals(@Nullable Object other) {
      return other instanceof TransformProviderForBuilder
          && Objects.equals(
              transformBuilder, ((TransformProviderForBuilder) other).transformBuilder);
    }

    @Override
    public int hashCode() {
      return transformBuilder.hashCode();
    }
  }

  private static <ConfigT> Class<ConfigT> getConfigClass(

View on GitHub (pinned to 12126d8942)

Solutions

  1. Read the 'Caused by' to find whether payload parsing or getDependencies failed
  2. Validate that any files/URLs referenced in the config exist and are reachable from the service
  3. Ensure the ExternalConfigurationPayload schema matches the config class
  4. Keep Beam versions aligned between the pipeline submitter and expansion service

Example fix

// before
WriteToFiles.withFileLocation('/nonexistent/dir')
// after: ensure the path exists and is accessible
WriteToFiles.withFileLocation('/data/output')
Defensive patterns

Strategy: try-catch

Validate before calling

// verify referenced dependency locations exist before expansion
for (String path : configDependencyPaths) { if (!Files.exists(Paths.get(path))) throw new FileNotFoundException(path); }

Try / catch

try { deps = service.getDependencies(spec, options); } catch (RuntimeException e) { if (e.getMessage().startsWith("Failed to get dependencies")) { /* check payload and referenced resources */ } throw e; }

Prevention

When it happens

Trigger: Calling getDependencies with a spec whose ExternalConfigurationPayload cannot be parsed, whose config class cannot be constructed, or whose transformBuilder.getDependencies() throws; also fires when the builder path fails before falling back to the default dependency list.

Common situations: Connector builders requiring files/URLs that do not exist to compute dependencies; malformed payload after client upgrade; config class schema drift between the sending SDK and the expansion service.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/eddb7f962ab3d521. Report an issue: GitHub.