apache/beam · error · IllegalArgumentException

Found multiple SchemaTransformProvider implementations with…

Error message

Found multiple SchemaTransformProvider implementations with the same identifier {schemaTransformProvider.identifier()}

What it means

ManagedSchemaTransformProvider.getAllProviders discovers SchemaTransformProvider implementations via ServiceLoader and caches them by identifier(). If two providers on the classpath declare the same identifier, discovery cannot disambiguate and this IllegalArgumentException is thrown. It protects against silently picking the wrong implementation.

Solutions

Run `mvn dependency:tree` / gradle dependencies to find duplicate Beam IO or provider jars and exclude one.
If shading, merge/relocate service files so only one provider per identifier is registered.
Rename your custom provider's identifier() so it is unique.
Restrict the supportedIdentifiers passed to Managed so the conflicting provider is not required, after removing the duplicate.

Example fix

// before: custom provider reuses existing id
public String identifier() { return "beam:schematransform:sql_server:v1"; }
// after
public String identifier() { return "myorg:schematransform:custom_sql_server:v1"; }

Defensive patterns

Strategy: validation

Validate before calling

Set<String> seen = new HashSet<>();
for (SchemaTransformProvider p : ServiceLoader.load(SchemaTransformProvider.class)) {
  if (!seen.add(p.identifier())) throw new IllegalStateException("Duplicate provider id: " + p.identifier());
}

Try / catch

try { pipeline.apply(Managed.read(Managed.BIGQUERY, config)); } catch (IllegalArgumentException e) { if (e.getMessage().contains("same identifier")) { /* fix classpath duplicates */ } throw e; }

Prevention

Audit fat jars for duplicated META-INF/services entries.
Never reuse built-in beam: identifiers in custom providers.
Run discovery (Managed.read on a dummy id) in CI to catch classpath issues early.

When it happens

Trigger: Building/running a pipeline where the classpath (fat jar, shaded uber-jar, or project dependencies) contains two jars registering SchemaTransformProvider services with the same identifier, triggering provider discovery.

Common situations: Shading Beam IO modules into an uber-jar that duplicates service files, depending on two Beam versions transitively, or adding a custom provider that reuses an existing identifier like 'beam:schematransform:...'.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/34468b7752b64c82. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:255

    Map<String, Map<String, String>> aliases = new HashMap<>();
    for (Map.Entry<String, Object> entry : YamlUtils.loadFromStream(inputStream).entrySet()) {
      aliases.put(entry.getKey(), (Map<String, String>) entry.getValue());
    }
    return aliases;
  }

  // We load providers separately, after construction, to prevent the
  // 'ManagedSchemaTransformProvider' from being initialized in a recursive loop
  // when being loaded using 'AutoValue'.
  synchronized Map<String, SchemaTransformProvider> getAllProviders() {
    if (this.providersCached) {
      return schemaTransformProvidersCache;
    }
    try {
      for (SchemaTransformProvider schemaTransformProvider :
          ServiceLoader.load(SchemaTransformProvider.class)) {
        if (schemaTransformProvidersCache.containsKey(schemaTransformProvider.identifier())) {
          throw new IllegalArgumentException(
              "Found multiple SchemaTransformProvider implementations with the same identifier "
                  + schemaTransformProvider.identifier());
        }
        if (supportedIdentifiers == null
            || supportedIdentifiers.contains(schemaTransformProvider.identifier())) {
          if (schemaTransformProvider.identifier().equals("beam:transform:managed:v1")) {
            // Prevent recursively adding the 'ManagedSchemaTransformProvider'.
            continue;
          }
          schemaTransformProvidersCache.put(
              schemaTransformProvider.identifier(), schemaTransformProvider);
        }
      }
      this.providersCached = true;
      return schemaTransformProvidersCache;
    } catch (Exception e) {
      throw new RuntimeException(e.getMessage());
    }

View on GitHub (pinned to 12126d8942)