apache/beam · error · IOException

Failed to access private field '%s' of AWS credential provid

Error message

Failed to access private field '%s' of AWS credential provider type '%s' with reflection

What it means

AwsModule.readField uses commons-lang FieldUtils.readField(provider, fieldName, true) to read private fields of AWS credential providers during (de)serialization. If reflection fails with IllegalArgumentException or IllegalAccessException, it wraps the cause in this IOException. It indicates the AWS SDK's provider class changed shape (field renamed/removed) or access was denied, so Beam cannot extract the needed attribute.

Source

Thrown at sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsModule.java:308

        jsonGenerator.writeStringField(ROLE_ARN, provider.assumedRoleArn());
        jsonGenerator.writeStringField(WEBID_TOKEN_FQCN, provider.webIdTokenProviderFQCN());
        Integer sessionDurationSecs = provider.sessionDurationSecs();
        if (sessionDurationSecs != null) {
          jsonGenerator.writeNumberField(SESSION_DURATION_SECONDS, sessionDurationSecs);
        }
      } else if (!SINGLETON_CREDENTIAL_PROVIDERS.contains(providerClass)) {
        throw new IllegalArgumentException(
            "Unsupported AWS credentials provider type " + providerClass);
      }
      // BEAM-11958 Use deprecated Jackson APIs to be compatible with older versions of jackson
      typeSerializer.writeTypeSuffixForObject(credentialsProvider, jsonGenerator);
    }

    private Object readField(AwsCredentialsProvider provider, String fieldName) throws IOException {
      try {
        return FieldUtils.readField(provider, fieldName, true);
      } catch (IllegalArgumentException | IllegalAccessException e) {
        throw new IOException(
            String.format(
                "Failed to access private field '%s' of AWS credential provider type '%s' with reflection",
                fieldName, provider.getClass().getSimpleName()),
            e);
      }
    }
  }

  /** A mixin to add Jackson annotations to {@link ProxyConfiguration}. */
  @JsonDeserialize(builder = ProxyConfiguration.Builder.class)
  @JsonAutoDetect(fieldVisibility = JsonAutoDetect.Visibility.ANY)
  @JsonIgnoreProperties(value = {"host", "port", "scheme"})
  @JsonInclude(value = JsonInclude.Include.NON_EMPTY)
  private static class ProxyConfigurationMixin {
    @JsonPOJOBuilder(withPrefix = "")
    static class Builder {}
  }

View on GitHub (pinned to 12126d8942)

Solutions

1. Pin the AWS SDK v2 version to the one declared by the Beam version you use (check Beam's dependency BOM).
2. Upgrade both Beam and AWS SDK together so AwsModule's expected field names match the provider classes.
3. Avoid shading/relocating the AWS SDK classes used by Beam, which can break reflective access.

Example fix

// before (pom.xml)
<dependency><groupId>software.amazon.awssdk</groupId><artifactId>kinesis</artifactId><version>2.30.0</version></dependency>
// after — use the Beam-managed version
<dependency><groupId>software.amazon.awssdk</groupId><artifactId>kinesis</artifactId><version>${beam-aws-sdk.version}</version></dependency>

// pin AWS SDK version to Beam's managed version in your build
// and fail fast in a smoke test that round-trips serialize/deserialize the provider

try {
  json = AwsSerializableUtils.serializeAwsCredentialsProvider(provider);
} catch (IllegalArgumentException | IOException e) {
  log.error("Provider reflection failed; check AWS SDK/Beam version alignment", e);
}

When it happens

Trigger: Serializing or deserializing a profile or web-identity-token credentials provider whose private field (e.g. profileName, request supplier) no longer exists under the expected name because the AWS SDK version differs from what AwsModule was written against.

Common situations: Mixing incompatible versions of software.amazon.awssdk with Beam's AWS2 SDK module; shading/relocation breaking reflective field access; provider classes changed in an SDK upgrade.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• AWS credential provider type '%s' is not supported
• Unsupported AWS credentials provider type
• can not be serialized to Json
• error when invoking Coder factory method
• cannot register Coder : does not have an accessible method n

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d45b9bf861d5b231. Report an issue: GitHub.