apache/hadoop · error · NoAuthWithAWSException

{component}: Invalid AWS credentials in {credentials} requir

Error message

{component}: Invalid AWS credentials in {credentials} required: {typeRequired}

What it means

Thrown by MarshalledCredentialBinding.toAWSCredentials when the marshalled static S3A credentials fail validation for the credential type the component requires. S3A converts the fs.s3a.access.key / fs.s3a.secret.key / fs.s3a.session.token triple into AWS SDK AwsBasicCredentials or AwsSessionCredentials; before doing so it checks that all fields the required type needs are present and non-empty. This is the connector's fail-fast guard against half-configured static credentials (note: a completely empty set throws NoAwsCredentialsException instead).

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/MarshalledCredentialBinding.java:168

   * SDK references out of that class, the logic is implemented here instead,
   * @param marshalled marshalled credentials
   * @param typeRequired type of credentials required
   * @param component component name for exception messages.
   * @return a new set of credentials
   * @throws NoAuthWithAWSException validation failure
   * @throws NoAwsCredentialsException the credentials are actually empty.
   */
  public static AwsCredentials toAWSCredentials(
      final MarshalledCredentials marshalled,
      final MarshalledCredentials.CredentialTypeRequired typeRequired,
      final String component)
      throws NoAuthWithAWSException, NoAwsCredentialsException {

    if (marshalled.isEmpty()) {
      throw new NoAwsCredentialsException(component, NO_AWS_CREDENTIALS);
    }
    if (!marshalled.isValid(typeRequired)) {
      throw new NoAuthWithAWSException(component + ":" +
          marshalled.buildInvalidCredentialsError(typeRequired));
    }
    final String accessKey = marshalled.getAccessKey();
    final String secretKey = marshalled.getSecretKey();
    if (marshalled.hasSessionToken()) {
      // a session token was supplied, so return session credentials
      return AwsSessionCredentials.create(accessKey, secretKey,
          marshalled.getSessionToken());
    } else {
      // these are full credentials
      return AwsBasicCredentials.create(accessKey, secretKey);
    }
  }

  /**
   * Request a set of credentials from an STS endpoint.
   * @param parentCredentials the parent credentials needed to talk to STS
   * @param configuration AWS client configuration

View on GitHub (pinned to 2add963021)

Solutions

1. Set the complete triple for the required type: fs.s3a.access.key + fs.s3a.secret.key, plus fs.s3a.session.token when session credentials are required, all non-empty and trimmed
2. If you intend to authenticate via IAM instance roles or environment variables instead, delete the static keys and set fs.s3a.aws.credentials.provider to the matching provider (e.g. InstanceProfileCredentialsProvider or EnvironmentVariableCredentialsProvider)
3. Print the effective config on the exact node/executor that fails (conf.getTrimmed for each key) to find which override is incomplete
4. For STS-derived session credentials, re-generate them: expired sessions often get replaced by half-updated configs

Example fix

<!-- before: session type required but token missing -->
<property><name>fs.s3a.access.key</name><value>AKIA...</value></property>
<property><name>fs.s3a.secret.key</name><value>secret...</value></property>

<!-- after: complete session triple, or drop all three and use fs.s3a.aws.credentials.provider instead -->
<property><name>fs.s3a.access.key</name><value>AKIA...</value></property>
<property><name>fs.s3a.secret.key</name><value>secret...</value></property>
<property><name>fs.s3a.session.token</name><value>FwoGZXIv...</value></property>

import org.apache.hadoop.fs.s3a.auth.MarshalledCredentials;
import org.apache.hadoop.fs.s3a.auth.MarshalledCredentialBinding;

MarshalledCredentials mc = MarshalledCredentials.fromConfiguration(conf);
MarshalledCredentials.CredentialTypeRequired required =
    MarshalledCredentials.CredentialTypeRequired.Session; // whatever your component needs
if (!mc.isValid(required)) {
  throw new IOException("Refusing to start: incomplete S3A credentials - "
      + mc.buildInvalidCredentialsError(required));
}

try {
  FileSystem fs = FileSystem.get(new URI("s3a://bucket"), conf);
} catch (NoAuthWithAWSException | NoAwsCredentialsException e) {
  // config problem, not transient: report which component/type failed and stop
  throw new JobSetupException("S3A credentials rejected: " + e.getMessage(), e);
}

When it happens

Trigger: A component (filesystem init, delegation-token binding) calls toAWSCredentials(marshalled, typeRequired, component) and marshalled.isValid(typeRequired) returns false. Concrete case: CredentialTypeRequired.Session demanded but fs.s3a.session.token is missing/blank, or fs.s3a.access.key / fs.s3a.secret.key is empty or contains surrounding whitespace.

Common situations: Hand-edited core-site.xml with a missing or typo'd secret key; switching between long-lived keys and STS session credentials and leaving one of the three fields behind; pasting keys with trailing whitespace/newlines; expecting S3A to read AWS_ACCESS_KEY_ID env vars (it does not for static keys); cluster-wide config differing from the job-level override.

Related errors

• SimpleAWSCredentialsProvider: No AWS credentials in the Hado
• No AWS credential providers created by Delegation Token Bind
• No COS Credential Providers
• Cannot find password option {key}
• SSE-C is enabled but no encryption key was declared in fs.s3

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6225fbaa9e2ffa21. Report an issue: GitHub.