no S3 access key/secret key is configured and no ambient AWS

Error message

no S3 access key/secret key is configured and no ambient AWS credentials could be loaded through the AWS SDK default chain (env vars, profile, ECS/EC2 instance role): {cause}. If an EC2/ECS instance role is expected to be used, the instance metadata service must be reachable from the process running Windmill — on EC2 the AWS Rust SDK only supports IMDSv2, so when Windmill runs in a Docker container the instance metadata hop limit (HttpPutResponseHopLimit) must be at least 2

What it means

The S3 credential cache in windmill-object-store refreshed ambient AWS credentials via the SDK default chain (env vars, shared profile, ECS/EC2 instance role) and none could be provided. The message distinguishes 'no static access/secret key configured' from 'no ambient credentials resolvable' and includes IMDSv2/hop-limit guidance for Docker-on-EC2 setups.

Source

Thrown at backend/windmill-object-store/src/lib.rs:832

        }
    }

    async fn get(&self) -> anyhow::Result<aws_sdk_sts::config::Credentials> {
        if let Some((creds, fetched_at)) = self.cached.read().await.as_ref() {
            if Self::still_valid(creds, fetched_at.elapsed()) {
                return Ok(creds.clone());
            }
        }
        // The write lock is held across the chain resolution so concurrent requests don't all
        // hit the metadata service at once.
        let mut guard = self.cached.write().await;
        if let Some((creds, fetched_at)) = guard.as_ref() {
            if Self::still_valid(creds, fetched_at.elapsed()) {
                return Ok(creds.clone());
            }
        }
        let creds = self.chain.provide_credentials().await.map_err(|e| {
            anyhow::anyhow!(
                "no S3 access key/secret key is configured and no ambient AWS credentials could \
                 be loaded through the AWS SDK default chain (env vars, profile, ECS/EC2 instance \
                 role): {cause}. If an EC2/ECS instance role is expected to be used, the instance \
                 metadata service must be reachable from the process running Windmill — on EC2 the \
                 AWS Rust SDK only supports IMDSv2, so when Windmill runs in a Docker container \
                 the instance metadata hop limit (HttpPutResponseHopLimit) must be at least 2",
                cause = format!("{:#}", anyhow::Error::new(e))
            )
        })?;
        *guard = Some((creds.clone(), std::time::Instant::now()));
        Ok(creds)
    }
}

#[cfg(feature = "parquet")]
lazy_static::lazy_static! {
    static ref AMBIENT_AWS_CREDS_PROVIDERS: Cache<String, Arc<AmbientAwsCredentials>> =
        Cache::new(20);

View on GitHub (pinned to e474e8803c)

Solutions

Set an access key/secret key explicitly on the S3 resource, or provide AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (and AWS_REGION) env vars to the Windmill containers
If relying on the EC2 instance role, raise the IMDS hop limit: aws ec2 modify-instance-metadata-options --http-put-response-hop-limit 2 (required for Docker containers using IMDSv2)
Verify the instance metadata service is reachable from the container: curl -s http://169.254.169.254/latest/api/token with the IMDSv2 PUT
On ECS, ensure the task definition has a task role and the metadata endpoint (169.254.170.2) is reachable
Check AWS_PROFILE and shared credentials file exist inside the container if using profiles

Example fix

// before: relying on ambient creds in Docker on EC2 (hop limit 1)
# no action
// after
aws ec2 modify-instance-metadata-options \
  --instance-id i-0123456789abcdef0 \
  --http-put-response-hop-limit 2 \
  --http-endpoint enabled
# and/or set in windmill container env:
# AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_REGION=us-east-1

Defensive patterns

Strategy: fallback

Validate before calling

// Pre-flight: fail early if no ambient credentials are resolvable
async fn has_ambient_aws_creds(chain: &aws_config::default_provider::credentials::DefaultCredentialsChain) -> bool {
    chain.provide_credentials().await.is_ok()
}

Type guard

fn s3_resource_has_static_keys(r: &S3Resource) -> bool {
    r.access_key.as_deref().map(|s| !s.is_empty()).unwrap_or(false)
        && r.secret_key.as_deref().map(|s| !s.is_empty()).unwrap_or(false)
}

Try / catch

match store.get(&path).await {
    Ok(v) => v,
    Err(e) if e.to_string().contains("no ambient AWS credentials") => {
        // surface actionable guidance: set static keys on the S3 resource or fix IMDS hop limit
        return Err(e.context("configure S3 credentials on the resource or fix instance role access"));
    }
    Err(e) => return Err(e),
}

Prevention

Prefer explicit access/secret keys on the S3 resource in non-AWS environments
On EC2+Docker, set HttpPutResponseHopLimit >= 2 when creating instances
Set AWS_REGION explicitly — missing region often accompanies credential failures
Smoke-test S3 connectivity after deployment changes
Verify instance profiles/task roles are attached before scaling out

When it happens

Trigger: S3 resource accessed without an access key/secret key in the resource config while get() refreshes credentials and chain.provide_credentials() fails — no AWS_* env vars, no ~/.aws profile, no instance metadata reachable.

Common situations: Windmill running in Docker on EC2 with container hop limit 1 (default) blocking IMDSv2 calls; running outside AWS without any AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY; wrong AWS_PROFILE; IMDS unreachable due to security-group/network-policy rules; ECS task missing the task-role IAM permission.

Related errors

AI-assisted analysis of windmill-labs/windmill@e474e8803c (2026-09-03). Data as JSON: /api/errors/4cf4ccccfee7a80a. Report an issue: GitHub.