apache/beam · critical

error loading AWS config

Error message

error loading AWS config: %v

What it means

s3.New panics if the AWS SDK's config.LoadDefaultConfig fails to assemble a configuration from the default chain (env vars, shared credentials/config files, EC2/ECS metadata). Without a valid config the SDK cannot construct an S3 client, so the filesystem constructor fails fast. This surfaces credential/provider initialization problems, not S3 request failures.

Solutions

1. Check AWS_* env vars and the shared credentials/config files (~/.aws/credentials, ~/.aws/config) for validity
2. Verify AWS_PROFILE names an existing profile
3. Ensure AWS_CONFIG_FILE / AWS_SHARED_CREDENTIALS_FILE point to readable files
4. Replace the default chain with explicit config: config.LoadDefaultConfig(ctx, config.WithSharedConfigProfile("name")) and handle the error yourself instead of panicking

Example fix

// before
fs := s3.New(ctx)
// after
cfg, err := config.LoadDefaultConfig(ctx)
if err != nil {
    log.Fatalf("AWS config error: %v", err)
}
client := s3.NewFromConfig(cfg)

// Pre-flight: run config.LoadDefaultConfig yourself at startup and fail with a clear message.
cfg, err := config.LoadDefaultConfig(ctx)
if err != nil { return fmt.Errorf("AWS config: %w", err) }

defer func() { if r := recover(); r != nil { log.Fatalf("s3.New: %v", r) } }()

When it happens

Trigger: Calling filesystem/s3.New(ctx) with malformed AWS_* environment variables (e.g. unparseable AWS_PROFILE or invalid AWS_CONFIG_FILE), or a shared credentials file unreadable/corrupt so LoadDefaultConfig returns an error.

Common situations: Containers missing IAM role metadata; mis-set AWS_CONFIG_FILE pointing to a directory; AWS_PROFILE referencing a profile absent from ~/.aws/config.

Related errors

• couldn't decode characteristic for variant
• error re-encoding characteristic for variant
• AfterProcessingTime trigger set without a delay or…
• All parts but the last must be larger than
• At least one subtrigger required for composite triggers.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f55eb08ce4c4bbce. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/filesystem/s3/s3.go:45

	"github.com/apache/beam/sdks/v2/go/pkg/beam/util/fsx"
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

func init() {
	filesystem.Register("s3", New)
}

type fs struct {
	client *s3.Client
}

// New creates a new S3 filesystem using AWS default configuration sources.
func New(ctx context.Context) filesystem.Interface {
	cfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		panic(fmt.Sprintf("error loading AWS config: %v", err))
	}

	client := s3.NewFromConfig(cfg)
	return &fs{client: client}
}

// Close closes the filesystem.
func (f *fs) Close() error {
	return nil
}

// List returns a slice of the files in the filesystem that match the glob pattern.
func (f *fs) List(ctx context.Context, glob string) ([]string, error) {
	bucket, keyPattern, err := parseURI(glob)
	if err != nil {
		return nil, fmt.Errorf("error parsing S3 uri: %v", err)
	}

View on GitHub (pinned to 12126d8942)