opendatalab/MinerU · error · InvalidConfig

the bucket_name in s3_configs: {s3_configs} must be unique

Error message

the bucket_name in s3_configs: {s3_configs} must be unique

What it means

InvalidConfig raised when two or more S3Config entries in s3_configs share the same bucket_name. The mixin keys its client cache by bucket name (`self._s3_clients_h[bucket_name]`), so duplicate bucket entries would make credential selection ambiguous and are rejected up front.

Source

Thrown at mineru/data/data_reader_writer/multi_bucket_s3.py:54

        arr = default_prefix.strip('/').split('/')
        self.default_bucket = arr[0]
        self.default_prefix = '/'.join(arr[1:])

        found_default_bucket_config = False
        for conf in s3_configs:
            if conf.bucket_name == self.default_bucket:
                found_default_bucket_config = True
                break

        if not found_default_bucket_config:
            raise InvalidConfig(
                f'default_bucket: {self.default_bucket} config must be provided in s3_configs: {s3_configs}'
            )

        uniq_bucket = set([conf.bucket_name for conf in s3_configs])
        if len(uniq_bucket) != len(s3_configs):
            raise InvalidConfig(
                f'the bucket_name in s3_configs: {s3_configs} must be unique'
            )

        self.s3_configs = s3_configs
        self._s3_clients_h: dict = {}


class MultiBucketS3DataReader(DataReader, MultiS3Mixin):
    def read(self, path: str) -> bytes:
        """Read the path from s3, select diffect bucket client for each request
        based on the bucket, also support range read.

        Args:
            path (str): the s3 path of file, the path must be in the format of s3://bucket_name/path?offset,limit.
            for example: s3://bucket_name/path?0,100.

        Returns:
            bytes: the content of s3 file.

View on GitHub (pinned to 4fe4bde114)

Solutions

1. Deduplicate by bucket_name before constructing the reader/writer, keeping the intended credential per bucket.
2. If credentials changed, replace the existing S3Config entry instead of adding a second one.
3. Assert uniqueness in your config-loading code so the failure surfaces with your own context.

Example fix

# before
s3_configs = team_a_configs + team_b_configs  # both contain 'shared-bucket'
reader = MultiBucketS3DataReader('shared-bucket/', s3_configs)

# after
merged = {c.bucket_name: c for c in team_a_configs + team_b_configs}
reader = MultiBucketS3DataReader('shared-bucket/', list(merged.values()))

def dedupe_configs(configs: list) -> list:
    by_bucket = {c.bucket_name: c for c in configs}
    if len(by_bucket) != len(configs):
        import logging
        logging.warning('duplicate bucket_name in s3_configs; keeping last entry per bucket')
    return list(by_bucket.values())

reader = MultiBucketS3DataReader(prefix, dedupe_configs(configs))

When it happens

Trigger: Passing s3_configs = [S3Config(bucket_name='x', ak='1', ...), S3Config(bucket_name='x', ak='2', ...)] — e.g. merging config lists from two environments or appending a 'default' config that already exists.

Common situations: Concatenating per-team config lists without dedup; updating credentials by appending the new config instead of replacing the old one; YAML anchors duplicating a bucket block.

Related errors

• default_prefix must be provided
• default_bucket: {self.default_bucket} config must be provide
• bucket name: {bucket_name} not found in s3_configs: {self.s3
• The provided path starts with '/'. This does not conform to
• Invalid S3 path format. Expected 's3://bucket-name/key' or '

AI-assisted analysis of opendatalab/MinerU@4fe4bde114 (2026-08-14). Data as JSON: /api/errors/53cedb096139faa2. Report an issue: GitHub.