opendatalab/MinerU · error · InvalidConfig

default_bucket: config must be provided in s3_configs

Error message

default_bucket: {self.default_bucket} config must be provided in s3_configs: {s3_configs}

What it means

InvalidConfig raised when the bucket named by the first segment of default_prefix has no matching S3Config in s3_configs. The mixin needs credentials for the default bucket, so it scans s3_configs for conf.bucket_name == self.default_bucket and aborts when none matches.

Solutions

1. Add an S3Config whose bucket_name exactly equals the first segment of default_prefix.
2. Check for typos/case differences between default_prefix's bucket segment and the S3Config bucket_name values (matching is exact).
3. Print the parsed default bucket (`'my-prefix'.strip('/').split('/')[0]`) and diff it against `[c.bucket_name for c in s3_configs]` to find the mismatch.

Example fix

# before
reader = MultiBucketS3DataReader(
    'docs-bucket/pdfs',
    [S3Config(bucket_name='archive-bucket', ...)],
)

# after
reader = MultiBucketS3DataReader(
    'docs-bucket/pdfs',
    [
        S3Config(bucket_name='docs-bucket', ...),
        S3Config(bucket_name='archive-bucket', ...),
    ],
)

from mineru.data.utils.exceptions import InvalidConfig

def validate_s3_setup(default_prefix: str, configs: list) -> None:
    default_bucket = default_prefix.strip('/').split('/')[0]
    names = [c.bucket_name for c in configs]
    if not default_bucket:
        raise ValueError('default_prefix must contain a bucket segment')
    if default_bucket not in names:
        raise ValueError(f'add S3Config for default bucket {default_bucket!r}; have {names}')

try:
    reader = MultiBucketS3DataReader(prefix, configs)
except InvalidConfig as e:
    raise RuntimeError(f'S3 setup invalid: {e.msg}') from e

When it happens

Trigger: MultiBucketS3DataReader('bucket-a/...', [S3Config(bucket_name='bucket-b', ...)]) — the default bucket 'bucket-a' is absent from the config list. Also triggered by leading/trailing whitespace or a '/' suffix on the bucket segment (only strip('/') is applied).

Common situations: Copy-paste mismatch between the default prefix and the credentials list; renaming a bucket in one place but not the other; per-environment configs (dev bucket list used with prod prefix); typos in bucket names.

Related errors

• default_prefix must be provided
• bucket name: not found in s3_configs
• the bucket_name in s3_configs
• ak, sk or endpoint not found in
• Invalid S3 path format. Expected 's3://bucket-name/key' or…

AI-assisted analysis of opendatalab/MinerU@4fe4bde114 (2026-08-14). Data as JSON: /api/errors/581fdf9f3017d26b. Report an issue: GitHub.

Appendix: source

Thrown at mineru/data/data_reader_writer/multi_bucket_s3.py:48

            InvalidConfig: default bucket config not in s3_configs.
            InvalidConfig: bucket name not unique in s3_configs.
            InvalidConfig: default bucket must be provided.
        """
        if len(default_prefix) == 0:
            raise InvalidConfig('default_prefix must be provided')

        arr = default_prefix.strip('/').split('/')
        self.default_bucket = arr[0]
        self.default_prefix = '/'.join(arr[1:])

        found_default_bucket_config = False
        for conf in s3_configs:
            if conf.bucket_name == self.default_bucket:
                found_default_bucket_config = True
                break

        if not found_default_bucket_config:
            raise InvalidConfig(
                f'default_bucket: {self.default_bucket} config must be provided in s3_configs: {s3_configs}'
            )

        uniq_bucket = set([conf.bucket_name for conf in s3_configs])
        if len(uniq_bucket) != len(s3_configs):
            raise InvalidConfig(
                f'the bucket_name in s3_configs: {s3_configs} must be unique'
            )

        self.s3_configs = s3_configs
        self._s3_clients_h: dict = {}


class MultiBucketS3DataReader(DataReader, MultiS3Mixin):
    def read(self, path: str) -> bytes:
        """Read the path from s3, select diffect bucket client for each request
        based on the bucket, also support range read.

View on GitHub (pinned to 4fe4bde114)