{"record":{"id":"53cedb096139faa2","repo":"opendatalab/MinerU","slug":"the-bucket-name-in-s3-configs-s3-configs-must-b","errorCode":null,"errorMessage":"the bucket_name in s3_configs: {s3_configs} must be unique","messagePattern":"the bucket_name in s3_configs: (.+?) must be unique","errorType":"validation","errorClass":"InvalidConfig","httpStatus":null,"severity":"error","filePath":"mineru/data/data_reader_writer/multi_bucket_s3.py","lineNumber":54,"sourceCode":"\n        arr = default_prefix.strip('/').split('/')\n        self.default_bucket = arr[0]\n        self.default_prefix = '/'.join(arr[1:])\n\n        found_default_bucket_config = False\n        for conf in s3_configs:\n            if conf.bucket_name == self.default_bucket:\n                found_default_bucket_config = True\n                break\n\n        if not found_default_bucket_config:\n            raise InvalidConfig(\n                f'default_bucket: {self.default_bucket} config must be provided in s3_configs: {s3_configs}'\n            )\n\n        uniq_bucket = set([conf.bucket_name for conf in s3_configs])\n        if len(uniq_bucket) != len(s3_configs):\n            raise InvalidConfig(\n                f'the bucket_name in s3_configs: {s3_configs} must be unique'\n            )\n\n        self.s3_configs = s3_configs\n        self._s3_clients_h: dict = {}\n\n\nclass MultiBucketS3DataReader(DataReader, MultiS3Mixin):\n    def read(self, path: str) -> bytes:\n        \"\"\"Read the path from s3, select diffect bucket client for each request\n        based on the bucket, also support range read.\n\n        Args:\n            path (str): the s3 path of file, the path must be in the format of s3://bucket_name/path?offset,limit.\n            for example: s3://bucket_name/path?0,100.\n\n        Returns:\n            bytes: the content of s3 file.","sourceCodeStart":36,"sourceCodeEnd":72,"githubUrl":"https://github.com/opendatalab/MinerU/blob/4fe4bde114a23ee5dd637eae99b767f4669bf58c/mineru/data/data_reader_writer/multi_bucket_s3.py#L36-L72","documentation":"InvalidConfig raised when two or more S3Config entries in s3_configs share the same bucket_name. The mixin keys its client cache by bucket name (`self._s3_clients_h[bucket_name]`), so duplicate bucket entries would make credential selection ambiguous and are rejected up front.","triggerScenarios":"Passing s3_configs = [S3Config(bucket_name='x', ak='1', ...), S3Config(bucket_name='x', ak='2', ...)] — e.g. merging config lists from two environments or appending a 'default' config that already exists.","commonSituations":"Concatenating per-team config lists without dedup; updating credentials by appending the new config instead of replacing the old one; YAML anchors duplicating a bucket block.","solutions":["Deduplicate by bucket_name before constructing the reader/writer, keeping the intended credential per bucket.","If credentials changed, replace the existing S3Config entry instead of adding a second one.","Assert uniqueness in your config-loading code so the failure surfaces with your own context."],"exampleFix":"# before\ns3_configs = team_a_configs + team_b_configs  # both contain 'shared-bucket'\nreader = MultiBucketS3DataReader('shared-bucket/', s3_configs)\n\n# after\nmerged = {c.bucket_name: c for c in team_a_configs + team_b_configs}\nreader = MultiBucketS3DataReader('shared-bucket/', list(merged.values()))","handlingStrategy":"validation","validationCode":"def dedupe_configs(configs: list) -> list:\n    by_bucket = {c.bucket_name: c for c in configs}\n    if len(by_bucket) != len(configs):\n        import logging\n        logging.warning('duplicate bucket_name in s3_configs; keeping last entry per bucket')\n    return list(by_bucket.values())\n\nreader = MultiBucketS3DataReader(prefix, dedupe_configs(configs))","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Deduplicate merged config lists by bucket_name before construction.","Update credentials in place (replace the S3Config) rather than appending a new one.","Keep bucket configs in a single mapping keyed by bucket_name in your own config layer."],"tags":["s3","config","validation","duplicate-keys"],"backgroundTag":null,"analyzedSha":"4fe4bde114a23ee5dd637eae99b767f4669bf58c","analyzedAt":"2026-08-14T21:29:18.456Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}