opendatalab/MinerU · error · ValueError

Invalid S3 path format. Expected 's3://bucket-name/key' or '

Error message

Invalid S3 path format. Expected 's3://bucket-name/key' or 's3a://bucket-name/key'.

What it means

ValueError from parse_s3path() when the path has neither an s3:// / s3a:// scheme nor a leading '/'. After remove_non_official_s3_args() and strip(), anything that is not an S3 URI falls into this branch — the parser has no way to derive a bucket and key from it.

Source

Thrown at mineru/data/utils/path_utils.py:23

    """
    example: s3://abc/xxxx.json?bytes=0,81350 ==> s3://abc/xxxx.json
    """
    arr = s3path.split("?")
    return arr[0]

def parse_s3path(s3path: str):
    # from s3pathlib import S3Path
    # p = S3Path(remove_non_official_s3_args(s3path))
    # return p.bucket, p.key
    s3path = remove_non_official_s3_args(s3path).strip()
    if s3path.startswith(('s3://', 's3a://')):
        prefix, path = s3path.split('://', 1)
        bucket_name, key = path.split('/', 1)
        return bucket_name, key
    elif s3path.startswith('/'):
        raise ValueError("The provided path starts with '/'. This does not conform to a valid S3 path format.")
    else:
        raise ValueError("Invalid S3 path format. Expected 's3://bucket-name/key' or 's3a://bucket-name/key'.")


def parse_s3_range_params(s3path: str):
    """
    example: s3://abc/xxxx.json?bytes=0,81350 ==> [0, 81350]
    """
    arr = s3path.split("?bytes=")
    if len(arr) == 1:
        return None
    return arr[1].split(",")

View on GitHub (pinned to 4fe4bde114)

Solutions

Prefix the path with the scheme: 's3://bucket/key'.
Convert HTTPS-style URLs to s3:// form before calling S3 readers.
Filter or normalize input paths early (reject non-s3:// entries) so bad rows are reported with context instead of a raw ValueError.
For local files, route to a local reader instead.

Example fix

# before
parse_s3path('my-bucket/doc.pdf')          # ValueError
parse_s3path('https://my-bucket.s3.amazonaws.com/doc.pdf')  # ValueError

# after
parse_s3path('s3://my-bucket/doc.pdf')

Defensive patterns

Strategy: type-guard

Validate before calling

def validate_s3_path(path: str) -> str:
    p = path.strip()
    if not p.startswith(('s3://', 's3a://')):
        raise ValueError(f"path must start with 's3://' or 's3a://', got {path!r}")
    rest = p.split('://', 1)[1]
    if '/' not in rest or not rest.split('/', 1)[0]:
        raise ValueError(f'path must be s3://bucket/key, got {path!r}')
    return p

Type guard

def is_valid_s3_path(path: str) -> bool:
    p = path.strip()
    if not p.startswith(('s3://', 's3a://')):
        return False
    rest = p.split('://', 1)[1]
    bucket, _, key = rest.partition('/')
    return bool(bucket) and bool(key)

Try / catch

try:
    bucket, key = parse_s3path(user_path)
except ValueError:
    # log the offending row and skip, or route to local reader if it is a local file
    raise

Prevention

Convert https:// S3 console URLs to s3://bucket/key during ingestion.
Reject empty strings and scheme-less paths at input validation time.
Separate local vs S3 path handling explicitly instead of a single shared parse.

When it happens

Trigger: parse_s3path('bucket/key.pdf') (scheme omitted), parse_s3path('https://bucket.s3.amazonaws.com/key'), parse_s3path('C:\\doc.pdf'), or an empty/whitespace string passed to S3 readers.

Common situations: Users pasting HTTPS S3 console URLs or plain bucket/key shorthand; Windows paths reaching S3 code by mistake; empty strings from optional config fields; upstream data files containing malformed URIs.

Related errors

AI-assisted analysis of opendatalab/MinerU@4fe4bde114 (2026-08-14). Data as JSON: /api/errors/0ad6875710e90025. Report an issue: GitHub.