unslothai/unsloth · critical · RuntimeError

The pinned {spec.name} archive contains duplicate files

Error message

The pinned {spec.name} archive contains duplicate files

What it means

The installer records each extracted relative path in a set; seeing the same relative path (under the package subtree) twice raises this error. Duplicate entries are a known tar smuggling technique — a benign file passes review, then a second entry with the same name overwrites it during extraction — so the library refuses rather than letting later entries win.

Source

Thrown at studio/backend/utils/third_party_source.py:670

                                )
                            parts = _archive_member_parts(member, spec)
                            if member.isdir():
                                continue
                            if not member.isfile() or member.size < 0:
                                raise RuntimeError(
                                    f"The pinned {spec.name} archive contains a non-regular file"
                                )
                            uncompressed_bytes += member.size
                            if uncompressed_bytes > _ARCHIVE_MAX_UNCOMPRESSED_BYTES:
                                raise RuntimeError(
                                    f"The pinned {spec.name} archive expands too large"
                                )
                            if len(parts) < 3 or parts[1] != spec.package:
                                continue
                            relative = "/".join(parts[1:])
                            _package_path_parts(relative, spec, kind = "archive")
                            if relative in extracted:
                                raise RuntimeError(
                                    f"The pinned {spec.name} archive contains duplicate files"
                                )
                            extracted.add(relative)
                            source_file = bundle.extractfile(member)
                            if source_file is None:
                                raise RuntimeError(
                                    f"The pinned {spec.name} archive contains an unreadable file"
                                )
                            destination_file = staging.joinpath(*parts[1:])
                            destination_file.parent.mkdir(parents = True, exist_ok = True)
                            remaining = member.size
                            with source_file, destination_file.open("wb") as handle:
                                while remaining:
                                    chunk = source_file.read(min(1024 * 1024, remaining))
                                    if not chunk:
                                        raise RuntimeError(
                                            f"The pinned {spec.name} archive ended unexpectedly"
                                        )

View on GitHub (pinned to 203007d190)

Solutions

  1. Find duplicates: tar -tzf source.tar.gz | sort | uniq -d
  2. Repack the tree cleanly (fresh export from the tagged revision) and re-pin digests
  3. Always pull from the canonical codeload URL for the revision rather than ad-hoc repacked mirrors
  4. If canonical artifacts duplicate, the pin itself is unusable — pin a different revision
Defensive patterns

Strategy: validation

Validate before calling

import tarfile, collections
with tarfile.open("source.tar.gz") as tf:
    names = [m.name for m in tf]
dups = [n for n, c in collections.Counter(names).items() if c > 1]
assert not dups, f"duplicate entries: {dups}"

Type guard

def archive_has_no_duplicates(archive: Path) -> bool:
    with tarfile.open(archive) as tf:
        seen = set()
        for m in tf:
            if m.name in seen:
                return False
            seen.add(m.name)
    return True

Try / catch

try:
    ensure_pinned_source(spec)
except RuntimeError as e:
    if "duplicate files" in str(e):
        raise SystemExit("archive contains duplicate entries — repin to a clean export")

Prevention

When it happens

Trigger: Two tar members normalize to the same `package/...` relative path — e.g. `repo-sha/foo/__init__.py` appearing twice, or case/directory variants that collapse to identical extracted paths within the parts[1]==spec.package subtree.

Common situations: Repacked archives accidentally containing a file twice (some zip->tar converters do this); maliciously crafted archives with a trailing duplicate overwrite entry; unusual archivers that emit both a dir-prefixed and bare variant of the same file.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/2b363a3d85e9f266. Report an issue: GitHub.