huggingface/pytorch-image-models · error · ValueError

Invalid or corrupt tar info cache file {cache_path}.

Error message

Invalid or corrupt tar info cache file {cache_path}.

What it means

When a timm tar dataset directory has a cached tar-info pickle (e.g. tartrees.pkl), it is loaded to avoid rescanning the tars; this error means the unpickled object is not the expected {'tartrees': [...]} dict structure, i.e. the cache is corrupt or from an incompatible version.

Source

Thrown at timm/data/readers/reader_image_in_tar.py:115

        tar_filenames = glob(os.path.join(root, '*.tar'), recursive=True)
    num_tars = len(tar_filenames)
    tar_bytes = sum([os.path.getsize(f) for f in tar_filenames])
    assert num_tars, f'No .tar files found at specified path ({root}).'

    _logger.info(f'Scanning {tar_bytes/1024**2:.2f}MB of tar files...')
    info = dict(tartrees=[])
    cache_path = ''
    if cache_tarinfo is None:
        cache_tarinfo = True if tar_bytes > 10*1024**3 else False  # FIXME magic number, 10GB
    if cache_tarinfo:
        cache_filename = '_' + root_name + CACHE_FILENAME_SUFFIX
        cache_path = os.path.join(root, cache_filename)
    if os.path.exists(cache_path):
        _logger.info(f'Reading tar info from cache file {cache_path}.')
        with open(cache_path, 'rb') as pf:
            info = _TarInfoUnpickler(pf).load()
        if not isinstance(info, dict) or not isinstance(info.get('tartrees'), list):
            raise ValueError(f'Invalid or corrupt tar info cache file {cache_path}.')
        assert len(info['tartrees']) == num_tars, "Cached tartree len doesn't match number of tarfiles"
    else:
        for i, fn in enumerate(tar_filenames):
            path = '' if root_is_tar else os.path.splitext(os.path.basename(fn))[0]
            with tarfile.open(fn, mode='r|') as tf:  # tarinfo scans done in streaming mode
                parent_info = dict(name=os.path.relpath(fn, root), path=path, ti=None, children=[], samples=[])
                num_samples = _extract_tarinfo(tf, parent_info, extensions=extensions)
                num_children = len(parent_info["children"])
                _logger.debug(
                    f'{i}/{num_tars}. Extracted tarinfos from {fn}. {num_children} children, {num_samples} samples.')
            info['tartrees'].append(parent_info)
        if cache_path:
            _logger.info(f'Writing tar info to cache file {cache_path}.')
            with open(cache_path, 'wb') as pf:
                pickle.dump(info, pf)

    samples = []
    labels = []

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Delete the cache file (default name like _tartrees.pkl / cache_filename in the tar root) so it is rebuilt by rescanning the tars.
2. If it recurs, check tar file integrity (tar -tf each shard) and disk space.
3. Re-run dataset creation to regenerate a fresh cache.

Example fix

rm /data/train_imagenet/tartrees.pkl
# then re-run dataset creation; tars are rescanned and cache rebuilt

import os
from timm.data.readers.reader_image_in_tar import CACHE_FILE_NAME as _  # module-specific name may vary
cache = os.path.join(root, cache_filename)
try:
    ds = ImageDataset(root)  # in_tar inferred from tar presence
except ValueError as e:
    if 'cache file' in str(e) and os.path.exists(cache):
        os.remove(cache)
        ds = ImageDataset(root)

try:
    reader = ReaderInTar(root)
except ValueError as e:
    if 'corrupt tar info cache' in str(e):
        os.remove(cache_path); reader = ReaderInTar(root)
    else:
        raise

When it happens

Trigger: Constructing ImageDataset/ReaderInTar on a tar-root directory where the cache file exists but was truncated, written by another tool, or is a stale format.

Common situations: A cache write was interrupted (disk full, killed process); the directory previously held different tar contents and the cache is stale; format changes between timm versions.

Related errors

• Input image must have positive dimensions, got H={height}, W
• Invalid class map file, expected a dict ({class_map_path}).
• Dataset length is unknown, please pass `num_samples` explici
• Found 0 images in subfolders of {root}. Supported image exte
• split {split} not found in info ({info.get('splits', {}).key

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/3832060b4158c5d4. Report an issue: GitHub.