What causes "tar archive exceeds maximum file count of %d" in vxcontrol/pentagi?

Split the archive into chunks of ≤1000 files and pull them separately.

tar archive exceeds maximum file count of %d

Error message

tar archive exceeds maximum file count of %d

What it means

ExtractTar enforces MaxPullFiles (1000) regular-file entries per archive; exceeding it aborts extraction with this error. It is a deliberate anti-DoS limit so a small malicious tarball cannot exhaust inodes or disk via many files.

Source

Thrown at backend/pkg/flowfiles/files.go:407

		}

		entryPath := filepath.Join(destDir, filepath.Clean(filepath.FromSlash(hdr.Name)))
		if !IsWithinDir(entryPath, destDir) {
			continue
		}

		switch hdr.Typeflag {
		case tar.TypeDir:
			if err := os.MkdirAll(entryPath, 0755); err != nil {
				return fmt.Errorf("failed to create directory '%s': %w", entryPath, err)
			}
		case tar.TypeReg, tar.TypeRegA:
			if hdr.Size < 0 {
				return fmt.Errorf("tar entry '%s' has invalid size %d", hdr.Name, hdr.Size)
			}
			filesCount++
			if filesCount > MaxPullFiles {
				return fmt.Errorf("tar archive exceeds maximum file count of %d", MaxPullFiles)
			}
			totalSize += hdr.Size
			if totalSize > MaxPullTotalSize {
				return fmt.Errorf("tar archive exceeds maximum total size of %d bytes", MaxPullTotalSize)
			}

			if err := os.MkdirAll(filepath.Dir(entryPath), 0755); err != nil {
				return fmt.Errorf("failed to create parent directory for '%s': %w", entryPath, err)
			}

			f, err := os.OpenFile(entryPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0644)
			if err != nil {
				return fmt.Errorf("failed to create file '%s': %w", entryPath, err)
			}
			_, copyErr := io.CopyN(f, tr, hdr.Size)
			f.Close()
			if copyErr != nil {
				return fmt.Errorf("failed to write file '%s': %w", entryPath, copyErr)

View on GitHub (pinned to ea665308ba)

Solutions

Split the archive into chunks of ≤1000 files and pull them separately.
Exclude unnecessary paths (dependencies, caches, .git) when creating the tar on the source side.
If the workload legitimately needs more files, raise the MaxPullFiles constant in backend/pkg/flowfiles/files.go and redeploy — accepting the higher resource cost.
Tar a single compressed bundle (one big file) instead of many small files.

Example fix

// before
tar -cf bundle.tar ./project/
// after
tar -cf bundle.tar --exclude='./project/.git' --exclude='./project/node_modules' ./project/

Defensive patterns

Strategy: validation

Validate before calling

// Count regular files without extracting (Go or shell):
// tar -tf bundle.tar | grep -vc '/$'
// if the count exceeds 1000, split or prune before calling ExtractTar.

Try / catch

err := flowfiles.ExtractTar(rc, destDir)
if err != nil {
    if strings.Contains(err.Error(), "exceeds maximum file count") {
        return fmt.Errorf("archive has too many files (limit %d); split or exclude paths", flowfiles.MaxPullFiles)
    }
    return err
}

Prevention

Exclude dependency/cache directories (.git, node_modules) when creating flow archives.
Count archive members with `tar -tf | wc -l` before pulling large project trees.
Prefer single-file compressed bundles over many small files.

When it happens

Trigger: Calling ExtractTar (directly or via PullFlowFiles) on an archive containing more than 1000 regular-file entries; file count is checked incrementally as entries stream through.

Common situations: Backing up a dependency tree (node_modules-like), a dataset directory with thousands of small files, or recursively including uploads; users legitimately archiving large project trees then pulling them into a flow.

Related errors

AI-assisted analysis of vxcontrol/pentagi@ea665308ba (2026-09-01). Data as JSON: /api/errors/41def34f3239c9da. Report an issue: GitHub.