What causes "tar archive exceeds maximum total size of %d bytes" in vxcontrol/pentagi?

Compress the payload (tar.gz of one file) so the total fits, or split into multiple archives under 2 GiB each.

tar archive exceeds maximum total size of %d bytes

Error message

tar archive exceeds maximum total size of %d bytes

What it means

ExtractTar enforces MaxPullTotalSize (2 GiB) as the running sum of regular-file entry sizes; exceeding it aborts with this error. Like the file-count limit, it is an anti-DoS guard against zip-bomb-style archives that would exhaust disk.

Source

Thrown at backend/pkg/flowfiles/files.go:411

			continue
		}

		switch hdr.Typeflag {
		case tar.TypeDir:
			if err := os.MkdirAll(entryPath, 0755); err != nil {
				return fmt.Errorf("failed to create directory '%s': %w", entryPath, err)
			}
		case tar.TypeReg, tar.TypeRegA:
			if hdr.Size < 0 {
				return fmt.Errorf("tar entry '%s' has invalid size %d", hdr.Name, hdr.Size)
			}
			filesCount++
			if filesCount > MaxPullFiles {
				return fmt.Errorf("tar archive exceeds maximum file count of %d", MaxPullFiles)
			}
			totalSize += hdr.Size
			if totalSize > MaxPullTotalSize {
				return fmt.Errorf("tar archive exceeds maximum total size of %d bytes", MaxPullTotalSize)
			}

			if err := os.MkdirAll(filepath.Dir(entryPath), 0755); err != nil {
				return fmt.Errorf("failed to create parent directory for '%s': %w", entryPath, err)
			}

			f, err := os.OpenFile(entryPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0644)
			if err != nil {
				return fmt.Errorf("failed to create file '%s': %w", entryPath, err)
			}
			_, copyErr := io.CopyN(f, tr, hdr.Size)
			f.Close()
			if copyErr != nil {
				return fmt.Errorf("failed to write file '%s': %w", entryPath, copyErr)
			}
		}
	}

View on GitHub (pinned to ea665308ba)

Solutions

Compress the payload (tar.gz of one file) so the total fits, or split into multiple archives under 2 GiB each.
Exclude bulky non-essential content (images, videos, build artifacts) from the tar on the producer side.
Free up expectations: if legitimate data must exceed 2 GiB, raise MaxPullTotalSize in backend/pkg/flowfiles/files.go and ensure the destination volume has headroom.
Transfer large artifacts out-of-band (object storage URL) instead of through the flow-files path.

Example fix

// before
tar -cf big.tar ./data/
// after
tar -czf big.tar.gz ./data/  # and split: split -b 1900M big.tar.gz big.tar.gz.part-

Defensive patterns

Strategy: validation

Validate before calling

// Check declared total size without extracting:
// tar -tvf bundle.tar | awk '{s+=$3} END {print s}'
// if s > 2147483648, compress or split before ExtractTar.

Try / catch

err := flowfiles.ExtractTar(rc, destDir)
if err != nil {
    if strings.Contains(err.Error(), "exceeds maximum total size") {
        return fmt.Errorf("archive exceeds %d byte limit; compress or split it", flowfiles.MaxPullTotalSize)
    }
    return err
}

Prevention

Compress large payloads (tar.gz single file) so the declared total stays under 2 GiB.
Split oversized datasets into multiple archives below the limit.
Transfer very large artifacts via object storage links rather than the flow-files channel.

When it happens

Trigger: Pulling an archive whose declared regular-file sizes total more than 2 GiB via PullFlowFiles/ExtractTar; the check happens before any parent directory or file is written for the offending entry.

Common situations: Archiving large datasets, VM images, model weights, or logs directories; compounding across many moderately sized files that individually seem fine.

Related errors

AI-assisted analysis of vxcontrol/pentagi@ea665308ba (2026-09-01). Data as JSON: /api/errors/d4913b399d65debb. Report an issue: GitHub.