{"record":{"id":"d4913b399d65debb","repo":"vxcontrol/pentagi","slug":"tar-archive-exceeds-maximum-total-size-of-d-bytes","errorCode":null,"errorMessage":"tar archive exceeds maximum total size of %d bytes","messagePattern":"tar archive exceeds maximum total size of (.+?) bytes","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"backend/pkg/flowfiles/files.go","lineNumber":411,"sourceCode":"\t\t\tcontinue\n\t\t}\n\n\t\tswitch hdr.Typeflag {\n\t\tcase tar.TypeDir:\n\t\t\tif err := os.MkdirAll(entryPath, 0755); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to create directory '%s': %w\", entryPath, err)\n\t\t\t}\n\t\tcase tar.TypeReg, tar.TypeRegA:\n\t\t\tif hdr.Size < 0 {\n\t\t\t\treturn fmt.Errorf(\"tar entry '%s' has invalid size %d\", hdr.Name, hdr.Size)\n\t\t\t}\n\t\t\tfilesCount++\n\t\t\tif filesCount > MaxPullFiles {\n\t\t\t\treturn fmt.Errorf(\"tar archive exceeds maximum file count of %d\", MaxPullFiles)\n\t\t\t}\n\t\t\ttotalSize += hdr.Size\n\t\t\tif totalSize > MaxPullTotalSize {\n\t\t\t\treturn fmt.Errorf(\"tar archive exceeds maximum total size of %d bytes\", MaxPullTotalSize)\n\t\t\t}\n\n\t\t\tif err := os.MkdirAll(filepath.Dir(entryPath), 0755); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to create parent directory for '%s': %w\", entryPath, err)\n\t\t\t}\n\n\t\t\tf, err := os.OpenFile(entryPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0644)\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to create file '%s': %w\", entryPath, err)\n\t\t\t}\n\t\t\t_, copyErr := io.CopyN(f, tr, hdr.Size)\n\t\t\tf.Close()\n\t\t\tif copyErr != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to write file '%s': %w\", entryPath, copyErr)\n\t\t\t}\n\t\t}\n\t}\n","sourceCodeStart":393,"sourceCodeEnd":429,"githubUrl":"https://github.com/vxcontrol/pentagi/blob/ea665308baaff015b226f308438a68d929d0f29b/backend/pkg/flowfiles/files.go#L393-L429","documentation":"ExtractTar enforces MaxPullTotalSize (2 GiB) as the running sum of regular-file entry sizes; exceeding it aborts with this error. Like the file-count limit, it is an anti-DoS guard against zip-bomb-style archives that would exhaust disk.","triggerScenarios":"Pulling an archive whose declared regular-file sizes total more than 2 GiB via PullFlowFiles/ExtractTar; the check happens before any parent directory or file is written for the offending entry.","commonSituations":"Archiving large datasets, VM images, model weights, or logs directories; compounding across many moderately sized files that individually seem fine.","solutions":["Compress the payload (tar.gz of one file) so the total fits, or split into multiple archives under 2 GiB each.","Exclude bulky non-essential content (images, videos, build artifacts) from the tar on the producer side.","Free up expectations: if legitimate data must exceed 2 GiB, raise MaxPullTotalSize in backend/pkg/flowfiles/files.go and ensure the destination volume has headroom.","Transfer large artifacts out-of-band (object storage URL) instead of through the flow-files path."],"exampleFix":"// before\ntar -cf big.tar ./data/\n// after\ntar -czf big.tar.gz ./data/  # and split: split -b 1900M big.tar.gz big.tar.gz.part-","handlingStrategy":"validation","validationCode":"// Check declared total size without extracting:\n// tar -tvf bundle.tar | awk '{s+=$3} END {print s}'\n// if s > 2147483648, compress or split before ExtractTar.","typeGuard":null,"tryCatchPattern":"err := flowfiles.ExtractTar(rc, destDir)\nif err != nil {\n    if strings.Contains(err.Error(), \"exceeds maximum total size\") {\n        return fmt.Errorf(\"archive exceeds %d byte limit; compress or split it\", flowfiles.MaxPullTotalSize)\n    }\n    return err\n}","preventionTips":["Compress large payloads (tar.gz single file) so the declared total stays under 2 GiB.","Split oversized datasets into multiple archives below the limit.","Transfer very large artifacts via object storage links rather than the flow-files channel."],"tags":["tar","limits","dos-protection","security"],"backgroundTag":"archive-size-limit-exceeded","analyzedSha":"ea665308baaff015b226f308438a68d929d0f29b","analyzedAt":"2026-09-01T14:16:31.421Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}