What causes "minimum S3 part size must be between 5MiB and 5GiB" in vitessio/vitess?

Clamp the computed part size to at least 5MiB (5*1024*1024) before starting the multipart upload.

ErrPartSize

ErrPartSize

Error message

minimum S3 part size must be between 5MiB and 5GiB

What it means

ErrPartSize is thrown by calculateUploadPartSize when the derived S3 multipart upload part size falls outside the AWS-mandated bounds: each part must be at least 5MiB and at most 5GiB. S3 multipart uploads reject parts outside this range, so the backup handle refuses to start with an invalid part size rather than failing mid-upload.

Source

Thrown at go/vt/mysqlctl/s3backupstorage/s3.go:101

	// forcePath is used to ensure that the certificate and path used match the endpoint + region
	forcePath bool

	tlsSkipVerifyCert bool

	// verboseLogging provides more verbose logging of AWS actions
	requiredLogLevel string

	// sse is the server-side encryption algorithm used when storing this object in S3
	sse string

	// path component delimiter
	delimiter = "/"

	// minimum part size
	minPartSize int64 = 5 * 1024 * 1024 // 5MiB - AWS requirement

	ErrPartSize = errors.New("minimum S3 part size must be between 5MiB and 5GiB")
)

func registerFlags(fs *pflag.FlagSet) {
	utils.SetFlagStringVar(fs, &region, "s3-backup-aws-region", "us-east-1", "AWS region to use.")
	utils.SetFlagIntVar(fs, &retryCount, "s3-backup-aws-retries", -1, "AWS request retries.")
	utils.SetFlagStringVar(fs, &endpoint, "s3-backup-aws-endpoint", "", "endpoint of the S3 backend (region must be provided).")
	utils.SetFlagStringVar(fs, &bucket, "s3-backup-storage-bucket", "", "S3 bucket to use for backups.")
	utils.SetFlagStringVar(fs, &root, "s3-backup-storage-root", "", "root prefix for all backup-related object names.")
	utils.SetFlagBoolVar(fs, &forcePath, "s3-backup-force-path-style", false, "force the s3 path style.")
	utils.SetFlagBoolVar(fs, &tlsSkipVerifyCert, "s3-backup-tls-skip-verify-cert", false, "skip the 'certificate is valid' check for SSL connections.")
	utils.SetFlagStringVar(fs, &requiredLogLevel, "s3-backup-log-level", "LogOff", "determine the S3 loglevel to use from LogOff, LogDebug, LogDebugWithSigning, LogDebugWithHTTPBody, LogDebugWithRequestRetries, LogDebugWithRequestErrors.")
	utils.SetFlagStringVar(fs, &sse, "s3-backup-server-side-encryption", "", "server-side encryption algorithm (e.g., AES256, aws:kms, sse_c:/path/to/key/file).")
	utils.SetFlagInt64Var(fs, &minPartSize, "s3-backup-aws-min-partsize", minPartSize, "Minimum part size to use, defaults to 5MiB but can be increased due to the dataset size.")
}

func init() {
	servenv.OnParseFor("vtbackup", registerFlags)
	servenv.OnParseFor("vtctl", registerFlags)

View on GitHub (pinned to 01a25a7d17)

Solutions

Clamp the computed part size to at least 5MiB (5*1024*1024) before starting the multipart upload.
Verify the backup file size passed to AddFile is sensible; tiny test backups may not support multipart semantics.
If using custom part-size configuration, ensure the resulting size stays within [5MiB, 5GiB].

Example fix

// before
partSize := fileSize / maxParts
bh.AddFile(ctx, filename, fileSize)

// after
const minPartSize = 5 * 1024 * 1024
const maxPartSize = 5 * 1024 * 1024 * 1024
if partSize < minPartSize {
    partSize = minPartSize
}
if partSize > maxPartSize {
    partSize = maxPartSize
}

Defensive patterns

Strategy: validation

Validate before calling

const minPart = int64(5 * 1024 * 1024)     // 5MiB
const maxPart = int64(5 * 1024 * 1024 * 1024) // 5GiB
if partSize < minPart || partSize > maxPart {
    return fmt.Errorf("part size %d outside [%d, %d]", partSize, minPart, maxPart)
}

Try / catch

wc, err := bh.AddFile(ctx, filename, fileSize)
if err != nil {
    if strings.Contains(err.Error(), "part size") {
        return vterrors.Errorf(vtrpcpb.Code_INVALID_ARGUMENT, "backup size yields invalid S3 part size: %v", err)
    }
    return err
}

Prevention

Clamp computed part sizes to [5MiB, 5GiB] before initiating multipart uploads.
Test the backup pipeline with both tiny and multi-TB backup sizes to exercise both bounds.
Rely on calculateUploadPartSize instead of custom part-size math.

When it happens

Trigger: Calling S3BackupHandle.AddFile (which calls calculateUploadPartSize) with a filesize whose computed part size is below 5MiB or above 5GiB — e.g. an extremely small backup where the intended part size would undercut the AWS 5MiB minimum, or a part count/part size division yielding out-of-range values.

Common situations: Backing up tiny databases where a computed part size dips under the AWS minimum; custom part-size math against huge backups exceeding limits; misconfigured backup flags that influence part sizing.

Related errors

AI-assisted analysis of vitessio/vitess@01a25a7d17 (2026-09-01). Data as JSON: /api/errors/631120bb7e6285c0. Report an issue: GitHub.