What causes "write split file for shard %q: %w" in weaviate/weaviate?

Inspect the wrapped inner error to see whether it is a destination-backend or compression failure.

write split file for shard %q: %w

Error message

write split file for shard %q: %w

What it means

Wraps a failure from zip.WriteSplitFile when a shard's data exceeds the max chunk size and must be written as a split-file part. If writing the split-file chunk to the destination (or compressing it) fails, the shard upload aborts with this error naming the shard.

Source

Thrown at usecases/backup/backend.go:716

			// Close writes tar/gzip trailers and could fail if the pipe is closed.
			// Use CloseWithError to signal any producer error to the consumer,
			// so the consumer's read fails instead of seeing EOF.
			closeErr := zip.CloseWithError(err)
			err = errors.Join(err, labelErr("close", closeErr))
		}()

		if err := ctx.Err(); err != nil {
			return nil, err
		}

		var fileSizeExceededInfo *SplitFile
		if fileSizeExceededWrite != nil {
			// Only write the split file part in this chunk; remaining space is intentionally
			// left unused to keep the logic simple and avoid mixing split file parts with
			// regular files in the same chunk.
			fileSizeExceededInfo, err = zip.WriteSplitFile(ctx, shard, fileSizeExceededWrite, &preCompressionSize, chunkKey)
			if err != nil {
				return nil, fmt.Errorf("write split file for shard %q: %w", shard.Name, err)
			}
		} else {
			_, fileSizeExceededInfo, err = zip.WriteShard(ctx, shard, filesInShard, firstChunkForShard, &preCompressionSize, chunkKey)
			if err != nil {
				return nil, fmt.Errorf("write files for shard %q: %w", shard.Name, err)
			}
		}
		shard.ClearTemporary()

		if zip.compressorWriter != nil {
			if err := zip.compressorWriter.Flush(); err != nil {
				return nil, fmt.Errorf("flush compressor: %w", err)
			}
		}

		return fileSizeExceededInfo, nil
	}

View on GitHub (pinned to 75aa4b6d11)

Solutions

Inspect the wrapped inner error to see whether it is a destination-backend or compression failure.
If chunk size is set very low, increase the backup chunk size limit so fewer shards need splitting.
Verify write permissions, quota, and connectivity to the backup destination (bucket policy, credentials).
Retry the backup; transient destination errors are common with large split writes.

Defensive patterns

Strategy: retry

Validate before calling

// preflight: destination is writable and quota OK
if err := probeDestinationWrite(bucket); err != nil {
    return fmt.Errorf("destination not writable: %w", err)
}

Try / catch

if err := runBackup(ctx); err != nil {
    if strings.Contains(err.Error(), "write split file for shard") {
        // increase chunk size or fix destination, then retry
        err = runBackupWithLargerChunks(ctx)
    }
}

Prevention

Set a chunk size large enough that most shards avoid split-file paths.
Verify destination credentials, quota, and permissions before large backups.
Retry large-shard backups on transient destination errors.

When it happens

Trigger: Backing up a shard whose files exceed the per-chunk size limit so the code takes the fileSizeExceededWrite branch, and WriteSplitFile returns an error — destination write failure, compression error, or destination backend error.

Common situations: Very large shards combined with a misconfigured or full backup destination (S3 bucket permissions, quota, network drops); an overly small backup chunk size setting forcing split files for most shards.

Related errors

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/47495effb5cc8fbd. Report an issue: GitHub.