weaviate/weaviate · error

read chunk %s from backend: %w

Error message

read chunk %s from backend: %w

What it means

Wrap from readAndUnzipChunk (usecases/backup/backend.go:1027) when the goroutine downloading a chunk from the backup backend (fw.backend.Read / ReadFromOtherBackup) returned an error other than io.ErrClosedPipe. io.ErrClosedPipe is deliberately ignored because it happens when the unzip side closes the pipe early. Any other error means the chunk could not be fully fetched from storage, so the restore fails.

Source

Thrown at usecases/backup/backend.go:1027

		err = readFn(w)
		if err != nil {
			fw.logger.WithField("chunk", chunkName).Errorf("failed to read chunk from backend: %v", err)
		}
	}, fw.logger)

	_, unzipErr := uz.ReadChunk()
	// Close the pipe reader so any in-progress pw.Write() in readFn unblocks
	// with ErrClosedPipe. Without this, readFn can hang forever if the
	// decompressor detected end-of-stream before io.Copy finished writing all
	// bytes from the backend.
	uz.Close()
	// Always drain readErrCh to prevent leaking the readFn goroutine.
	readErr := <-readErrCh
	if unzipErr != nil {
		return fmt.Errorf("unzip chunk %s: %w", chunkName, unzipErr)
	}
	if readErr != nil && !errors.Is(readErr, io.ErrClosedPipe) {
		return fmt.Errorf("read chunk %s from backend: %w", chunkName, readErr)
	}
	return nil
}

func chunkKey(class string, id int32) string {
	return fmt.Sprintf("%s/chunk-%d", class, id)
}

func routinePoolSize(percentage int) int {
	if percentage == 0 { // default value
		percentage = DefaultCPUPercentage
	} else if percentage > maxCPUPercentage {
		percentage = maxCPUPercentage
	}
	if x := (numCPU() * percentage) / 100; x > 0 {
		return x
	}
	return 1

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Read the wrapped inner error: if it's 'object not found', verify the chunk exists in the bucket and that lifecycle policies are not deleting backup objects.
2. Verify backend credentials/IAM permissions (S3 access keys, GCS service account, Azure storage key) for the configured backup backend.
3. For incremental backups, ensure every base backup in the chain (FilesPerBackup) still exists — do not delete older backups that newer ones depend on.
4. Retry the restore after confirming network connectivity and storage rate limits; transient timeouts often resolve on retry.

Example fix

// before: incremental base backup deleted; restore fails with 'read chunk ...: object not found'
// after: protect backups from lifecycle deletion (S3)
// { "Rules": [{ "Status": "Enabled", "Filter": {"Prefix": "backups/"}, "Expiration": {"Days": 365} }] }

// before restore: confirm every referenced chunk (including incremental bases) exists
for _, key := range allChunkKeys(backupMeta) {
    if !objectExists(backend, key) { // e.g. s3api head-object
        return fmt.Errorf("chunk %s missing from backend; restore would fail", key)
    }
}

func isReadChunkFailure(err error) bool {
    return strings.Contains(err.Error(), "read chunk ") && strings.Contains(err.Error(), "from backend:")
}

err := client.Backup().Restore(ctx, backend, backupID, cfg)
var retriable = []string{"timeout", "connection reset", "rate", "503"}
if err != nil && isReadChunkFailure(err) {
    for _, s := range retriable {
        if strings.Contains(err.Error(), s) { backoffAndRetry(); return }
    }
    log.Printf("non-retriable backend read failure, check credentials/bucket contents: %v", err)
}

When it happens

Trigger: Restoring a class when a chunk object is missing (object not found / 404), backend credentials are invalid, rate limiting or network timeouts hit the object store, or the referenced incremental base backup no longer exists.

Common situations: Bucket contents deleted or expired (S3 lifecycle rules) between backup and restore; wrong AWS/GCS/Azure credentials or IAM permissions; bandwidth throttling or transient network failure to the storage endpoint; incremental backup chain broken because an older backup was deleted.

Related errors

• get files: %w
• unzip chunk %s: %w
• can-commit request: %w
• abort request: %w
• get remote object: shard=%s: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/798126d6f826ef9d. Report an issue: GitHub.