{"record":{"id":"798126d6f826ef9d","repo":"weaviate/weaviate","slug":"read-chunk-s-from-backend-w","errorCode":null,"errorMessage":"read chunk %s from backend: %w","messagePattern":"read chunk (.+?) from backend: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"usecases/backup/backend.go","lineNumber":1027,"sourceCode":"\t\terr = readFn(w)\n\t\tif err != nil {\n\t\t\tfw.logger.WithField(\"chunk\", chunkName).Errorf(\"failed to read chunk from backend: %v\", err)\n\t\t}\n\t}, fw.logger)\n\n\t_, unzipErr := uz.ReadChunk()\n\t// Close the pipe reader so any in-progress pw.Write() in readFn unblocks\n\t// with ErrClosedPipe. Without this, readFn can hang forever if the\n\t// decompressor detected end-of-stream before io.Copy finished writing all\n\t// bytes from the backend.\n\tuz.Close()\n\t// Always drain readErrCh to prevent leaking the readFn goroutine.\n\treadErr := <-readErrCh\n\tif unzipErr != nil {\n\t\treturn fmt.Errorf(\"unzip chunk %s: %w\", chunkName, unzipErr)\n\t}\n\tif readErr != nil && !errors.Is(readErr, io.ErrClosedPipe) {\n\t\treturn fmt.Errorf(\"read chunk %s from backend: %w\", chunkName, readErr)\n\t}\n\treturn nil\n}\n\nfunc chunkKey(class string, id int32) string {\n\treturn fmt.Sprintf(\"%s/chunk-%d\", class, id)\n}\n\nfunc routinePoolSize(percentage int) int {\n\tif percentage == 0 { // default value\n\t\tpercentage = DefaultCPUPercentage\n\t} else if percentage > maxCPUPercentage {\n\t\tpercentage = maxCPUPercentage\n\t}\n\tif x := (numCPU() * percentage) / 100; x > 0 {\n\t\treturn x\n\t}\n\treturn 1","sourceCodeStart":1009,"sourceCodeEnd":1045,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/usecases/backup/backend.go#L1009-L1045","documentation":"Wrap from readAndUnzipChunk (usecases/backup/backend.go:1027) when the goroutine downloading a chunk from the backup backend (fw.backend.Read / ReadFromOtherBackup) returned an error other than io.ErrClosedPipe. io.ErrClosedPipe is deliberately ignored because it happens when the unzip side closes the pipe early. Any other error means the chunk could not be fully fetched from storage, so the restore fails.","triggerScenarios":"Restoring a class when a chunk object is missing (object not found / 404), backend credentials are invalid, rate limiting or network timeouts hit the object store, or the referenced incremental base backup no longer exists.","commonSituations":"Bucket contents deleted or expired (S3 lifecycle rules) between backup and restore; wrong AWS/GCS/Azure credentials or IAM permissions; bandwidth throttling or transient network failure to the storage endpoint; incremental backup chain broken because an older backup was deleted.","solutions":["Read the wrapped inner error: if it's 'object not found', verify the chunk exists in the bucket and that lifecycle policies are not deleting backup objects.","Verify backend credentials/IAM permissions (S3 access keys, GCS service account, Azure storage key) for the configured backup backend.","For incremental backups, ensure every base backup in the chain (FilesPerBackup) still exists — do not delete older backups that newer ones depend on.","Retry the restore after confirming network connectivity and storage rate limits; transient timeouts often resolve on retry."],"exampleFix":"// before: incremental base backup deleted; restore fails with 'read chunk ...: object not found'\n// after: protect backups from lifecycle deletion (S3)\n// { \"Rules\": [{ \"Status\": \"Enabled\", \"Filter\": {\"Prefix\": \"backups/\"}, \"Expiration\": {\"Days\": 365} }] }","handlingStrategy":"retry","validationCode":"// before restore: confirm every referenced chunk (including incremental bases) exists\nfor _, key := range allChunkKeys(backupMeta) {\n    if !objectExists(backend, key) { // e.g. s3api head-object\n        return fmt.Errorf(\"chunk %s missing from backend; restore would fail\", key)\n    }\n}","typeGuard":"func isReadChunkFailure(err error) bool {\n    return strings.Contains(err.Error(), \"read chunk \") && strings.Contains(err.Error(), \"from backend:\")\n}","tryCatchPattern":"err := client.Backup().Restore(ctx, backend, backupID, cfg)\nvar retriable = []string{\"timeout\", \"connection reset\", \"rate\", \"503\"}\nif err != nil && isReadChunkFailure(err) {\n    for _, s := range retriable {\n        if strings.Contains(err.Error(), s) { backoffAndRetry(); return }\n    }\n    log.Printf(\"non-retriable backend read failure, check credentials/bucket contents: %v\", err)\n}","preventionTips":["Protect backup buckets from lifecycle deletion and manual cleanup.","Never delete incremental base backups that newer backups depend on.","Validate backend credentials/IAM with a pre-restore connectivity check.","Configure timeouts/retries on the object-storage client and restore over stable networks."],"tags":["backup","object-storage","network","go"],"backgroundTag":"object-not-found","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}