{"record":{"id":"246e446ec873350a","repo":"thanos-io/thanos","slug":"failed-to-sync-v-v-blocks","errorCode":null,"errorMessage":"failed to sync %v/%v blocks","messagePattern":"failed to sync (.+?)/(.+?) blocks","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/shipper/shipper.go","lineNumber":450,"sourceCode":"\t\t\t// No error returned, just log line. This is because we want other blocks to be uploaded even\n\t\t\t// though this one failed. It will be retried on second Sync iteration.\n\t\t\tlevel.Error(s.logger).Log(\"msg\", \"shipping failed\", \"block\", m.ULID, \"err\", err)\n\t\t\tuploadErrs++\n\t\t\tcontinue\n\t\t}\n\t\tmeta.Uploaded = append(meta.Uploaded, m.ULID)\n\t\tuploaded++\n\t\ts.metrics.uploads.Inc()\n\t}\n\tif err := WriteMetaFile(s.logger, s.metadataFilePath, meta); err != nil {\n\t\tlevel.Warn(s.logger).Log(\"msg\", \"updating meta file failed\", \"err\", err)\n\t}\n\n\tfailedExecution = false\n\tif uploadErrs > 0 || len(failedBlocks) > 0 {\n\t\ts.metrics.uploadFailures.Add(float64(uploadErrs))\n\t\ts.metrics.corruptedBlocks.Add(float64(len(failedBlocks)))\n\t\treturn uploaded, errors.Errorf(\"failed to sync %v/%v blocks\", uploadErrs, len(failedBlocks))\n\t}\n\n\tif s.uploadCompacted {\n\t\ts.metrics.uploadedCompacted.Set(1)\n\t} else {\n\t\ts.metrics.uploadedCompacted.Set(0)\n\t}\n\treturn uploaded, nil\n}\n\nfunc (s *Shipper) UploadedBlocks() map[ulid.ULID]struct{} {\n\tmeta, err := ReadMetaFile(s.metadataFilePath)\n\tif err != nil {\n\t\t// NOTE(GiedriusS): Sync() will inform users about any problems.\n\t\treturn nil\n\t}\n\n\tret := make(map[ulid.ULID]struct{}, len(meta.Uploaded))","sourceCodeStart":432,"sourceCodeEnd":468,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/shipper/shipper.go#L432-L468","documentation":"Summary error returned at the end of Sync when at least one upload failed or a corrupted block was detected (uploadErrs > 0 or failedBlocks > 0) and out-of-order uploads are enabled (errors were logged instead of returned immediately). Note the count formatting shows uploadErrs as the first number and len(failedBlocks) as the denominator, which can be misleading; consult the per-block 'shipping failed' log lines for the actual failing ULIDs.","triggerScenarios":"Sync completes its loop with allowOutOfOrderUploads=true and one or more s.upload() calls failed, or blocks were detected as corrupted (missing/invalid segment files) during iteration.","commonSituations":"Flaky object storage connectivity during a sync cycle; corrupted blocks on disk (e.g. after unclean shutdown or disk errors); persistent upload failures retried every sync interval.","solutions":["Search sidecar logs for 'shipping failed' and 'corrupted block' entries to find the exact block ULIDs.","For corrupted blocks, verify the block in the bucket and delete/restore the local copy (Prometheus will re-compact).","For transient storage errors, wait for the next sync cycle — failed uploads are automatically retried.","If failures persist, inspect the object store health and increase timeouts/retries in the objstore config."],"exampleFix":"// before\n// failed to sync 2/1 blocks   (misleading counts; check logs)\n// after\n// $ kubectl logs thanos-sidecar | grep 'shipping failed'\n// # identify ULID, then: thanos tools bucket verify --id=<ULID> and repair local data","handlingStrategy":"retry","validationCode":"// before relying on the summary, capture per-block results\n_, err := shipper.Sync(ctx)\nif err != nil && strings.Contains(err.Error(), \"failed to sync\") {\n    log.Printf(\"partial sync; check 'shipping failed' log lines for ULIDs\")\n}","typeGuard":null,"tryCatchPattern":"ticker := time.NewTicker(syncInterval)\nfor range ticker.C {\n    if _, err := shipper.Sync(ctx); err != nil {\n        log.Printf(\"sync incomplete, will retry next tick: %v\", err)\n    }\n}","preventionTips":["Treat this error as retryable — Sync is idempotent","Grep logs for 'shipping failed' to get real failing ULIDs (counts in the message are misleading)","Alert on the shipper's uploadFailures/corruptedBlocks metrics rather than log scraping","Fix persistent object-store issues quickly; retries are cheap but unbounded"],"tags":["upload","retry","corruption","sidecar"],"backgroundTag":"upload-retry-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}