{"record":{"id":"29c202e789b82d32","repo":"thanos-io/thanos","slug":"unable-to-sync-blocks","errorCode":null,"errorMessage":"unable to sync blocks","messagePattern":"unable to sync blocks","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/tools_bucket.go","lineNumber":1513,"sourceCode":"\t\t\tbkt,\n\t\t\ttbcDir,\n\t\t\tshipper.WithLogger(logger),\n\t\t\tshipper.WithRegisterer(reg),\n\t\t\tshipper.WithSource(metadata.BucketUploadSource),\n\t\t\tshipper.WithMetaFileName(shipper.DefaultMetaFilename),\n\t\t\tshipper.WithLabels(func() labels.Labels { return lset }),\n\t\t\tshipper.WithUploadCompacted(tbc.uploadCompacted),\n\t\t\tshipper.WithUploadConcurrency(tbc.uploadConcurrency),\n\t\t)\n\n\t\tctx, cancel := context.WithCancel(context.Background())\n\t\tg.Add(func() error {\n\t\t\tdefer runutil.CloseWithLogOnErr(logger, bkt, \"bucket client\")\n\t\t\tdefer runutil.CloseWithLogOnErr(logger, tbcDir, \"tbc directory\")\n\n\t\t\tn, err := s.Sync(ctx)\n\t\t\tif err != nil {\n\t\t\t\treturn errors.Wrap(err, \"unable to sync blocks\")\n\t\t\t}\n\t\t\tlevel.Info(logger).Log(\"msg\", \"synced blocks\", \"uploaded\", n)\n\t\t\treturn nil\n\t\t}, func(error) {\n\t\t\tcancel()\n\t\t})\n\n\t\treturn nil\n\t})\n}\n","sourceCodeStart":1495,"sourceCodeEnd":1524,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/tools_bucket.go#L1495-L1524","documentation":"This error is returned when shipper.Sync(ctx) fails during the upload command's run group. Sync walks the local directory, uploads new blocks to the bucket, and updates the meta; any failure reading local blocks, uploading to object storage, or updating remote metadata surfaces here wrapped as 'unable to sync blocks'. The count of uploaded blocks is only logged on success.","triggerScenarios":"Underlying object-store operations fail (auth rejected, bucket missing, network timeout, rate limiting), a local block is corrupted or partially written (bad meta.json, missing chunks), or the context is cancelled mid-sync.","commonSituations":"Expired or missing cloud credentials (AWS/GCS); bucket name wrong or bucket deleted; network/proxy issues blocking storage endpoints; partially-downloaded or corrupted local TSDB blocks; uploading blocks whose external labels conflict or compaction is in progress concurrently.","solutions":["Inspect the wrapped underlying error: if it mentions object storage auth, refresh credentials (env vars, credential files, instance roles)","Verify the bucket exists and is writable with the configured credentials (thanos tools bucket verify)","Look for corrupted local blocks (invalid meta.json / missing files) in the path and remove or repair them","Check network connectivity/proxy settings to the object-store endpoint and retry; if a concurrent compaction is writing blocks, run the upload against a quiesced directory"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Before running sync, verify bucket reachability and credentials:\n// thanos tools bucket verify --objstore.bucket.config-file=bucket.yaml\n// and locally: find <path> -name meta.json -exec jq -e . {} \\; to catch corrupt blocks","typeGuard":null,"tryCatchPattern":"n, err := s.Sync(ctx)\nif err != nil {\n    if ctx.Err() != nil {\n        return ctx.Err() // cancelled; do not retry\n    }\n    logger.Error(err, \"block sync failed; check object-store credentials/network and local block integrity\")\n    return backoff.Retry(func() error { _, err := s.Sync(ctx); return err }, backoff.NewExponentialBackOff())\n}","preventionTips":["Rotate and test cloud credentials before long-running upload jobs","Monitor object-store API rate limits and error rates during uploads","Never run uploads against a directory undergoing live compaction or ingestion","Keep local block directories intact; avoid partial copies/rsync interruptions","Use `thanos tools bucket verify` to catch bucket permission problems early"],"tags":["go","object-storage","sync","thanos"],"backgroundTag":"api-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}