{"record":{"id":"74ece8f3a807236e","repo":"thanos-io/thanos","slug":"sync-blocks","errorCode":null,"errorMessage":"sync blocks","messagePattern":"sync blocks","errorType":"console","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"cmd/thanos/tools_bucket.go","lineNumber":903,"sourceCode":"\t\t\tsy, err = compact.NewMetaSyncer(\n\t\t\t\tlogger,\n\t\t\t\treg,\n\t\t\t\tinsBkt,\n\t\t\t\tcf,\n\t\t\t\tduplicateBlocksFilter,\n\t\t\t\tignoreDeletionMarkFilter,\n\t\t\t\tstubCounter,\n\t\t\t\tstubCounter,\n\t\t\t\t0,\n\t\t\t)\n\t\t\tif err != nil {\n\t\t\t\treturn errors.Wrap(err, \"create syncer\")\n\t\t\t}\n\t\t}\n\n\t\tlevel.Info(logger).Log(\"msg\", \"syncing blocks metadata\")\n\t\tif err := sy.SyncMetas(ctx); err != nil {\n\t\t\treturn errors.Wrap(err, \"sync blocks\")\n\t\t}\n\n\t\tlevel.Info(logger).Log(\"msg\", \"synced blocks done\")\n\n\t\tcompact.BestEffortCleanAbortedPartialUploads(ctx, logger, sy.Partial(), insBkt, stubCounter, stubCounter, stubCounter, ignoreDeletionMarkFilter.DeletionMarkBlocks())\n\t\tif _, err := blocksCleaner.DeleteMarkedBlocks(ctx); err != nil {\n\t\t\treturn errors.Wrap(err, \"error cleaning blocks\")\n\t\t}\n\n\t\tlevel.Info(logger).Log(\"msg\", \"cleanup done\")\n\t\treturn nil\n\t})\n}\n\ntype tablePrinter func(w io.Writer, t Table) error\n\nfunc printTable(w io.Writer, t Table) error {\n\ttable := tablewriter.NewWriter(w)","sourceCodeStart":885,"sourceCodeEnd":921,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/tools_bucket.go#L885-L921","documentation":"The command calls sy.SyncMetas(ctx) to download and load metadata for all blocks in the bucket. Any failure during this phase (list failures, meta.json download/unmarshal errors, context cancellation) is wrapped as 'sync blocks'. This means the bucket view of block metadata could not be fully built.","triggerScenarios":"Running compact/replicate when SyncMetas fails at cmd/thanos/tools_bucket.go:903 — ListObjects errors, corrupted meta.json in the bucket, fetching errors on individual blocks, or the context being cancelled (Ctrl-C / timeout) mid-sync.","commonSituations":"Flaky object-store connectivity, expired cloud credentials mid-run, partially uploaded/corrupted meta.json from an interrupted upload, or timeouts on very large buckets exceeding the command deadline.","solutions":["Re-run the command; sync is idempotent and transient network/credential failures often resolve.","Check the wrapped inner error: if it mentions a specific block, inspect/repair that block's meta.json or delete the partial upload.","Verify credentials/session expiry if the run is long; refresh tokens or use instance profiles.","Increase timeouts / ensure the context isn't cancelled (don't interrupt during 'syncing blocks metadata').","Use `thanos tools bucket verify` to find corrupted blocks, and `thanos tools bucket mark --mark=deletion-mark` + cleanup for unrecoverable ones."],"exampleFix":"// before: fixing a corrupted block meta found by the inner error\n// (bucket: thanos-blocks, block 01ARZ.../meta.json is truncated)\nthanos tools bucket mark --objstore.config-file=b.yml --mark=deletion-mark --id=01ARZ3NDEKTSV4RRFFQ69G5FAV --details=\"corrupted meta.json\"\n// after\nthanos tools bucket compact --objstore.config-file=b.yml  # re-run sync after marking the bad block","handlingStrategy":"retry","validationCode":"// Pre-flight: ensure credentials are fresh and bucket is reachable\nif err := checkBucketListable(ctx, bkt); err != nil {\n    return err // fail fast before SyncMetas\n}","typeGuard":null,"tryCatchPattern":"if err := sy.SyncMetas(ctx); err != nil {\n    if ctx.Err() != nil {\n        return fmt.Errorf(\"sync cancelled: %w\", ctx.Err())\n    }\n    if isRetryable(errors.Unwrap(err)) {\n        return retryWithBackoff(func() error { return sy.SyncMetas(ctx) })\n    }\n    return errors.Wrap(err, \"sync blocks\")\n}","preventionTips":["Use long-lived credentials or auto-refreshing instance profiles for long runs","Don't interrupt the command during 'syncing blocks metadata'","Schedule compaction off-peak to reduce transient storage errors","Run `thanos tools bucket verify` periodically to catch corrupted meta.json early","Increase consistency delay/timeout settings for very large buckets"],"tags":["object-storage","block-metadata","sync"],"backgroundTag":"api-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}