{"record":{"id":"f3f6125fa5d22d6d","repo":"thanos-io/thanos","slug":"could-not-sync-metas","errorCode":null,"errorMessage":"could not sync metas","messagePattern":"could not sync metas","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/compact.go","lineNumber":660,"sourceCode":"\t\t\t\trs := compact.NewRetentionProgressCalculator(reg, retentionByResolution)\n\t\t\t\tvar ds *compact.DownsampleProgressCalculator\n\t\t\t\tif !conf.disableDownsampling {\n\t\t\t\t\tds = compact.NewDownsampleProgressCalculator(reg)\n\t\t\t\t}\n\n\t\t\t\treturn runutil.Repeat(conf.progressCalculateInterval, ctx.Done(), func() error {\n\n\t\t\t\t\tif err := sy.SyncMetas(ctx); err != nil {\n\t\t\t\t\t\t// The RetryError signals that we hit an retriable error (transient error, no connection).\n\t\t\t\t\t\t// You should alert on this being triggered too frequently.\n\t\t\t\t\t\tif compact.IsRetryError(err) {\n\t\t\t\t\t\t\tlevel.Error(logger).Log(\"msg\", \"retriable error\", \"err\", err)\n\t\t\t\t\t\t\tcompactMetrics.retried.Inc()\n\n\t\t\t\t\t\t\treturn nil\n\t\t\t\t\t\t}\n\n\t\t\t\t\t\treturn errors.Wrapf(err, \"could not sync metas\")\n\t\t\t\t\t}\n\n\t\t\t\t\tmetas := sy.Metas()\n\t\t\t\t\tgroups, err := grouper.Groups(metas)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\treturn errors.Wrapf(err, \"could not group metadata for compaction\")\n\t\t\t\t\t}\n\n\t\t\t\t\tif err = ps.ProgressCalculate(ctx, groups); err != nil {\n\t\t\t\t\t\treturn errors.Wrapf(err, \"could not calculate compaction progress\")\n\t\t\t\t\t}\n\n\t\t\t\t\tretGroups, err := grouper.Groups(metas)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\treturn errors.Wrapf(err, \"could not group metadata for retention\")\n\t\t\t\t\t}\n\n\t\t\t\t\tif err = rs.ProgressCalculate(ctx, retGroups); err != nil {","sourceCodeStart":642,"sourceCodeEnd":678,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/compact.go#L642-L678","documentation":"The periodic compaction-progress worker calls sy.SyncMetas(ctx) to fetch block metadata from the bucket via the MetaFetcher. If syncing fails with an error that is not a RetryError, the iteration is aborted with 'could not sync metas'. It means the compactor could not obtain a consistent view of the blocks in the bucket.","triggerScenarios":"sy.SyncMetas(ctx) returns a non-retryable error inside the progressCalculateInterval repeat loop — e.g. bucket ListObjects consistently failing, unparseable meta.json in a block, or an object-store API error not wrapped as retry.","commonSituations":"Corrupted or hand-edited meta.json in the bucket causing fetch errors, expired object-store credentials, or an S3/GCS outage that returns errors the fetcher does not classify as retriable.","solutions":["Inspect the wrapped inner error; if it points at a specific block's meta.json, validate/repair or remove that block.","Test bucket access with 'thanos tools bucket ls' to separate auth/network issues from metadata issues.","Retry the process; if the error is transient and misclassified, upgrade Thanos so SyncMetas errors are retried.","Alert on frequent occurrences — the whole progress-calculation loop is skipped, so metrics will go stale."],"exampleFix":"// before\n// corrupted meta.json causes sync failure every iteration\n// after\nthanos tools bucket inspect --objstore.bucket=thanos\n# or delete/repair the offending block's meta.json, then restart","handlingStrategy":"retry","validationCode":"// Preflight metadata listing before starting the progress worker\nmetas, _, err := fetcher.Fetch(ctx)\nif err != nil {\n    log.Printf(\"metadata preflight failed: %v\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := sy.SyncMetas(ctx); err != nil {\n    if compact.IsRetryError(err) {\n        return nil // retry next tick\n    }\n    log.Errorf(\"could not sync metas: %v\", err) // alert; inspect inner block errors\n    return nil\n}","preventionTips":["Alert on repeated sync failures — progress metrics silently go stale.","Never hand-edit meta.json files in the bucket; use bucket tools.","Validate credentials rotation so they don't expire mid-run.","Use 'thanos tools bucket ls' in CI/ops checks to validate bucket health."],"tags":["thanos","metadata","object-storage"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}