{"record":{"id":"78b3565e4d27b63e","repo":"thanos-io/thanos","slug":"sync-before-first-pass-of-downsampling-78b356","errorCode":null,"errorMessage":"sync before first pass of downsampling","messagePattern":"sync before first pass of downsampling","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/downsample.go","lineNumber":131,"sourceCode":"\tstatusProber := prober.Combine(\n\t\thttpProbe,\n\t\tprober.NewInstrumentation(comp, logger, extprom.WrapRegistererWithPrefix(\"thanos_\", reg)),\n\t)\n\n\tmetrics := newDownsampleMetrics(reg)\n\t// Start cycle of syncing blocks from the bucket and garbage collecting the bucket.\n\t{\n\t\tctx, cancel := context.WithCancel(context.Background())\n\n\t\tg.Add(func() error {\n\t\t\tdefer runutil.CloseWithLogOnErr(logger, insBkt, \"bucket client\")\n\t\t\tstatusProber.Ready()\n\n\t\t\treturn runutil.Repeat(waitInterval, ctx.Done(), func() error {\n\t\t\t\tlevel.Info(logger).Log(\"msg\", \"start first pass of downsampling\")\n\t\t\t\tmetas, _, err := metaFetcher.Fetch(ctx)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn errors.Wrap(err, \"sync before first pass of downsampling\")\n\t\t\t\t}\n\n\t\t\t\tfor _, meta := range metas {\n\t\t\t\t\tresolutionLabel := meta.Thanos.ResolutionString()\n\t\t\t\t\tmetrics.downsamples.WithLabelValues(resolutionLabel)\n\t\t\t\t\tmetrics.downsampleFailures.WithLabelValues(resolutionLabel)\n\t\t\t\t}\n\t\t\t\tif err := downsampleBucket(ctx, logger, metrics, insBkt, metas, dataDir, downsampleConcurrency, blockFilesConcurrency, hashFunc, false); err != nil {\n\t\t\t\t\treturn errors.Wrap(err, \"downsampling failed\")\n\t\t\t\t}\n\n\t\t\t\tlevel.Info(logger).Log(\"msg\", \"start second pass of downsampling\")\n\t\t\t\tmetas, _, err = metaFetcher.Fetch(ctx)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn errors.Wrap(err, \"sync before second pass of downsampling\")\n\t\t\t\t}\n\t\t\t\tif err := downsampleBucket(ctx, logger, metrics, insBkt, metas, dataDir, downsampleConcurrency, blockFilesConcurrency, hashFunc, false); err != nil {\n\t\t\t\t\treturn errors.Wrap(err, \"downsampling failed\")","sourceCodeStart":113,"sourceCodeEnd":149,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/downsample.go#L113-L149","documentation":"In the first downsampling loop, RunDownsample repeatedly calls metaFetcher.Fetch(ctx) to sync block metadata from the bucket. Any Fetch failure (context canceled, network/object-store error, malformed meta.json) is wrapped as \"sync before first pass of downsampling\" and retried after waitInterval by runutil.Repeat.","triggerScenarios":"metaFetcher.Fetch returns an error because the object store is unreachable, a block's meta.json is corrupt/invalid, listing fails, or the context was canceled during shutdown.","commonSituations":"S3/GCS credentials expired mid-run; rate limiting (SlowDown) from the provider; partially uploaded/corrupt meta.json in the bucket; operator Ctrl-C (context canceled) surfacing as this wrap.","solutions":["Inspect the wrapped cause: context canceled means shutdown — just exit; object-store errors need credential/endpoint checks.","Check bucket health and permissions; retry after provider rate limits clear.","Find and re-upload or delete the corrupt block meta.json reported in the error chain.","Verify network egress/firewall allows the object-store endpoint."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Go: preflight a cheap listing before starting the long job\nif err := bkt.Iter(ctx, \"\", func(string) error { return nil }); err != nil {\n    return err\n}","typeGuard":null,"tryCatchPattern":"// Go\ncerr := errors.Cause(err)\nif cerr == context.Canceled { return nil } // normal shutdown\n// otherwise rely on runutil.Repeat backoff or alert on repeated failures","preventionTips":["Use long-lived, auto-renewing credentials for the object store.","Monitor provider rate limits (S3 SlowDown) and set request-concurrency flags accordingly.","Guard against corrupt meta.json uploads by validating before upload in your compactor workflow.","Expect context-canceled entries during graceful shutdowns; filter them from alerts."],"tags":["object-storage","downsample","network","blocks"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}