{"record":{"id":"214ae5011ec013d3","repo":"thanos-io/thanos","slug":"sync-before-second-pass-of-downsampling-214ae5","errorCode":null,"errorMessage":"sync before second pass of downsampling","messagePattern":"sync before second pass of downsampling","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/downsample.go","lineNumber":146,"sourceCode":"\t\t\t\tlevel.Info(logger).Log(\"msg\", \"start first pass of downsampling\")\n\t\t\t\tmetas, _, err := metaFetcher.Fetch(ctx)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn errors.Wrap(err, \"sync before first pass of downsampling\")\n\t\t\t\t}\n\n\t\t\t\tfor _, meta := range metas {\n\t\t\t\t\tresolutionLabel := meta.Thanos.ResolutionString()\n\t\t\t\t\tmetrics.downsamples.WithLabelValues(resolutionLabel)\n\t\t\t\t\tmetrics.downsampleFailures.WithLabelValues(resolutionLabel)\n\t\t\t\t}\n\t\t\t\tif err := downsampleBucket(ctx, logger, metrics, insBkt, metas, dataDir, downsampleConcurrency, blockFilesConcurrency, hashFunc, false); err != nil {\n\t\t\t\t\treturn errors.Wrap(err, \"downsampling failed\")\n\t\t\t\t}\n\n\t\t\t\tlevel.Info(logger).Log(\"msg\", \"start second pass of downsampling\")\n\t\t\t\tmetas, _, err = metaFetcher.Fetch(ctx)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn errors.Wrap(err, \"sync before second pass of downsampling\")\n\t\t\t\t}\n\t\t\t\tif err := downsampleBucket(ctx, logger, metrics, insBkt, metas, dataDir, downsampleConcurrency, blockFilesConcurrency, hashFunc, false); err != nil {\n\t\t\t\t\treturn errors.Wrap(err, \"downsampling failed\")\n\t\t\t\t}\n\t\t\t\treturn nil\n\t\t\t})\n\t\t}, func(error) {\n\t\t\tcancel()\n\t\t})\n\t}\n\n\tsrv := httpserver.New(logger, reg, comp, httpProbe,\n\t\thttpserver.WithListen(httpBindAddr),\n\t\thttpserver.WithGracePeriod(httpGracePeriod),\n\t\thttpserver.WithTLSConfig(httpTLSConfig),\n\t)\n\n\tg.Add(func() error {","sourceCodeStart":128,"sourceCodeEnd":164,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/downsample.go#L128-L164","documentation":"The second pass re-fetches metadata (metas, _, err = metaFetcher.Fetch(ctx)) so newly created 5m-resolution blocks can be downsampled to 1h. A Fetch error here is wrapped as \"sync before second pass of downsampling\". Same failure modes as the first sync: object-store access issues, corrupt metadata, or canceled context.","triggerScenarios":"metaFetcher.Fetch fails between the two downsampleBucket passes — network blip to the object store, expired credentials, invalid meta.json of a block just uploaded, or shutdown canceling the context.","commonSituations":"Bucket eventually-consistent listing missing/including fresh blocks; transient S3 5xx/SlowDown; operator restart during the long second phase; blocks written by another instance concurrently.","solutions":["Check the wrapped cause; treat context canceled as normal shutdown.","Verify object-store connectivity and credentials; retry — the loop repeats automatically.","Inspect the ULID reported in the error and validate/repair its meta.json.","Ensure only one downsampler operates on the bucket to avoid racing metadata writes."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Go: check object store reachability between passes\n_, err := bkt.Attributes(ctx, \"\")\nreturn err","typeGuard":null,"tryCatchPattern":"// Go\nif err := metaFetcher.Fetch(ctx); err != nil {\n    if ctx.Err() != nil { return nil } // shutdown, not a real failure\n    log.Printf(\"second-pass sync failed, retrying: %+v\", err)\n}","preventionTips":["Use STS/refreshable credentials sized for multi-hour downsample jobs.","Ensure bucket listing consistency; avoid aggressive negative-caching proxies in front of S3.","Handle SIGTERM so the job drains instead of canceling mid-fetch."],"tags":["object-storage","downsample","network"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}