{"record":{"id":"9f73dae189a04651","repo":"thanos-io/thanos","slug":"download-block-s","errorCode":null,"errorMessage":"download block %s","messagePattern":"download block (.+?)","errorType":"console","errorClass":"RetryError","httpStatus":null,"severity":"error","filePath":"cmd/thanos/downsample.go","lineNumber":362,"sourceCode":"\nfunc processDownsampling(\n\tctx context.Context,\n\tlogger log.Logger,\n\tbkt objstore.Bucket,\n\tm *metadata.Meta,\n\tdir string,\n\tresolution int64,\n\thashFunc metadata.HashFunc,\n\tmetrics *DownsampleMetrics,\n\tacceptMalformedIndex bool,\n\tblockFilesConcurrency int,\n) error {\n\tbegin := time.Now()\n\tbdir := filepath.Join(dir, m.ULID.String())\n\n\terr := block.Download(ctx, logger, bkt, m.ULID, bdir, objstore.WithFetchConcurrency(blockFilesConcurrency))\n\tif err != nil {\n\t\treturn compact.NewRetryError(errors.Wrapf(err, \"download block %s\", m.ULID))\n\t}\n\tlevel.Info(logger).Log(\"msg\", \"downloaded block\", \"id\", m.ULID, \"duration\", time.Since(begin), \"duration_ms\", time.Since(begin).Milliseconds())\n\n\tif err := block.VerifyIndex(ctx, logger, filepath.Join(bdir, block.IndexFilename), m.MinTime, m.MaxTime); err != nil && !acceptMalformedIndex {\n\t\treturn errors.Wrap(err, \"input block index not valid\")\n\t}\n\n\tbegin = time.Now()\n\n\tvar pool chunkenc.Pool\n\tif m.Thanos.Downsample.Resolution == 0 {\n\t\tpool = chunkenc.NewPool()\n\t} else {\n\t\tpool = downsample.NewPool()\n\t}\n\n\tb, err := tsdb.OpenBlock(logutil.GoKitLogToSlog(logger), bdir, pool, nil)\n\tif err != nil {","sourceCodeStart":344,"sourceCodeEnd":380,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/downsample.go#L344-L380","documentation":"processDownsampling wraps block.Download failures with compact.NewRetryError(errors.Wrapf(err, \"download block %s\", m.ULID)). Downloading the TSDB block from the object store failed. Because it is a RetryError, the compactor treats it as transient and retries the block on a later cycle instead of halting.","triggerScenarios":"block.Download for block m.ULID fails: object not found (block deleted concurrently by compactor), network timeouts, S3/GCS 5xx or throttling, wrong credentials, or fetch-concurrency issues.","commonSituations":"Two downsamplers/compactors racing so one deletes a block the other is downloading; bucket rate limits under high block download concurrency; expired/misconfigured object store credentials; proxy or VPC endpoint misconfiguration.","solutions":["Let the retry mechanism work — RetryError failures are retried on subsequent cycles; just confirm the downsampler stays up.","Check object store credentials and network reachability from the downsampler pod to the bucket endpoint.","Ensure only one downsampler operates against the same bucket and that block deletion/gc is not racing downloads.","Reduce --block-files-concurrency if the store throttles concurrent GETs; inspect inner error for 404 vs 403 vs 5xx."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before running downsampler, check reachability\n_, err := bkt.Iter(ctx, \"\", func(string) error { return nil })\nif err != nil { log.Fatalf(\"object store unreachable: %v\", err) }","typeGuard":null,"tryCatchPattern":"if compact.IsRetryable(err) { /* schedule retry with backoff */ } else { /* alert: bucket auth/config problem */ }","preventionTips":["Use exponential backoff and retry-friendly object store client settings.","Alert on object-store 404s to detect racing deletions between compactors/downsamplers.","Verify credentials and bucket policy in staging with the same flags used in production."],"tags":["object-storage","download","retry"],"backgroundTag":"resource-not-found","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}