{"record":{"id":"f7d0a41aa5cbd39b","repo":"thanos-io/thanos","slug":"download-block-s-f7d0a4","errorCode":null,"errorMessage":"download block %s","messagePattern":"download block (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/compact/compact.go","lineNumber":1131,"sourceCode":"\t\treturn errors.Errorf(\"Given error is not an issue347 error: %v\", issue347Err)\n\t}\n\n\tlevel.Info(logger).Log(\"msg\", \"Repairing block broken by https://github.com/prometheus/tsdb/issues/347\", \"id\", ie.id, \"err\", issue347Err)\n\n\ttmpdir, err := os.MkdirTemp(\"\", fmt.Sprintf(\"repair-issue-347-id-%s-\", ie.id))\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tdefer func() {\n\t\tif err := os.RemoveAll(tmpdir); err != nil {\n\t\t\tlevel.Warn(logger).Log(\"msg\", \"failed to remote tmpdir\", \"err\", err, \"tmpdir\", tmpdir)\n\t\t}\n\t}()\n\n\tbdir := filepath.Join(tmpdir, ie.id.String())\n\tif err := block.Download(ctx, logger, bkt, ie.id, bdir); err != nil {\n\t\treturn retry(errors.Wrapf(err, \"download block %s\", ie.id))\n\t}\n\n\tmeta, err := metadata.ReadFromDir(bdir)\n\tif err != nil {\n\t\treturn errors.Wrapf(err, \"read meta from %s\", bdir)\n\t}\n\n\tresid, err := block.Repair(ctx, logger, tmpdir, ie.id, metadata.CompactorRepairSource, block.IgnoreIssue347OutsideChunk)\n\tif err != nil {\n\t\treturn errors.Wrapf(err, \"repair failed for block %s\", ie.id)\n\t}\n\n\t// Verify repaired id before uploading it.\n\tif err := block.VerifyIndex(ctx, logger, filepath.Join(tmpdir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {\n\t\treturn errors.Wrapf(err, \"repaired block is invalid %s\", resid)\n\t}\n\n\tlevel.Info(logger).Log(\"msg\", \"uploading repaired block\", \"newID\", resid)","sourceCodeStart":1113,"sourceCodeEnd":1149,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/compact.go#L1113-L1149","documentation":"During RepairIssue347, the affected block is re-downloaded from the object store bucket into a temp dir before it can be rebuilt. This error wraps any failure of block.Download for the corrupted block ID — network failures, missing object parts, or bucket access errors — and is passed to retry() so the whole repair can be retried.","triggerScenarios":"block.Download(ctx, logger, bkt, ie.id, bdir) fails: object missing (404) in the bucket, transient network/S3 errors, permission denied on bucket read, or local disk full/write error while writing chunk/index files into tmpdir.","commonSituations":"Bucket credentials lacking read permission for the block prefix; object store throttling (S3 503/slow-down); partially uploaded block missing meta.json or index; DNS/proxy outages between compactor and object storage; tmp disk quota exceeded.","solutions":["Verify the bucket credentials/permissions allow reading the block prefix (meta.json, index, chunks/*)","Check the block actually exists in the bucket (`thanos tools bucket ls` / verify); if partially uploaded, remove or re-upload it","Rely on the built-in retry — for persistent failures check object-store connectivity (endpoint, DNS, TLS) and rate limits","Free disk space on the compactor tmp volume if the download fails writing locally","If the object is unrecoverable, delete/mark the corrupted block for deletion so compaction proceeds"],"exampleFix":"// before: wrong credentials\n--objstore.bucket=s3 --s3.access-key=<readonly-on-other-bucket>\n// after: grant read on the bucket\n--objstore.bucket=s3 --s3.access-key=<key-with-bucket-read>","handlingStrategy":"retry","validationCode":"// before repair, check the block is downloadable\nrc, err := bkt.Get(ctx, path.Join(ie.id.String(), \"meta.json\"))\nif err != nil {\n    return fmt.Errorf(\"block %s not readable in bucket: %w\", ie.id, err)\n}\nrc.Close()\n// and check local space\nif free(tmpdir) < minBlockBytes {\n    return fmt.Errorf(\"insufficient disk space for download\")\n}","typeGuard":null,"tryCatchPattern":"// Go\nerr := block.Download(ctx, logger, bkt, id, bdir)\nif err != nil {\n    if isRetryable(err) { // net.Error, 5xx, throttling\n        backoff.Retry(func() error { return block.Download(ctx, logger, bkt, id, bdir) }, ...)\n    }\n    return errors.Wrapf(err, \"download block %s\", id)\n}","preventionTips":["Grant the compactor bucket read permissions for all block prefixes","Set object-store timeout/retry (backoff) config for S3/GCS throttling","Monitor tmp volume disk usage on the compactor","Run `thanos tools bucket ls/verify` to catch missing or partial blocks early"],"tags":["object-storage","download","retry"],"backgroundTag":"network-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}