{"record":{"id":"d54cff93f8070bdb","repo":"thanos-io/thanos","slug":"upload-downsampled-block-s","errorCode":null,"errorMessage":"upload downsampled block %s","messagePattern":"upload downsampled block (.+?)","errorType":"console","errorClass":"RetryError","httpStatus":null,"severity":"error","filePath":"cmd/thanos/downsample.go","lineNumber":423,"sourceCode":"\tif err != nil {\n\t\treturn errors.Wrap(err, \"read meta\")\n\t}\n\n\tif stats.ChunkMaxSize > 0 {\n\t\tmeta.Thanos.IndexStats.ChunkMaxSize = stats.ChunkMaxSize\n\t}\n\tif stats.SeriesMaxSize > 0 {\n\t\tmeta.Thanos.IndexStats.SeriesMaxSize = stats.SeriesMaxSize\n\t}\n\tif err := meta.WriteToDir(logger, resdir); err != nil {\n\t\treturn errors.Wrap(err, \"write meta\")\n\t}\n\n\tbegin = time.Now()\n\n\terr = block.Upload(ctx, logger, bkt, resdir, hashFunc, objstore.WithUploadConcurrency(blockFilesConcurrency))\n\tif err != nil {\n\t\treturn compact.NewRetryError(errors.Wrapf(err, \"upload downsampled block %s\", id))\n\t}\n\n\tlevel.Info(logger).Log(\"msg\", \"uploaded block\", \"id\", id, \"duration\", time.Since(begin), \"duration_ms\", time.Since(begin).Milliseconds())\n\n\t// It is not harmful if these fails.\n\tif err := os.RemoveAll(bdir); err != nil {\n\t\tlevel.Warn(logger).Log(\"msg\", \"failed to clean directory\", \"dir\", bdir, \"err\", err)\n\t}\n\tif err := os.RemoveAll(resdir); err != nil {\n\t\tlevel.Warn(logger).Log(\"msg\", \"failed to clean directory\", \"resdir\", bdir, \"err\", err)\n\t}\n\n\treturn nil\n}\n","sourceCodeStart":405,"sourceCodeEnd":438,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/downsample.go#L405-L438","documentation":"This error wraps a failure to upload a downsampled Prometheus TSDB block to object storage in Thanos. It is produced in processDownsampling when block.Upload fails, and it is wrapped in compact.NewRetryError so the compaction/worker loop retries it. The underlying cause (object storage connectivity, permissions, partial files) is in the wrapped error.","triggerScenarios":"block.Upload returns a non-nil error: object store unreachable or credentials invalid, remote write of a block file fails mid-transfer, context cancelled during upload, or the hashed temp dir contents are unreadable.","commonSituations":"Misconfigured S3/GCS/Azure credentials or endpoint in the objstore config; network outage between Thanos Compactor and bucket; bucket permission (no put-object); too-aggressive ctx timeout killing long uploads of large downsampled blocks.","solutions":["Check the wrapped cause in the error for the object-store failure and verify bucket credentials/endpoint/permissions with the same config via `thanos tools bucket verify` or objstore client.","Retry: the error is a compact.RetryError, so transient network failures resolve on the next compaction iteration; fix permanent causes (creds, bucket policy) so retries succeed.","Confirm the downsampled block directory (resdir) is complete and readable before upload; re-run downsample for that block if files are corrupt.","Increase upload concurrency/timeout settings (blockFilesConcurrency, ctx deadline) if uploads are timing out on large blocks."],"exampleFix":"// before\nerr = block.Upload(ctx, logger, bkt, resdir, hashFunc, objstore.WithUploadConcurrency(blockFilesConcurrency))\n// after: fail fast with clearer diagnostics\nif err := objstore.UploadProbe(ctx, bkt); err != nil {\n    level.Error(logger).Log(\"msg\", \"object store unreachable, skipping downsample upload\", \"err\", err)\n    return compact.NewRetryError(errors.Wrapf(err, \"upload downsampled block %s\", id))\n}\nerr = block.Upload(ctx, logger, bkt, resdir, hashFunc, objstore.WithUploadConcurrency(blockFilesConcurrency))","handlingStrategy":"retry","validationCode":"if err := bkt.Iter(ctx, \"/\", func(string) error { return nil }); err != nil { return fmt.Errorf(\"object store unreachable before upload: %w\", err) }","typeGuard":"func isRetryable(err error) bool { var re *compact.RetryError; return errors.As(err, &re) }","tryCatchPattern":"err := block.Upload(ctx, logger, bkt, resdir, hashFunc, opts)\nvar re *compact.RetryError\nif errors.As(err, &re) { logTransient(err); return err } // retried by compactor\nif errors.Is(err, context.Canceled) { return err }\nlogPermanent(err); return err","preventionTips":["Validate objstore config with `thanos tools bucket verify` before production uploads.","Set generous context timeouts for large block uploads.","Monitor bucket credentials rotation and IAM policy changes.","Pre-check free disk space and readability of the block directory before upload."],"tags":["object-storage","thanos","retry","upload"],"backgroundTag":"api-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}