{"record":{"id":"4a94e032f8215771","repo":"thanos-io/thanos","slug":"upload","errorCode":null,"errorMessage":"upload","messagePattern":"upload","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/thanos/tools_bucket.go","lineNumber":1322,"sourceCode":"\n\t\t\t\tif tbc.dryRun {\n\t\t\t\t\tlevel.Info(logger).Log(\"msg\", \"dry run finished. Changes should be printed to stderr\", \"Block ID\", id)\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\n\t\t\t\tlevel.Info(logger).Log(\"msg\", \"wrote new block after modifications; flushing\", \"source\", id, \"new\", newID)\n\t\t\t\tmeta.Stats, err = d.Flush()\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn errors.Wrap(err, \"flush\")\n\t\t\t\t}\n\t\t\t\tif err := meta.WriteToDir(logger, filepath.Join(tbc.tmpDir, newID.String())); err != nil {\n\t\t\t\t\treturn err\n\t\t\t\t}\n\n\t\t\t\tlevel.Info(logger).Log(\"msg\", \"uploading new block\", \"source\", id, \"new\", newID)\n\t\t\t\tif tbc.promBlocks {\n\t\t\t\t\tif err := block.UploadPromBlock(ctx, logger, insBkt, filepath.Join(tbc.tmpDir, newID.String()), metadata.HashFunc(*hashFunc)); err != nil {\n\t\t\t\t\t\treturn errors.Wrap(err, \"upload\")\n\t\t\t\t\t}\n\t\t\t\t} else {\n\t\t\t\t\tif err := block.Upload(ctx, logger, insBkt, filepath.Join(tbc.tmpDir, newID.String()), metadata.HashFunc(*hashFunc)); err != nil {\n\t\t\t\t\t\treturn errors.Wrap(err, \"upload\")\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tlevel.Info(logger).Log(\"msg\", \"uploaded\", \"source\", id, \"new\", newID)\n\n\t\t\t\tif !tbc.dryRun && tbc.deleteBlocks {\n\t\t\t\t\tif err := block.MarkForDeletion(ctx, logger, insBkt, id, \"block rewritten\", stubCounter); err != nil {\n\t\t\t\t\t\tlevel.Error(logger).Log(\"msg\", \"failed to mark block for deletion\", \"id\", id.String(), \"err\", err)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\tlevel.Info(logger).Log(\"msg\", \"rewrite done\", \"IDs\", strings.Join(tbc.blockIDs, \",\"))\n\t\t\treturn nil\n\t\t}, func(err error) {\n\t\t\tcancel()","sourceCodeStart":1304,"sourceCodeEnd":1340,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/cmd/thanos/tools_bucket.go#L1304-L1340","documentation":"Wraps block.UploadPromBlock failing when uploading the rewritten block as a Prometheus-style block to the object store. Upload writes all block files (index, chunks, meta.json) and verifies checksums; failures stem from object store errors, network issues, permission problems, or corrupted local files (hash mismatch on upload).","triggerScenarios":"tbc.promBlocks is set and the upload of the new block directory fails: wrong bucket/credentials, network interruption, object-store rate limiting, or a local file corruption detected during hashing.","commonSituations":"Expired cloud credentials during long rewrite runs; S3/GCS throttling when uploading many chunk files; bucket write permissions missing for the service account; partial local flush producing bad checksums.","solutions":["Verify object store write permissions and credentials for the bucket, then retry the rewrite.","Retry after confirming network stability; the operation is safe to re-run since IDs are freshly generated.","If checksum errors persist, clear tmpDir and re-download/re-rewrite the source block."],"exampleFix":"// before (stale credentials)\nexport AWS_SECRET_ACCESS_KEY=old\n// after\nexport AWS_SECRET_ACCESS_KEY=$(aws get-new-key) # re-auth before long-running rewrite","handlingStrategy":"retry","validationCode":"// pre-check write access\nif err := bkt.Upload(ctx, path.Join(testPrefix, \"probe\"), []byte(\"ok\")); err != nil {\n    return errors.Wrap(err, \"bucket not writable\")\n}","typeGuard":null,"tryCatchPattern":"if err := block.UploadPromBlock(ctx, logger, bkt, dir, hashFn); err != nil {\n    if isRateLimit(err) { time.Sleep(30*time.Second); return retry() }\n    return errors.Wrap(err, \"upload\")\n}","preventionTips":["Use long-lived credentials for long-running rewrites","Verify bucket write permissions before starting","Enable retries/backoff in the objstore client config"],"tags":["network","thanos","object-store","upload"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}