{"record":{"id":"6fe14c37058bb07a","repo":"thanos-io/thanos","slug":"upload-of-s-failed-6fe14c","errorCode":null,"errorMessage":"upload of %s failed","messagePattern":"upload of (.+?) failed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/verifier/index_issue.go","lineNumber":121,"sourceCode":"\t\tdir,\n\t\tid,\n\t\tmetadata.BucketRepairSource,\n\t\tblock.IgnoreCompleteOutsideChunk,\n\t\tblock.IgnoreDuplicateOutsideChunk,\n\t\tblock.IgnoreIssue347OutsideChunk,\n\t)\n\tif err != nil {\n\t\treturn errors.Wrapf(err, \"repair failed for block %s\", id)\n\t}\n\tlevel.Info(ctx.Logger).Log(\"msg\", \"verifying repaired block\", \"id\", id, \"newID\", resid)\n\n\tif err := block.VerifyIndex(ctx, ctx.Logger, filepath.Join(dir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {\n\t\treturn errors.Wrapf(err, \"repaired block is invalid %s\", resid)\n\t}\n\n\tlevel.Info(ctx.Logger).Log(\"msg\", \"uploading repaired block\", \"newID\", resid)\n\tif err = block.Upload(ctx, ctx.Logger, ctx.Bkt, filepath.Join(dir, resid.String()), metadata.NoneFunc); err != nil {\n\t\treturn errors.Wrapf(err, \"upload of %s failed\", resid)\n\t}\n\n\tlevel.Info(ctx.Logger).Log(\"msg\", \"safe deleting broken block\", \"id\", id, \"issue\")\n\tif err := BackupAndDeleteDownloaded(ctx, filepath.Join(dir, id.String()), id); err != nil {\n\t\treturn errors.Wrapf(err, \"safe deleting old block %s failed\", id)\n\t}\n\n\treturn nil\n}\n\nfunc verifyIndex(ctx Context, id ulid.ULID, dir string, meta *metadata.Meta) (stats block.HealthStats, err error) {\n\tif err := objstore.DownloadFile(ctx, ctx.Logger, ctx.Bkt, path.Join(id.String(), block.IndexFilename), filepath.Join(dir, block.IndexFilename)); err != nil {\n\t\treturn stats, errors.Wrapf(err, \"download index file %s\", path.Join(id.String(), block.IndexFilename))\n\t}\n\n\tstats, err = block.GatherIndexHealthStats(ctx, ctx.Logger, filepath.Join(dir, block.IndexFilename), meta.MinTime, meta.MaxTime)\n\tif err != nil {\n\t\treturn stats, errors.Wrapf(err, \"gather index issues %s\", id)","sourceCodeStart":103,"sourceCodeEnd":139,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/verifier/index_issue.go#L103-L139","documentation":"After the repaired block passes verification, repairIndex uploads it to the object store via block.Upload; a failure is wrapped as \"upload of %s failed\". The repaired block exists locally but was not persisted to the bucket.","triggerScenarios":"block.Upload to ctx.Bkt fails: object store unreachable, credentials invalid/expired, bucket write permission missing, network timeout mid-multipart, or the destination block ID already exists in the bucket.","commonSituations":"Expired S3/GCS credentials; misconfigured objstore config in the verifier; bucket policy denying PutObject; proxy/firewall cutting long uploads; a previous partial upload leaving the resid prefix present.","solutions":["Inspect the wrapped cause: for auth errors, refresh/rotate the object store credentials.","Test bucket write access with the same credentials (e.g. 'thanos tools bucket ls' and a test put).","Re-run the repair; Upload is idempotent per block ID and will retry the transfer.","If 'already exists' style cause, verify the existing remote block is not corrupt; remove it manually before re-uploading.","Increase object store client timeout/retry settings in the objstore config."],"exampleFix":"// before: verifier objstore config with a short-lived static key\n{\"type\":\"S3\",\"config\":{\"access_key\":\"AKIA...\"}}\n// after: use instance role / credential refresh\n{\"type\":\"S3\",\"config\":{\"role_arn\":\"arn:aws:iam::...:role/thanos-verifier\"}}","handlingStrategy":"retry","validationCode":"// pre-check bucket write access\n_, err := bkt.Upload(ctx, \".probe/thanos-verifier-probe\", strings.NewReader(\"ok\"))","typeGuard":null,"tryCatchPattern":"if err := verifier.VerifyRepair(...); err != nil {\n    if strings.Contains(err.Error(), \"upload of \") {\n        // backoff-retry the whole repair; upload is safe to repeat per block ID\n    }\n}","preventionTips":["Use non-expiring credentials (instance roles/workload identity)","Enable retries and adequate timeouts in the objstore config","Test PutObject permission before long verification runs"],"tags":["go","object-storage","upload","thanos"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}