{"record":{"id":"8282504ea900eb39","repo":"thanos-io/thanos","slug":"upload-of-s-failed","errorCode":null,"errorMessage":"upload of %s failed","messagePattern":"upload of (.+?) failed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/compact/compact.go","lineNumber":1151,"sourceCode":"\n\tmeta, err := metadata.ReadFromDir(bdir)\n\tif err != nil {\n\t\treturn errors.Wrapf(err, \"read meta from %s\", bdir)\n\t}\n\n\tresid, err := block.Repair(ctx, logger, tmpdir, ie.id, metadata.CompactorRepairSource, block.IgnoreIssue347OutsideChunk)\n\tif err != nil {\n\t\treturn errors.Wrapf(err, \"repair failed for block %s\", ie.id)\n\t}\n\n\t// Verify repaired id before uploading it.\n\tif err := block.VerifyIndex(ctx, logger, filepath.Join(tmpdir, resid.String(), block.IndexFilename), meta.MinTime, meta.MaxTime); err != nil {\n\t\treturn errors.Wrapf(err, \"repaired block is invalid %s\", resid)\n\t}\n\n\tlevel.Info(logger).Log(\"msg\", \"uploading repaired block\", \"newID\", resid)\n\tif err = block.Upload(ctx, logger, bkt, filepath.Join(tmpdir, resid.String()), metadata.NoneFunc); err != nil {\n\t\treturn retry(errors.Wrapf(err, \"upload of %s failed\", resid))\n\t}\n\n\tlevel.Info(logger).Log(\"msg\", \"deleting broken block\", \"id\", ie.id)\n\n\t// Spawn a new context so we always mark a block for deletion in full on shutdown.\n\tdelCtx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)\n\tdefer cancel()\n\n\t// TODO(bplotka): Issue with this will introduce overlap that will halt compactor. Automate that (fix duplicate overlaps caused by this).\n\tif err := block.MarkForDeletion(delCtx, logger, bkt, ie.id, \"source of repaired block\", blocksMarkedForDeletion); err != nil {\n\t\treturn errors.Wrapf(err, \"marking old block %s for deletion has failed\", ie.id)\n\t}\n\treturn nil\n}\n\nfunc (cg *Group) compact(ctx context.Context, dir string, planner Planner, comp Compactor, blockDeletableChecker BlockDeletableChecker, compactionLifecycleCallback CompactionLifecycleCallback, errChan chan error) (bool, []ulid.ULID, error) {\n\tcg.mtx.Lock()\n\tdefer cg.mtx.Unlock()","sourceCodeStart":1133,"sourceCodeEnd":1169,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/compact/compact.go#L1133-L1169","documentation":"This error wraps a failure of block.Upload when the compactor, after successfully repairing and verifying a block, uploads the new block (resid) from tmpdir to the object store bucket (compact.go repairBucketBlock). It is wrapped with retry(...), so the compactor treats it as retryable. It means the repaired, valid block could not be persisted to storage.","triggerScenarios":"block.Upload(ctx, logger, bkt, filepath.Join(tmpdir, resid.String()), metadata.NoneFunc) fails: object storage returns 5xx/403/timeout mid-upload, network interruption between compactor and bucket, bucket write permission missing, multipart upload aborted by the provider, or local tmpdir files were removed mid-upload (external cleanup job deleting tmp).","commonSituations":"S3/GCS/Azure credentials expired or lacking PutObject permission on the bucket; object storage rate limiting or throttling during large uploads; proxy/firewall cutting long connections; compactor's tmp cleanup (or k8s emptyDir eviction) racing with the upload; bucket versioning/retention policies rejecting writes.","solutions":["Check the inner error for the storage-specific cause (HTTP status, credential error); fix IAM/bucket policy to allow write on the bucket","Rely on the built-in retry: transient network errors are retried automatically; investigate only if it exhausts retries","Verify no external process (tmp cleaner, k8s gc) deletes tmpdir/<resid> during compaction; pin the compactor's tmpdir to durable storage","Check object storage service health/quotas (request rate, egress) if failures correlate with large uploads","Validate objstore configuration (thanos tools bucket verify) and credentials rotation settings"],"exampleFix":"// before: failing bucket config with missing write scope (S3 IAM)\n// after: grant the compactor's role PutObject on the bucket prefix\n// aws iam: add statement {\"Effect\":\"Allow\",\"Action\":[\"s3:PutObject\",\"s3:AbortMultipartUpload\"],\"Resource\":\"arn:aws:s3:::thanos-bucket/*\"}","handlingStrategy":"retry","validationCode":"// preflight: ensure the compactor can write to the bucket before long repair jobs\nctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)\ndefer cancel()\nif err := bkt.Upload(ctx, \"health-check\", bytes.NewReader(nil)); err != nil {\n    return fmt.Errorf(\"bucket not writable: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"err := retry(func() error { // thanos' own retry wrapper already applies here\n    return block.Upload(ctx, logger, bkt, blockDir, metadata.NoneFunc)\n})\nif err != nil {\n    logger.Error(\"upload exhausted retries; check object storage health and IAM\", \"err\", err)\n}","preventionTips":["Grant the compactor's service account explicit write (PutObject) permissions on the bucket","Use durable storage for tmpdir so external cleanup does not race with uploads","Monitor object storage 5xx/throttling metrics; schedule compaction away from provider incidents","Validate objstore config at startup with 'thanos tools bucket verify'"],"tags":["thanos","object-storage","upload","network","retryable"],"backgroundTag":"http-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}