{"record":{"id":"e30b10e3260343a4","repo":"thanos-io/thanos","slug":"upload-v","errorCode":null,"errorMessage":"upload %v","messagePattern":"upload (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/shipper/shipper.go","lineNumber":429,"sourceCode":"\t\tok, err := s.bucket.Exists(ctx, path.Join(m.ULID.String(), block.MetaFilename))\n\t\tif err != nil {\n\t\t\treturn uploaded, errors.Wrap(err, \"check exists\")\n\t\t}\n\t\tif ok {\n\t\t\tmeta.Uploaded = append(meta.Uploaded, m.ULID)\n\t\t\tcontinue\n\t\t}\n\n\t\t// Skip overlap check if out of order uploads is enabled.\n\t\tif m.Compaction.Level > 1 && !s.allowOutOfOrderUploads {\n\t\t\tif err := checker.IsOverlapping(ctx, m.BlockMeta); err != nil {\n\t\t\t\treturn uploaded, errors.Errorf(\"Found overlap or error during sync, cannot upload compacted block, details: %v\", err)\n\t\t\t}\n\t\t}\n\n\t\tif err := s.upload(ctx, m); err != nil {\n\t\t\tif !s.allowOutOfOrderUploads {\n\t\t\t\treturn uploaded, errors.Wrapf(err, \"upload %v\", m.ULID)\n\t\t\t}\n\n\t\t\t// No error returned, just log line. This is because we want other blocks to be uploaded even\n\t\t\t// though this one failed. It will be retried on second Sync iteration.\n\t\t\tlevel.Error(s.logger).Log(\"msg\", \"shipping failed\", \"block\", m.ULID, \"err\", err)\n\t\t\tuploadErrs++\n\t\t\tcontinue\n\t\t}\n\t\tmeta.Uploaded = append(meta.Uploaded, m.ULID)\n\t\tuploaded++\n\t\ts.metrics.uploads.Inc()\n\t}\n\tif err := WriteMetaFile(s.logger, s.metadataFilePath, meta); err != nil {\n\t\tlevel.Warn(s.logger).Log(\"msg\", \"updating meta file failed\", \"err\", err)\n\t}\n\n\tfailedExecution = false\n\tif uploadErrs > 0 || len(failedBlocks) > 0 {","sourceCodeStart":411,"sourceCodeEnd":447,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/shipper/shipper.go#L411-L447","documentation":"Wraps an error from s.upload(ctx, m) during Sync when out-of-order uploads are disabled — any block upload failure aborts the whole sync and returns the partial uploaded list. The underlying error (in [908]/[909] or bucket Put calls) is included with the failing block ULID.","triggerScenarios":"Sync with allowOutOfOrderUploads=false where uploading a block fails: hard-link creation into the upload dir fails, a file read fails (corrupted block), or the object storage Put/multipart upload errors.","commonSituations":"Corrupted segment files in a block (Checksum mismatches); object store throttling or auth expiry mid-sync; disk full or read-only data dir preventing hard-link creation; network interruption during a large compacted-block upload.","solutions":["Look at the wrapped cause for the specific block; if the block files are corrupted, delete the local block and restore it or let Prometheus re-create it.","If the cause is object storage, fix credentials/network/throttling (see [904]) and re-run Sync — it is idempotent.","Consider enabling --shipper.allow-out-of-order-uploads so single-block failures don't block all other uploads; failed blocks are retried on the next sync.","Free disk space / fix data-dir permissions if hard-link creation failed."],"exampleFix":"// before\n// upload 01ARZ...: upload file /data/01ARZ.../chunks/000001: read /data/01ARZ.../chunks/000001: input/output error\n// after (resilient config)\n// thanos sidecar --data-dir=/data --bucket=thanos --shipper.allow-out-of-order-uploads\n// # then repair/replace the corrupted block 01ARZ...","handlingStrategy":"retry","validationCode":"if err := validateBlockIntegrity(blockDir); err != nil {\n    return fmt.Errorf(\"block %s corrupt, skip upload: %w\", blockDir, err)\n}","typeGuard":null,"tryCatchPattern":"if _, err := shipper.Sync(ctx); err != nil {\n    var partial map[ulid.ULID]struct{}\n    if errors.As(err, &uploaded) {\n        // uploaded is returned alongside the error; retry only the missing ULIDs\n    }\n}","preventionTips":["Enable --shipper.allow-out-of-order-uploads so one bad block doesn't block all uploads","Monitor disk health (SMART) — upload failures often follow I/O errors","Keep headroom on the data volume for hard-linked upload copies","Set object-store retries/timeouts generously for large compacted blocks"],"tags":["upload","object-storage","corruption","sidecar"],"backgroundTag":"file-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}