{"record":{"id":"fca3d7dbdbbd87ff","repo":"thanos-io/thanos","slug":"download-from-source","errorCode":null,"errorMessage":"download from source","messagePattern":"download from source","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/verifier/safe_delete.go","lineNumber":67,"sourceCode":"\tif found {\n\t\treturn errors.Errorf(\"%s dir seems to exists in backup bucket. Remove this block manually if you are sure it is safe to do\", id)\n\t}\n\n\t// Create a tempdir to locally store TSDB block.\n\ttempdir, err := os.MkdirTemp(\"\", fmt.Sprintf(\"safe-delete-%s\", id.String()))\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer func() {\n\t\tif err := os.RemoveAll(tempdir); err != nil {\n\t\t\tlevel.Warn(ctx.Logger).Log(\"msg\", \"failed to delete dir\", \"dir\", tempdir, \"err\", err)\n\t\t}\n\t}()\n\n\t// Download the TSDB block.\n\tdir := filepath.Join(tempdir, id.String())\n\tif err := block.Download(ctx, ctx.Logger, ctx.Bkt, id, dir); err != nil {\n\t\treturn errors.Wrap(err, \"download from source\")\n\t}\n\n\t// Backup the block.\n\tif err := backupDownloaded(ctx, ctx.Logger, dir, ctx.BackupBkt, id); err != nil {\n\t\treturn err\n\t}\n\n\t// Block uploaded, so we are ok to remove from src bucket.\n\tif ctx.DeleteDelay.Seconds() == 0 {\n\t\tlevel.Info(ctx.Logger).Log(\"msg\", \"Deleting block\", \"id\", id.String())\n\t\tif err := block.Delete(ctx, ctx.Logger, ctx.Bkt, id); err != nil {\n\t\t\treturn errors.Wrap(err, \"delete from source\")\n\t\t}\n\t}\n\n\tlevel.Info(ctx.Logger).Log(\"msg\", \"Marking block as deleted\", \"id\", id.String())\n\tif err := block.MarkForDeletion(ctx, ctx.Logger, ctx.Bkt, id, \"manual verify-repair\", ctx.metrics.blocksMarkedForDeletion); err != nil {\n\t\treturn errors.Wrap(err, \"marking delete from source\")","sourceCodeStart":49,"sourceCodeEnd":85,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/verifier/safe_delete.go#L49-L85","documentation":"BackupAndDelete downloads the block to a local temp dir with block.Download before backing it up; a failure is wrapped as \"download from source\". The broken block could not be fully retrieved from the source bucket, so neither backup nor delete happened.","triggerScenarios":"block.Download fails: missing objects in the block dir (incomplete block), GetObject permission/network errors, checksum/size mismatch, or local temp-dir write failures (disk full, permissions).","commonSituations":"Source block partially deleted or never fully uploaded; object store outage mid-download; insufficient disk space in the OS temp dir; IAM policy allowing List but not Get on the block's objects.","solutions":["Read the wrapped cause: missing-object errors indicate an incomplete block in the source bucket.","Check temp disk space and permissions (os.MkdirTemp target filesystem).","Retry after transient object-store/network issues.","If the block is incomplete in the bucket, restore from backup or delete it manually — repair/delete cannot proceed without a full download."],"exampleFix":"// before: small temp filesystem filling up\nTMPDIR=/run/user/tmp\n// after: temp dir on a volume with enough free space\nTMPDIR=/var/tmp th (...verifier run...)","handlingStrategy":"retry","validationCode":"// ensure temp volume has room for the block before download\nif st, err := os.Stat(tempdir); err == nil && syscall.Statfs(tempdir).Bavail*uint64(st.Size()) < blockSize { ... }\n// simpler: confirm all block objects exist in source bucket first","typeGuard":null,"tryCatchPattern":"if err := verifier.VerifyRepair(...); err != nil {\n    if strings.Contains(err.Error(), \"download from source\") {\n        // retry on transient errors; missing objects mean the block is incomplete — restore or delete manually\n    }\n}","preventionTips":["Point TMPDIR at a volume with ample free space","Grant GetObject on all block objects to the verifier identity","Detect incomplete blocks in the bucket during routine maintenance"],"tags":["go","object-storage","download","thanos"],"backgroundTag":"file-read-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}