{"record":{"id":"31999342671685d0","repo":"thanos-io/thanos","slug":"replace-chunks-dir","errorCode":null,"errorMessage":"replace chunks dir","messagePattern":"replace chunks dir","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/block/writer.go","lineNumber":143,"sourceCode":"\n\t// Close temp dir before rename block dir (for windows platform).\n\tif err = df.Close(); err != nil {\n\t\treturn tsdb.BlockStats{}, errors.Wrap(err, \"close temporary dir\")\n\t}\n\tdf = nil\n\n\tif err := tsdb_errors.CloseAll(d.closers); err != nil {\n\t\td.closers = nil\n\t\treturn tsdb.BlockStats{}, err\n\t}\n\td.closers = nil\n\n\t// Block files successfully written, make them visible by moving files from tmp dir.\n\tif err := fileutil.Replace(filepath.Join(d.bTmp, IndexFilename), filepath.Join(d.bDir, IndexFilename)); err != nil {\n\t\treturn tsdb.BlockStats{}, errors.Wrap(err, \"replace index file\")\n\t}\n\tif err := fileutil.Replace(filepath.Join(d.bTmp, ChunksDirname), filepath.Join(d.bDir, ChunksDirname)); err != nil {\n\t\treturn tsdb.BlockStats{}, errors.Wrap(err, \"replace chunks dir\")\n\t}\n\treturn d.stats, nil\n}\n\ntype statsGatheringSeriesWriter struct {\n\tiw tsdb.IndexWriter\n\tcw tsdb.ChunkWriter\n\n\tstats   tsdb.BlockStats\n\tsymbols int64\n}\n\nfunc (s *statsGatheringSeriesWriter) AddSymbol(sym string) error {\n\tif err := s.iw.AddSymbol(sym); err != nil {\n\t\treturn err\n\t}\n\ts.symbols++\n\treturn nil","sourceCodeStart":125,"sourceCodeEnd":161,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/block/writer.go#L125-L161","documentation":"Thanos compactor's block writer (pkg/block/writer.go Flush) finalizes a block by atomically moving the index file and chunks directory from a temporary directory (bTmp) into the final block directory (bDir) with fileutil.Replace. This error wraps a failure to move the 'chunks' directory into place after the index was already moved. It means the block was mostly written but the final rename/exchange failed, typically due to filesystem-level problems or leftover destination state.","triggerScenarios":"fileutil.Replace (renameat2 RENAME_EXCHANGE or rename fallback) fails when the destination bDir/chunks already exists on a filesystem without RENAME_EXCHANGE support, when source and destination are on different mounts/filesystems, when the tmp dir was removed concurrently, or on EACCES/EXDEV/ENOSPC during the rename.","commonSituations":"Block directory left behind by a previously crashed compaction; object-storage-backed local cache on a different filesystem than the tmp dir; read-only or full disk on the compactor; parallel compaction processes writing the same block ID; network filesystems (NFS) lacking renameat2 support.","solutions":["Delete the stale/partial destination block directory (bDir) so the replace target no longer exists, then rerun compaction","Ensure the tmp dir and block dir are on the same filesystem (configure --data-dir and tmp on the same mount)","Check disk space and write permissions on the block store directory","If on NFS/fuse without RENAME_EXCHANGE, move to a local filesystem or a fileutil version with a rename fallback","Verify no concurrent compactor process is writing the same block ( Thanos compactor must run as a singleton)"],"exampleFix":"// before: tmp on different FS\n# thanos compact --data-dir=/data/thanos --tmp.dir=/tmp\n// after: same filesystem\n# thanos compact --data-dir=/data/thanos --tmp.dir=/data/thanos-tmp","handlingStrategy":"retry","validationCode":"// before triggering compaction flush\nif _, err := os.Stat(bDir); err == nil {\n    return fmt.Errorf(\"destination block dir %s already exists; clean it first\", bDir)\n}\nif st, st2, err := statFS(bTmp), statFS(bDir), error(nil); sameMount(bTmp, bDir) == false {\n    return fmt.Errorf(\"tmp dir and block dir must be on the same filesystem\")\n}","typeGuard":null,"tryCatchPattern":"// Go\nif err := Flush(...); err != nil {\n    if strings.Contains(err.Error(), \"replace chunks dir\") {\n        os.RemoveAll(blockDir) // clear stale destination\n        err = retryFlush()\n    }\n    return err\n}","preventionTips":["Keep --data-dir and tmp dir on the same local filesystem","Use local disk (ext4/xfs), not NFS, for block writes","Ensure the compactor runs as a single instance per bucket","Monitor disk free space on the block store volume"],"tags":["filesystem","compaction","rename"],"backgroundTag":"file-write-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}