{"record":{"id":"e5a93010ce9c0d2a","repo":"thanos-io/thanos","slug":"next-symbol","errorCode":null,"errorMessage":"next symbol","messagePattern":"next symbol","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/block/index.go","lineNumber":591,"sourceCode":"\n// rewrite writes all data from the readers back into the writers while cleaning\n// up mis-ordered and duplicated chunks.\nfunc rewrite(\n\tctx context.Context,\n\tlogger log.Logger,\n\tindexr tsdb.IndexReader, chunkr tsdb.ChunkReader,\n\tindexw tsdb.IndexWriter, chunkw tsdb.ChunkWriter,\n\tmeta *metadata.Meta,\n\tignoreChkFns []ignoreFnType,\n) error {\n\tsymbols := indexr.Symbols()\n\tfor symbols.Next() {\n\t\tif err := indexw.AddSymbol(symbols.At()); err != nil {\n\t\t\treturn errors.Wrap(err, \"add symbol\")\n\t\t}\n\t}\n\tif symbols.Err() != nil {\n\t\treturn errors.Wrap(symbols.Err(), \"next symbol\")\n\t}\n\n\tkey, value := index.AllPostingsKey()\n\tall, err := indexr.Postings(ctx, key, value)\n\tif err != nil {\n\t\treturn errors.Wrap(err, \"postings\")\n\t}\n\tall = indexr.SortedPostings(all)\n\n\t// We fully rebuild the postings list index from merged series.\n\tvar (\n\t\tpostings = index.NewMemPostings()\n\t\tvalues   = map[string]stringset{}\n\t\ti        = storage.SeriesRef(0)\n\t\tseries   = []seriesRepair{}\n\t)\n\n\tvar builder labels.ScratchBuilder","sourceCodeStart":573,"sourceCodeEnd":609,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/block/index.go#L573-L609","documentation":"This error wraps symbols.Err() after iterating the source index's symbol table in block.Rewrite (pkg/block/index.go:591). It means the Symbol iterator itself returned an error while reading the symbol table from the source index file — i.e. the source index is truncated or corrupt at the symbol section, not a write failure. Raised in the same Repair/TestRewrite path as 'add symbol' but signals a read-side problem.","triggerScenarios":"After symbols.Next() returns false, symbols.Err() is non-nil because reading the symbol table slice from the source index file failed (bad offset, truncated file, CRC/decode failure in the symbols section).","commonSituations":"Repairing blocks from a node that crashed mid-write; blocks copied with rsync/scp interrupted leaving a truncated index file; filesystem corruption after power loss; promtool tsdb repair on damaged historical blocks.","solutions":["Treat the source block as corrupt: restore it from backup, another replica, or re-fetch via Thanos/Cortex compactor before attempting rewrite.","Run promtool tsdb analyze on the block to confirm the index file is unreadable, then delete/quarantine the block so TSDB can repair metadata.","Check for truncated 'index' files (compare size against meta.json expectations) and re-copy the block.","If the error persists on valid-looking blocks, check for Prometheus/storage version mismatch producing an index layout the reader cannot decode."],"exampleFix":"// before: blindly repairing a possibly truncated block\nif err := block.Rewrite(ctx, meta, src, dst, indexr, chunkr, nil); err != nil { return err }\n// after: validate index file integrity first\nfi, err := os.Stat(filepath.Join(src, \"index\"))\nif err != nil || fi.Size() < minIndexSize { return fmt.Errorf(\"index truncated (%d bytes), restore block\", fi.Size()) }\nreturn block.Rewrite(ctx, meta, src, dst, indexr, chunkr, nil)","handlingStrategy":"validation","validationCode":"fi, err := os.Stat(filepath.Join(blockDir, \"index\"))\nif err != nil { return err }\nif fi.Size() == 0 { return fmt.Errorf(\"index file truncated/empty: %s\", blockDir) }","typeGuard":"func isSymbolIterError(err error) bool { return err != nil && strings.Contains(err.Error(), \"next symbol\") }","tryCatchPattern":"if err := block.Rewrite(ctx, meta, src, dst, indexr, chunkr, nil); err != nil {\n    if strings.Contains(err.Error(), \"next symbol\") {\n        // source index corrupt: restore from replica/backup instead of retrying\n    }\n    return err\n}","preventionTips":["Never interrupt block copies (rsync/scp) between nodes; verify file sizes afterwards","Restore blocks from replicas rather than rewriting obviously truncated index files","Run promtool tsdb analyze periodically to detect corrupt blocks early","Use journaled/fsync-safe storage for TSDB data directories"],"tags":["tsdb","index","corruption","repair"],"backgroundTag":"file-read-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}