thanos-io/thanos · error

failed to sync / blocks

Error message

failed to sync %v/%v blocks

What it means

Summary error returned at the end of Sync when at least one upload failed or a corrupted block was detected (uploadErrs > 0 or failedBlocks > 0) and out-of-order uploads are enabled (errors were logged instead of returned immediately). Note the count formatting shows uploadErrs as the first number and len(failedBlocks) as the denominator, which can be misleading; consult the per-block 'shipping failed' log lines for the actual failing ULIDs.

Solutions

1. Search sidecar logs for 'shipping failed' and 'corrupted block' entries to find the exact block ULIDs.
2. For corrupted blocks, verify the block in the bucket and delete/restore the local copy (Prometheus will re-compact).
3. For transient storage errors, wait for the next sync cycle — failed uploads are automatically retried.
4. If failures persist, inspect the object store health and increase timeouts/retries in the objstore config.

Example fix

// before
// failed to sync 2/1 blocks   (misleading counts; check logs)
// after
// $ kubectl logs thanos-sidecar | grep 'shipping failed'
// # identify ULID, then: thanos tools bucket verify --id=<ULID> and repair local data

// before relying on the summary, capture per-block results
_, err := shipper.Sync(ctx)
if err != nil && strings.Contains(err.Error(), "failed to sync") {
    log.Printf("partial sync; check 'shipping failed' log lines for ULIDs")
}

ticker := time.NewTicker(syncInterval)
for range ticker.C {
    if _, err := shipper.Sync(ctx); err != nil {
        log.Printf("sync incomplete, will retry next tick: %v", err)
    }
}

When it happens

Trigger: Sync completes its loop with allowOutOfOrderUploads=true and one or more s.upload() calls failed, or blocks were detected as corrupted (missing/invalid segment files) during iteration.

Common situations: Flaky object storage connectivity during a sync cycle; corrupted blocks on disk (e.g. after unclean shutdown or disk errors); persistent upload failures retried every sync interval.

Related errors

• upload
• upload downsampled block
• get all block meta.
• first pass of downsampling failed
• download block

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/246e446ec873350a. Report an issue: GitHub.

Appendix: source

Thrown at pkg/shipper/shipper.go:450

			// No error returned, just log line. This is because we want other blocks to be uploaded even
			// though this one failed. It will be retried on second Sync iteration.
			level.Error(s.logger).Log("msg", "shipping failed", "block", m.ULID, "err", err)
			uploadErrs++
			continue
		}
		meta.Uploaded = append(meta.Uploaded, m.ULID)
		uploaded++
		s.metrics.uploads.Inc()
	}
	if err := WriteMetaFile(s.logger, s.metadataFilePath, meta); err != nil {
		level.Warn(s.logger).Log("msg", "updating meta file failed", "err", err)
	}

	failedExecution = false
	if uploadErrs > 0 || len(failedBlocks) > 0 {
		s.metrics.uploadFailures.Add(float64(uploadErrs))
		s.metrics.corruptedBlocks.Add(float64(len(failedBlocks)))
		return uploaded, errors.Errorf("failed to sync %v/%v blocks", uploadErrs, len(failedBlocks))
	}

	if s.uploadCompacted {
		s.metrics.uploadedCompacted.Set(1)
	} else {
		s.metrics.uploadedCompacted.Set(0)
	}
	return uploaded, nil
}

func (s *Shipper) UploadedBlocks() map[ulid.ULID]struct{} {
	meta, err := ReadMetaFile(s.metadataFilePath)
	if err != nil {
		// NOTE(GiedriusS): Sync() will inform users about any problems.
		return nil
	}

	ret := make(map[ulid.ULID]struct{}, len(meta.Uploaded))

View on GitHub (pinned to 35b8b99117)