benbjohnson/litestream · error

sync hydration meta directory

Error message

sync hydration meta directory: %w

What it means

Raised by the VFS hydrator when fsync-ing the directory that contains the hydration meta file fails after renaming a temporary meta file into place. The rename succeeded but the directory entry could not be flushed to durable storage, so a crash could leave the hydration state file missing or stale. The library throws this to guarantee the meta file's existence is durable before proceeding.

Solutions

1. Check the health and free space of the filesystem holding the hydration directory (dmesg for I/O errors)
2. Verify the hydration directory still exists and the process has write/search permissions on it
3. Move the hydration path to a reliable local filesystem that supports fsync (not NFS/overlayfs)
4. Retry open; if persistent, disable hydration (unset hydrationPath) and continue without it

Example fix

// before
if err := syncDir(filepath.Dir(h.metaPath())); err != nil {
    return fmt.Errorf("sync hydration meta directory: %w", err)
}
// after
if err := syncDir(filepath.Dir(h.metaPath())); err != nil {
    h.logger.Warn("hydration meta dir sync failed, hydration may be stale after crash", "error", err)
    // or relocate hydrationPath to local durable storage
}

// Go: verify hydration dir is usable before enabling hydration
if fi, err := os.Stat(hydrationDir); err != nil || !fi.IsDir() { /* disable hydration or fix path */ }

func hydrationDirOK(dir string) bool { fi, err := os.Stat(dir); return err == nil && fi.IsDir() }

if err := file.Open(ctx); err != nil {
    if strings.Contains(err.Error(), "sync hydration meta directory") {
        logger.Warn("hydration durability failed; continuing without hydration", "error", err)
    } else { return err }
}

When it happens

Trigger: os.Open(dir) fails (directory removed/permissions) or the Sync() on the directory fd fails (EIO from disk, EROFS, ENOSPC on metadata) during hydration meta finalization.

Common situations: Disk full or failing disk on the hydration path; hydration directory on a filesystem that does not support fsync (some network/overlay mounts); permissions changed on the hydration directory mid-run.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• sync before truncate
• sync L0 file
• sync ltx dir
• sync temp meta file
• sync txid dir

AI-assisted analysis of benbjohnson/litestream@4ed7a308f6 (2026-09-06). Data as JSON: /api/errors/4f1e61ea9ced00d2. Report an issue: GitHub.

Appendix: source

Thrown at vfs.go:1024

		if removeErr := os.Remove(tmpPath); removeErr != nil {
			h.logger.Warn("failed to remove temp meta file during cleanup", "error", removeErr)
		}
		return fmt.Errorf("sync temp meta file: %w", err)
	}
	if err := tmp.Close(); err != nil {
		if removeErr := os.Remove(tmpPath); removeErr != nil {
			h.logger.Warn("failed to remove temp meta file during cleanup", "error", removeErr)
		}
		return fmt.Errorf("close temp meta file: %w", err)
	}
	if err := os.Rename(tmpPath, h.metaPath()); err != nil {
		if removeErr := os.Remove(tmpPath); removeErr != nil {
			h.logger.Warn("failed to remove temp meta file during cleanup", "error", removeErr)
		}
		return fmt.Errorf("rename hydration meta: %w", err)
	}
	if err := syncDir(filepath.Dir(h.metaPath())); err != nil {
		return fmt.Errorf("sync hydration meta directory: %w", err)
	}
	return nil
}

func syncDir(path string) error {
	dir, err := os.Open(path)
	if err != nil {
		return err
	}
	defer dir.Close()
	return dir.Sync()
}

func NewVFSFile(client ReplicaClient, name string, logger *slog.Logger) *VFSFile {
	f := &VFSFile{
		client:       client,
		name:         name,
		index:        make(map[uint32]ltx.PageIndexElem),

View on GitHub (pinned to 4ed7a308f6)