ArchiveBox/ArchiveBox · critical · SystemExit

[X] OSError: Failed to write {path} with fcntl.F_FULLFSYNC.

Error message

[X] OSError: Failed to write {path} with fcntl.F_FULLFSYNC. ({e})

What it means

atomic_write performs an fsync'd atomic write using fcntl F_FULLFSYNC (macOS/BSD). When the fsync syscall fails (e.g. filesystem doesn't support it, like some network shares/exFAT), the writer retries without F_FULLFSYNC only after printing the guidance that the main data dir must support FSYNC — the message with this text is the OSError report printed before that fallback/exit path, and it raises SystemExit(1) when writes can't be made durable.

Source

Thrown at archivebox/misc/system.py:41

    encoding = None if isinstance(contents, bytes) else "utf-8"  # enforce utf-8 on all text writes

    try:
        with lib_atomic_write(path, mode=mode, overwrite=overwrite, encoding=encoding) as f:
            if isinstance(contents, dict):
                dump(contents, f, indent=4, sort_keys=True, cls=ExtendedEncoder)
            elif isinstance(contents, (bytes, str)):
                f.write(contents)
    except OSError as e:
        config = config or get_config(**config_kwargs)
        if config.ENFORCE_ATOMIC_WRITES:
            print(f"[X] OSError: Failed to write {path} with fcntl.F_FULLFSYNC. ({e})")
            print(
                "    You can store the archive/ subfolder on a hard drive or network share that doesn't support support synchronous writes,",
            )
            print(
                "    but the main folder containing the index.sqlite3 and ArchiveBox.conf files must be on a filesystem that supports FSYNC.",
            )
            raise SystemExit(1)

        # retry the write without forcing FSYNC (aka atomic mode)
        with open(path, mode=mode, encoding=encoding) as f:
            if isinstance(contents, dict):
                dump(contents, f, indent=4, sort_keys=True, cls=ExtendedEncoder)
            elif isinstance(contents, (bytes, str)):
                f.write(contents)

    # set file permissions
    config = config or get_config(**config_kwargs)
    os.chmod(path, int(config.OUTPUT_PERMISSIONS, base=8))


@enforce_types
def get_dir_size(path: str | Path, recursive: bool = True, pattern: str | None = None) -> tuple[int, int, int]:
    """get the total disk size of a given directory, optionally summing up
    recursively and limiting to a given filter list
    """

View on GitHub (pinned to 74564b2822)

Solutions

1. Move DATA_DIR to a local filesystem that supports fsync (APFS/ext4); keep only archive/ on the external share.
2. Update macOS/Docker/VM so F_FULLFSYNC is supported by the underlying mount.
3. If only the fallback path matters, ensure the code's retry-without-FSYNC branch can write (check remaining permissions/space) — but note index/SQLite dirs must support FSYNC.
4. Report/inspect the underlying errno `e` in the message to identify the exact failing operation.

Example fix

// before
DATA_DIR=/Volumes/EXFAT_DRIVE/archivebox   # F_FULLFSYNC unsupported
// after
DATA_DIR=/Users/me/archivebox/data          # local APFS, fsync OK
# keep /Volumes/EXFAT_DRIVE/archivebox/archive as an output dir instead

import fcntl
def fs_supports_fsync(dir_path: str) -> bool:
    import os, tempfile
    fd, tmp = tempfile.mkstemp(dir=dir_path)
    try:
        fcntl.fsync(fd)          # plain fsync
        try:
            fcntl.fcntl(fd, fcntl.F_FULLFSYNC)  # will raise on Linux/unsupported
        except (AttributeError, OSError):
            pass  # F_FULLFSYNC unavailable; atomic_write falls back
        return True
    except OSError:
        return False
    finally:
        os.close(fd); os.remove(tmp)

try:
    atomic_write(path, contents)
except SystemExit as e:
    if e.code == 1:
        log.critical("data dir filesystem does not support FSYNC; move DATA_DIR to a local fs")
    raise
except OSError as e:
    log.critical(f"write failed: {e}")
    raise

When it happens

Trigger: Calling atomic_write (directly or via write_config_file / write_json_details / write_html_details / _write_file_if_changed) with the data dir located on a filesystem where fcntl(fd, F_FULLFSYNC) fails: SMB/NFS shares, exFAT/FAT32 drives, certain virtualized mounts. Called during `archivebox init`, config updates, and snapshot detail writes.

Common situations: macOS users pointing DATA_DIR at an exFAT external drive or SMB network share; Docker volumes on filesystems lacking F_FULLFSYNC; moving index.sqlite3/ArchiveBox.conf storage to a NAS.

Related errors

• Refusing to delete output path outside DATA_DIR: {path}
• Migration would overwrite a different output: {destination}
• Migration cannot safely copy special output: {source}
• Migration incomplete: {source.relative_to(old_dir)}
• [X] No archivebox index found in the current directory.

AI-assisted analysis of ArchiveBox/ArchiveBox@74564b2822 (2026-08-28). Data as JSON: /api/errors/87de008d07a334bf. Report an issue: GitHub.