stamparm/maltrail · error · ValueError

not a Maltrail provenance sidecar (bad magic)

Error message

not a Maltrail provenance sidecar (bad magic)

What it means

The provenance sidecar file's header magic does not match _MAGIC, so __init__ refuses to treat it as a Maltrail provenance sidecar. The file exists and mmaps fine, but is not the expected binary format (wrong file, old/corrupt format, or version mismatch).

Solutions

1. Point the reader at the correct provenance sidecar file, not the trail bin
2. Regenerate the sidecar with the current library version
3. Check the file was fully transferred (compare size/checksum) and is not a placeholder
4. Verify no earlier writer produced the file with a different format

Example fix

// before
prov = ProvenanceSidecar("trails.bin")
// after
prov = ProvenanceSidecar("trails.bin.prov")

def sidecar_looks_valid(path):
    import struct, os
    with open(path, "rb") as f:
        magic = f.read(8)  # size of _MAGIC as packed
    return magic == expected_magic_bytes

def is_probable_sidecar(path):
    import os
    return os.path.isfile(path) and os.path.getsize(path) > 16

try:
    prov = ProvenanceSidecar(path)
except ValueError as e:
    if "bad magic" in str(e):
        prov = regenerate_sidecar(path)

When it happens

Trigger: Opening a non-sidecar file (log, tarball, different bin format) via the sidecar reader; a sidecar written by an incompatible older/newer version with a changed magic.

Common situations: Wrong path in config pointing at trails.bin instead of its provenance sidecar; partially written/zero-length placeholder file; sidecar regenerated by a different tool version.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• bad trail bin magic
• provenance sidecar is truncated
• trail bin too small
• [x] invalid IP address
• packet too short for header-protection sample

AI-assisted analysis of stamparm/maltrail@77cfb06d76 (2026-09-13). Data as JSON: /api/errors/4408d8b7bd51210e. Report an issue: GitHub.

Appendix: source

Thrown at core/provenance.py:83

    os.replace(tmp, path)
    return len(rows), len(pairs)


class Provenance(object):
    """An opened sidecar. Read-only, mmap'd, safe to share between request threads."""

    def __init__(self, path):
        self._file = open(path, "rb")
        try:
            self._map = mmap.mmap(self._file.fileno(), 0, access=mmap.ACCESS_READ)
        except Exception:
            self._file.close()
            raise

        magic, count, table_len = _HEADER.unpack_from(self._map, 0)
        if magic != _MAGIC:
            self.close()
            raise ValueError("not a Maltrail provenance sidecar (bad magic)")

        self.count = count
        self._pairs = json.loads(self._map[_HEADER.size:_HEADER.size + table_len].decode("utf8"))
        self._base = _HEADER.size + table_len

        if self._base + count * _ENTRY_SIZE > len(self._map):
            self.close()
            raise ValueError("provenance sidecar is truncated")

    def _hash_at(self, i):
        offset = self._base + i * _ENTRY_SIZE
        return struct.unpack_from("<Q", self._map, offset)[0]

    def lookup(self, trail):
        """(reference, source_path) for `trail`, or None.

        The order matches what core/httpd.py's on-demand scan returned, so the caller does not care
        which of the two answered.

View on GitHub (pinned to 77cfb06d76)