stamparm/maltrail · error · ValueError

provenance sidecar is truncated

Error message

provenance sidecar is truncated

What it means

The sidecar header declares `count` entries, but the mapped file ends before the full entry table (header + table + count*_ENTRY_SIZE), so the file is rejected as truncated. This guards reads of hash entries past the end of the mapping.

Solutions

1. Regenerate or re-download the sidecar completely
2. Compare file size against the header-declared size before opening
3. Validate with the producing tool's integrity command/checksum
4. Catch ValueError on open and fall back to a fresh build

Example fix

// before
prov = ProvenanceSidecar(path)
// after
try:
    prov = ProvenanceSidecar(path)
except ValueError:
    prov = rebuild_sidecar(path)

import os
def sidecar_size_ok(path):
    return os.path.getsize(path) > 64  # at least header + table for nonzero entries

try:
    prov = ProvenanceSidecar(path)
except ValueError:
    prov = rebuild_sidecar(path)  # truncated -> rebuild

When it happens

Trigger: Incomplete download/copy of the sidecar; writer crashed or was killed mid-flush; disk full during generation; file truncated by log rotation/cleanup tools.

Common situations: rsync/scp interrupted, partial cache restore from backup, sidecar written while the process was OOM-killed.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• not a Maltrail provenance sidecar (bad magic)
• bad trail bin magic
• truncated trail bin (have , need )
• [x] invalid IP address
• packet too short for header-protection sample

AI-assisted analysis of stamparm/maltrail@77cfb06d76 (2026-09-13). Data as JSON: /api/errors/9e4f652bc6c7910d. Report an issue: GitHub.

Appendix: source

Thrown at core/provenance.py:91

        self._file = open(path, "rb")
        try:
            self._map = mmap.mmap(self._file.fileno(), 0, access=mmap.ACCESS_READ)
        except Exception:
            self._file.close()
            raise

        magic, count, table_len = _HEADER.unpack_from(self._map, 0)
        if magic != _MAGIC:
            self.close()
            raise ValueError("not a Maltrail provenance sidecar (bad magic)")

        self.count = count
        self._pairs = json.loads(self._map[_HEADER.size:_HEADER.size + table_len].decode("utf8"))
        self._base = _HEADER.size + table_len

        if self._base + count * _ENTRY_SIZE > len(self._map):
            self.close()
            raise ValueError("provenance sidecar is truncated")

    def _hash_at(self, i):
        offset = self._base + i * _ENTRY_SIZE
        return struct.unpack_from("<Q", self._map, offset)[0]

    def lookup(self, trail):
        """(reference, source_path) for `trail`, or None.

        The order matches what core/httpd.py's on-demand scan returned, so the caller does not care
        which of the two answered.
        """

        target = trail_hash(trail)
        lo, hi = 0, self.count
        while lo < hi:
            mid = (lo + hi) // 2
            if self._hash_at(mid) < target:
                lo = mid + 1

View on GitHub (pinned to 77cfb06d76)