gohugoio/hugo · error

Error reading blob header\n

Error message

Error reading blob header\n

What it means

Printed by genavif's handle_commands (avif.c:319) when fread cannot read the full 16-byte blob header (magic 'TAK35EM1' + uint32 id + uint32 size) that must follow each JSON command line. A short or zero read means the stdin stream ended early, was closed, or is out of sync because a previous command consumed the wrong number of bytes. The worker jumps to cleanup, abandoning this iteration without writing a response.

Source

Thrown at internal/warpc/genavif/avif.c:319

        uint32_t blob_size = 0;

        // Remove newline character if present
        line[strcspn(line, "\n")] = 0;

        if (strlen(line) == 0)
        {
            continue;
        }

        input = parse_input_message(line);

        // Next in stream is a blob header defined in https://github.com/bep/textandbinaryreader
        // T', 'A', 'K', '3', '5', 'E', 'M', '1' id uint32, size uint32
        uint8_t blob_header[16];
        size_t read_bytes = fread(blob_header, 1, sizeof(blob_header), stream);
        if (read_bytes != sizeof(blob_header))
        {
            fprintf(stderr, "Error reading blob header\n");
            goto cleanup;
        }
        uint32_t blob_id = *(uint32_t *)&blob_header[8];
        blob_size = *(uint32_t *)&blob_header[12];
        blob_data = malloc((size_t)blob_size);
        if (blob_data == NULL)
        {
            // Out of memory. Drain the blob from the input stream so the next
            // command stays aligned, then report the error to the client instead
            // of leaving the stream corrupted with no response.
            drain_bytes(stream, (size_t)blob_size);
            OutputMessage err_output = {0};
            err_output.header = input.header;
            snprintf(err_output.header.err, sizeof(err_output.header.err),
                     "out of memory allocating %u bytes for blob data", blob_size);
            write_output_message(&err_output);
            goto cleanup;
        }

View on GitHub (pinned to 52c9bd7908)

Solutions

1. Treat this on the Go side as a fatal worker state: tear down the genavif subprocess and spawn a fresh one rather than reusing the desynchronized pipe.
2. Confirm the host writes JSON line + 16-byte header + blob payload atomically per command and that no short writes are masked by retries that drop bytes.
3. Audit the prior command's blob_size field — a wrong size there desyncs every subsequent read.
4. In tests, always close stdin only after the full blob has been written.
5. Add a read-side timeout in the Go warpc client so a stuck worker is detected instead of looking like a clean EOF.

Example fix

// before (Go host): wrote JSON then context cancelled before blob
w.Write(jsonLine)
blob, err := readPixels(ctx)
if err != nil { return err } // blob never sent, worker hits short header read next loop
w.Write(blob)

// after: write JSON + header + blob as one buffered unit, or skip the command entirely on error
if err != nil { return err }
w.Write(fullFrame) // jsonLine + 16-byte header + payload, written together

// Write the full request (JSON line + 16-byte header + blob) atomically; if any
// step fails, discard the worker and start a new one.
func writeRequest(w io.Writer, cmd []byte, blobID uint32, blob []byte) error {
    if !bytes.HasSuffix(cmd, []byte{'\n'}) { cmd = append(cmd, '\n') }
    hdr := make([]byte, 16)
    copy(hdr[0:8], []byte("TAK35EM1"))
    binary.LittleEndian.PutUint32(hdr[8:12], blobID)
    binary.LittleEndian.PutUint32(hdr[12:16], uint32(len(blob)))
    frame := append(append(cmd, hdr...), blob...)
    n, err := w.Write(frame)
    if err != nil || n != len(frame) { return errShortWrite }
    return nil
}

When it happens

Trigger: EOF reached on stdin (host closed the pipe after the JSON line but before sending the blob), a previous iteration left the stream misaligned so what should be the next blob header is actually mid-pixel-data, or the host crashed / was killed between writing the JSON line and the blob header. Reading from a closed file or a truncated test fixture also triggers it.

Common situations: Host process killed by context cancellation mid-request, broken pipe silently swallowed by the Go writer, test fixtures that send a JSON line with no following blob, or a prior 'Error reading blob data' that left the stream pointer in the wrong place so the next iteration reads pixel bytes as the magic header.

Related errors

• [%d] Error reading blob data (size: %llu read: %zu) \n
• Error reading blob header\n
• Error parsing JSON line\n
• [%d] Error reading blob data (size: %llu read: %zu) \n
• Error: Line did not parse to a valid JSON object\n

AI-assisted analysis of gohugoio/hugo@52c9bd7908 (2026-08-09). Data as JSON: /api/errors/4fa48ebcf1a60b45. Report an issue: GitHub.