Investigating the audit log

How to verify the integrity of Tavrik’s audit log and what to do when verification fails. Written for the pilot deployment shape (on-prem VM runbook); every command below runs on the Tavrik host.

What exists today, and what does not

Read this first. Earlier documents described a daily Merkle seal and an audit.seal_verify_mismatch alert; neither exists.

Shipped (verified against main on 2026-09-02):

• The audit log is an append-only JSONL file (/var/lib/tavrik-audit/audit.jsonl). Every event carries hash (SHA-256 over a canonical form of the event, ADR-0026) and prev_hash (the hash of the previous event written by the same writer process). Editing an event changes its hash; the next event’s prev_hash then no longer matches.
• POST /v1/audit/verify on the admin API (read:audit, every role) takes {"event_ids": [...]} — up to 1000 ids, in chain order, oldest first — recomputes each event’s hash and checks each prev_hash link, and returns {"verified": true} or {"verified": false, "broken_at": "<event_id>", "expected_hash": …, "actual_hash": …}. An id that is not in the file returns verified: false with broken_at set to it. An empty list verifies nothing and returns verified: true — a request body of {} is not a check.
• The console /audit page’s Verify button verifies the hash of the one selected event (it posts a single id). It does not walk the chain.
• Every call to the verify endpoint is itself audited (admin.audit_accessed with endpoint: verify, the requested count and the result).

Not yet available:

• The daily Merkle seal scheduler, POST /v1/audit/seal/verify, and the audit.seal_created / audit.seal_verified / audit.seal_verify_mismatch events. The event types are declared in the closed enum and the seal engine exists in internal/audit/seal, but nothing schedules a seal, no route verifies one, and no code path emits those events. Tracked as M3.7.B in the post-pilot register of ../healthcare/pilot-blockers.md. Do not put audit.seal_verify_mismatch in a SIEM alert rule; it will never fire.
• A one-call “verify the whole file” operation. Verification is over lists you construct (procedure below).
• An external trust anchor (transparency log, signed checkpoints, customer-side mirror). Without one, truncation of the tail of the file is not detectable from the file alone; the off-host copy in step 1 is the mitigation until scheduled backups land (B14).
• A gateway-admin CLI. gateway-admin is the admin API server binary; all operator actions are HTTP calls or SQL.

Two writers, two chains. On the VM shape both tavrik-gateway and tavrik-gateway-admin append to the same file (GATEWAY_AUDIT_FILE, runbook §4). The chain is per writer process (ADR-0026 §“Hash-chain semantics”): each process seeds from the file’s last line when it starts and then links only to its own events. The file therefore holds interleaved chains, and a slice of consecutive lines will not verify as one chain. Each service start also creates a fork — a new chain whose first prev_hash points at whatever the last line was at that moment. Forks are expected; one per service start. The procedure below reconstructs chains by following links, which makes the interleaving irrelevant.

When to use this runbook

• Quarterly compliance review and the design-partner sign-off (onboarding.md §Sign-off).
• After any host-level incident (unexpected reboot, disk full, unplanned access to the host, restore from backup).
• Any external signal that the audit trail may have been altered.
• A verified: false from any verify call, or a console Verify showing a broken hash.

Severity / SLA

Finding	Severity	Response
verified: false with expected_hash ≠ actual_hash on an event that exists	P0 — content of a logged event differs from what was written	Immediate; preserve evidence, engage incident response, notify the customer’s compliance lead the same day
A dangling prev_hash (points at a hash no event in the file has)	P0 — an event was removed or rewritten	Same
A fork with no matching service start in the journal	P2 — unexplained; may indicate a file restored from an older copy	1 business day
Forks that match service starts; unhashed lines only at the head of the file	Expected	None

Pre-flight checks

KEY=<an operator API key>            # any role; verify needs read:audit
A=http://127.0.0.1:9090              # admin API, loopback only on this shape
F=/var/lib/tavrik-audit/audit.jsonl
sudo ls -l $F && sudo wc -l $F
systemctl show -p ActiveEnterTimestamp tavrik-gateway tavrik-gateway-admin
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $KEY" "$A/v1/audit/events?limit=1"   # expect 200

A 404 from the audit endpoints means the admin API is not running with GATEWAY_AUDIT_BACKEND=file; the routes register only for the file backend.

Steps

1. Preserve evidence before anything else

D=/var/tmp/audit-investigation-$(date -u +%Y%m%dT%H%M%SZ); sudo mkdir -p $D
sudo cp -a $F $D/audit.jsonl && sudo sha256sum $D/audit.jsonl | sudo tee $D/audit.jsonl.sha256
sudo stat $F | sudo tee $D/stat.txt
sudo journalctl -u tavrik-gateway -u tavrik-gateway-admin -o short-iso --since '-30d' | grep -iE 'started|stopped|audit' | sudo tee $D/service-starts.txt

Copy $D off the host (to the customer’s evidence store) before continuing. Do not stop the services: the log keeps chaining, and stopping them creates one more fork.

2. Reconstruct the chains

Save the script below as /tmp/verify-chains.py. It follows prev_hash links through the whole file, reports forks and dangling links on stderr, and prints one verify-request body per chain slice (≤ 1000 ids, oldest first) on stdout. It reads only event_id, hash and prev_hash; it does not compute hashes — the admin API does that in step 3.

#!/usr/bin/env python3
"""Partition an audit JSONL file into hash chains by following prev_hash
links; report dangling links and forks on stderr; print one
POST /v1/audit/verify body per chain slice (max 1000 ids) on stdout."""
import json
import sys

GENESIS = "0" * 64
path = sys.argv[1]
chains = []   # each chain: list of events in link order
tails = {}    # hash of an open chain's last event -> chain index
seen = {}     # every hash in the file -> its event_id
forks, dangling, unhashed = [], [], 0

with open(path, encoding="utf-8") as f:
    for n, line in enumerate(f, 1):
        line = line.strip()
        if not line:
            continue
        e = json.loads(line)
        h, p = e.get("hash"), e.get("prev_hash")
        if not h:
            unhashed += 1  # written before hash chaining shipped
            continue
        if p in (None, "", GENESIS):
            chains.append([e])
            tails[h] = len(chains) - 1
        elif p in tails:
            i = tails.pop(p)
            chains[i].append(e)
            tails[h] = i
        elif p in seen:
            forks.append((n, e["event_id"], seen[p]))
            chains.append([e])
            tails[h] = len(chains) - 1
        else:
            dangling.append((n, e["event_id"], p))
            chains.append([e])
            tails[h] = len(chains) - 1
        seen[h] = e["event_id"]

total = sum(len(c) for c in chains)
print(f"events={total} unhashed={unhashed} chains={len(chains)} "
      f"forks={len(forks)} dangling={len(dangling)}", file=sys.stderr)
for n, eid, parent in forks:
    print(f"fork     line {n} {eid}: parent {parent} already has a child "
          f"(expect one per service start)", file=sys.stderr)
for n, eid, p in dangling:
    print(f"DANGLING line {n} {eid}: prev_hash {p[:16]}... is not the hash "
          f"of any event in the file", file=sys.stderr)
for c in chains:
    ids = [e["event_id"] for e in c]
    for i in range(0, len(ids), 1000):
        print(json.dumps({"event_ids": ids[i:i + 1000]}))

Run it against the preserved copy:

sudo python3 /tmp/verify-chains.py $D/audit.jsonl > /tmp/verify-batches.jsonl

Read the stderr summary before going on. dangling=0 is required. unhashed should be zero unless the file predates the hash chain, in which case the unhashed lines are the oldest lines only. The number of chains is roughly the number of service starts since the file began, plus one.

3. Verify every chain slice with the admin API

while read -r body; do
  curl -s -X POST -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
    -d "$body" "$A/v1/audit/verify"; echo
done < /tmp/verify-batches.jsonl | sort | uniq -c

Expected output is a single line: N {"verified":true} where N is the number of batches. Any "verified":false line names the first failing event in broken_at; find it with sudo grep -n '"event_id":"<id>"' $F.

Each call scans the whole file server-side; on a file of tens of thousands of events, a few dozen calls take well under a minute. Each call also appends an admin.audit_accessed event, so the file grows during the check — run the script against the preserved copy, as above, not the live file.

4. Interpret

Result	Meaning	Action
All batches verified: true, dangling=0, every fork matches a service start in $D/service-starts.txt	Intact for everything the chain can attest	Record the summary line, the batch count and the file checksum as the evidence artefact
verified: false, expected_hash ≠ actual_hash	The named event’s content differs from what produced its hash	P0. The event and everything after it in that chain is suspect
DANGLING reported	The event that the named event linked to is gone or rewritten	P0. Compare against the most recent off-host copy to identify what changed
Fork with no service start near its timestamp	Something re-seeded a writer: a restored older file, a second process pointed at the file, or manual editing followed by a restart	P2 until explained
unhashed lines in the middle of the file	Lines written without a hash after chaining was enabled — not something the shipped writers do	Treat as tampering until explained

What the chain cannot detect: removal of events from the very end of the file, and a wholesale rewrite by someone with write access to the host and the patience to re-hash every event (the canonical form is public). The off-host copy from step 1, compared line-for-line against the current file (comm -23 <(sort old) <(sort new)), is the control for both until an external anchor exists.

5. Contain and escalate (P0 only)

1. Treat the host as compromised until shown otherwise: engage the customer’s incident-response process and the Tavrik product team (onboarding “mutual escalation”).
2. Keep the services running so that subsequent activity is still logged, but move the preserved copy and the verification output off the host immediately.
3. Rotate operator API keys and the bootstrap key after the host is confirmed clean; every use of the audit read endpoints is itself in the log (admin.audit_accessed), so the reads made during this investigation will be visible in the trail.

Verification

The investigation is complete when step 3 returns only verified: true, dangling=0, and every fork is matched to a service start — or, on a positive finding, when incident response has taken ownership of the preserved evidence and the finding is logged in the customer’s IR system.

Communication

• Clean quarterly run: file the summary line, batch count and checksum with the compliance evidence for the quarter.
• P0 finding: the customer’s compliance / privacy lead and Tavrik product support on the same day, with $D (copied off-host) as the attachment. Breach assessment under HIPAA §164.402 is the customer’s determination; Tavrik supplies the technical facts.

Post-mortem trigger

Any P0 finding, and any P2 fork that cannot be explained within one business day.

References

• ADR-0026 — audit-read endpoints and the per-event hash chain (§“Hash-chain semantics” for the per-writer rule)
• docs/operations/admin-api.md §POST /v1/audit/verify
• docs/operations/on-prem-vm-runbook.md §7.4 and §10 — the quick check used at deploy time
• docs/healthcare/pilot-blockers.md — post-pilot register: seal scheduler (M3.7.B), scheduled backups (B14)