Operating the connectors

Day-to-day operational runbook for the two EHR-adjacent audit-forwarding connectors, hl7_mirth_mllps (HL7 v2.x ARV over MLLPS, M5.A.8) and fhir_audit_https (FHIR R4 AuditEvent over HTTPS, M5.A.9), on the pilot deployment shape (on-prem VM runbook). Every command runs on the Tavrik host against the loopback admin API and the Postgres container.

What exists today, and what does not

Read this first; an earlier version of this runbook described a CLI and surfaces that do not exist.

Shipped (verified against main on 2026-09-02):

Admin API (docs/operations/admin-api.md): GET/PUT/DELETE /v1/tenants/{id}/connectors/{kind} and PUT …/connectors/{kind}/policy ({"mode": "enabled" | "disabled"}). GET returns mode, has_credential, display_name and timestamps — never the credential material. The console page /tenants/[id]/connectors fronts the same calls for the two healthcare kinds.
The dispatcher in gateway-admin (ADR-0046): the audit file is the outbox; per-(tenant, connector) cursors in connector_cursors; failed events park in connector_retry with exponential backoff (1 s doubling, capped at 5 min); after 10 attempts they move to connector_dlq. Config changes are picked up on the next 5 s tick (senders are cached by a fingerprint of the config, so a changed credential rebuilds the transport).
Audit events: connector.event_forwarded, connector.event_failed, connector.mllps_handshake_failed (failure_class: cert_expired / unknown_authority / connection_refused / timeout / unknown), connector.fhir_endpoint_unreachable (failure_class: dns_failure / connection_refused / cert_invalid / cert_expired / unknown_authority / auth_rejected / server_error / client_error / timeout / tls_handshake / unknown), connector.client_cert_expiring (both writers, when an mTLS client cert is configured and within 14 days of NotAfter; one per 24 h per writer instance), connector.config_created / _updated / _deleted, connector.policy_updated.
FHIR forwarding is verified live (2026-09-02 demo). MLLPS forwarding has never been exercised against a real integration engine (pilot-blocker B12); run the acceptance gate before relying on it.

Not yet available:

A gateway-admin connector … / gateway-admin audit … CLI. gateway-admin is the admin API server binary and has no subcommands. Everything below is curl or SQL.
~~A connector status endpoint and a DLQ replay endpoint~~ — shipped 2026-09-02 (ADR-0046 amendment, B5): GET …/connectors/{kind}/status, POST …/dlq/replay, POST …/dlq/discard, POST …/cursor/rewind, and start_from on the policy PUT. The SQL reads below remain valid for forensics; the endpoints are the operator path.
The connector.event_dlq audit event. It is declared but never emitted; a DLQ transition shows up only as a dispatch: event moved to DLQ line in the admin journal and a row in connector_dlq. Do not alert on connector.event_dlq.
Diagnostic aggregation: an endpoint that is down emits one fhir_endpoint_unreachable / mllps_handshake_failed per failed delivery attempt (B10b).
SMART-on-FHIR token refresh: the FHIR connector accepts a static bearer token (or mTLS) only.
Starting delivery from the tail: enabling a connector for the first time delivers from the beginning of the audit file (#302, B10a).
Seal-based audit verification: see audit-log-tamper-investigation.md for what exists.

When to use this runbook

Client cert rotation (operator-driven; every ~12 months or per customer policy)
CA bundle rotation (hospital integration engine’s TLS server cert chain changed)
Integration engine outage handling
FHIR endpoint URL drift / bearer token rotation
Recurring connector.event_failed / connector.mllps_handshake_failed / connector.fhir_endpoint_unreachable events, or growth of connector_retry / connector_dlq
Quarterly audit-log verification (companion: audit-log-tamper-investigation.md)

Severity / SLA

Trigger	Severity	Target response
`connector_dlq` row count growing	P2	1 business day
`connector.client_cert_expiring` (`days_until_expiry` < 7)	P2	1 business day
`connector.client_cert_expiring` (`days_until_expiry` < 3)	P1	4 hours
`connector.mllps_handshake_failed` sustained	P1	4 hours
`connector.fhir_endpoint_unreachable` sustained	P1	4 hours
`POST /v1/audit/verify` returns `verified: false`	P0	Immediate; engage `audit-log-tamper-investigation.md`

Conventions used below

A=http://127.0.0.1:9090                 # admin API (loopback; runbook §1)
KEY=<operator API key>                  # role operator or root for writes (manage:connectors); viewer can read
T=<tenant uuid>                         # from: curl -s -H "Authorization: Bearer $KEY" $A/v1/tenants
K=fhir_audit_https                      # or hl7_mirth_mllps
H='-H "Authorization: Bearer $KEY" -H "Content-Type: application/json"'
psql() { sudo docker exec -i tavrik-postgres psql -U postgres -d ai_esb "$@"; }

A 403 on any call means the key’s role lacks the action; a 400 with kek_ref is required means GATEWAY_BYOK_DEFAULT_KEK_REF is unset on the host (runbook §4).

Pre-flight checks

Before any procedure:

# 1. Connector config + policy state (mode, has_credential, updated_at)
curl -s -H "Authorization: Bearer $KEY" $A/v1/tenants/$T/connectors/$K

# 2. Recent connector audit events (newest first) — or console /audit, filter by event type
curl -s -H "Authorization: Bearer $KEY" "$A/v1/audit/events?surface=connector&tenant_id=$T&limit=50"

# 3. Delivery state: cursor, parked retries, DLQ depth
psql -c "SELECT connector, byte_offset, last_event_id, updated_at FROM connector_cursors WHERE tenant_id = '$T';"
psql -c "SELECT connector, count(*) AS parked, min(next_attempt_at) AS next_due, max(attempts) AS max_attempts FROM connector_retry WHERE tenant_id = '$T' GROUP BY connector;"
psql -c "SELECT connector, count(*) AS dlq_depth, min(failed_at), max(failed_at) FROM connector_dlq WHERE tenant_id = '$T' GROUP BY connector;"

# 4. Dispatcher journal
sudo journalctl -u tavrik-gateway-admin -o cat --since -1h | grep 'dispatch:'

Updating a connector’s configuration (used by every procedure)

PUT /v1/tenants/{id}/connectors/{kind} replaces the whole config. The validator requires every mandatory key on every save, and credentials are write-only (the API never returns them), so keep the full config in your own secret store and resend it with the changed field. On success the API emits connector.config_updated, and the dispatcher rebuilds the transport on its next tick.

MLLPS (hl7_mirth_mllps) — required: endpoint_host, endpoint_port, ca_bundle_pem, mllp_application_id, mllp_facility, receiving_application, receiving_facility; optional: client_cert_pem + client_key_pem (both or neither), tls_min_version (only 1.3 accepted), version_id (only 2.5). tls_enabled: false is rejected (connector_config_rejected_plaintext_mllp).

CA=$(python3 -c 'import json,sys; print(json.dumps(open(sys.argv[1]).read()))' hospital-ca.pem)
CERT=$(python3 -c 'import json,sys; print(json.dumps(open(sys.argv[1]).read()))' client-cert.pem)
CKEY=$(python3 -c 'import json,sys; print(json.dumps(open(sys.argv[1]).read()))' client-key.pem)
curl -s -X PUT -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d "{\"display_name\":\"Mirth (hospital audit)\",\"config\":{\"endpoint_host\":\"mirth.hospital.example\",\"endpoint_port\":6661,\"ca_bundle_pem\":$CA,\"client_cert_pem\":$CERT,\"client_key_pem\":$CKEY,\"mllp_application_id\":\"TAVRIK\",\"mllp_facility\":\"TAVRIK_GW\",\"receiving_application\":\"MIRTH\",\"receiving_facility\":\"HOSPITAL_AUDIT\"}}" \
  $A/v1/tenants/$T/connectors/hl7_mirth_mllps

FHIR (fhir_audit_https) — required: endpoint_url (https only; http:// is rejected with connector_config_rejected_http_scheme), auth_kind (bearer with bearer_token, or mtls with client_cert_pem + client_key_pem), ca_bundle_pem, observer_id; optional: observer_site, timeout_seconds, tls_min_version (1.3).

curl -s -X PUT -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d "{\"display_name\":\"FHIR audit aggregator\",\"config\":{\"endpoint_url\":\"https://audit.hospital.example/fhir/r4/AuditEvent\",\"auth_kind\":\"bearer\",\"bearer_token\":\"<token>\",\"ca_bundle_pem\":$CA,\"observer_id\":\"tavrik-gateway\",\"observer_site\":\"<site>\"}}" \
  $A/v1/tenants/$T/connectors/fhir_audit_https

Enable or pause delivery without touching the config:

curl -s -X PUT -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"mode":"enabled"}'  $A/v1/tenants/$T/connectors/$K/policy
curl -s -X PUT -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"mode":"disabled"}' $A/v1/tenants/$T/connectors/$K/policy

Procedure — Client cert rotation (MLLPS or FHIR mTLS)

When

Pre-emptive: connector.client_cert_expiring with days_until_expiry ≤ 14 (attributes also carry not_after)
Reactive: cert already expired and connector.mllps_handshake_failed / connector.fhir_endpoint_unreachable with failure_class: cert_expired is firing

Steps

Generate new client cert + key per your hospital’s CA process (CSR + signature, not self-signed for production).
Check NotAfter: openssl x509 -in new-cert.pem -noout -dates.
Resend the full config (section above) with the new client_cert_pem / client_key_pem. The validator rejects a cert without a key or vice versa (connector_config_invalid_mtls_pair). The API envelope-encrypts the new material per ADR-0035.
Verify: within a minute, connector.event_forwarded events resume for the connector (pre-flight step 2). No further connector.client_cert_expiring events should appear; the cooldown is per writer instance per 24 h, and the rebuilt writer reads the new cert’s NotAfter.

This procedure has not yet been exercised through a live rotation cycle; record the first one.

Procedure — CA bundle rotation

When

The hospital integration engine’s or FHIR aggregator’s TLS server cert chain rotates
connector.mllps_handshake_failed / connector.fhir_endpoint_unreachable with failure_class: unknown_authority is firing

Steps

Obtain the new CA bundle and check it covers the server cert: openssl verify -CAfile new-ca-bundle.pem server-cert.pem.
Resend the full config with the new ca_bundle_pem.
Verify: connector.event_forwarded resumes; parked rows in connector_retry drain on their next next_attempt_at.

Notes

The connector validates the server cert against the operator-supplied bundle only (no system-root fallback, ADR-0043). A server cert chained to a public CA still needs that CA in the bundle.
No certificate pinning (ADR-0040 Decision 3); TLS-inspecting proxies keep working.

Procedure — Integration engine outage

When

Integration engine or FHIR endpoint offline / unreachable
connector.mllps_handshake_failed with failure_class: connection_refused or timeout; connector.fhir_endpoint_unreachable with dns_failure / connection_refused / timeout / server_error
connector_retry growing; rows appearing in connector_dlq

Steps

Confirm the outage is on the far side, from the Tavrik host:

nc -zv mirth.hospital.example 6661          # TCP-level reachability
curl -sv --cacert hospital-ca.pem https://audit.hospital.example/fhir/r4/metadata -o /dev/null

Read the delivery state from the status endpoint (every role):
Terminal window
```
curl -s -H "Authorization: Bearer $KEY" "$A/v1/tenants/$T/connectors/$K/status" | python3 -m json.tool
```
retry.depth is the parked count and retry.oldest_next_attempt_at the next attempt; dlq.depth the dead-lettered count; cursor.lag_bytes how far behind the audit file’s tail the pair is; last_failure.class the categorical cause (auth_failed, connection_refused, dns_failure, timeout, tls_failed, server_error, send_failed). The same numbers are on the console connectors page and the dashboard’s Connector DLQ tile. Each failed event parks in connector_retry; the cursor keeps advancing past failures (no head-of-line blocking). After 10 attempts (about fourteen minutes of sustained failure with the 1 s → 5 min backoff) a row moves to connector_dlq with final_error, and the journal logs dispatch: event moved to DLQ.
Expect one diagnostic event per failed attempt in the audit log until B10b lands. A long outage writes many fhir_endpoint_unreachable / mllps_handshake_failed events; the dispatcher does not forward its own diagnostics to the failing sink (#299), so this does not loop, but it is noisy. If the outage will be long, disabled the policy and re-enable afterwards.
When the far side is restored, nothing needs to be done for parked rows: the retry loop drains connector_retry in next_attempt_at order. Watch connector.event_forwarded resume and the parked count fall to zero.
Replay the dead-letter queue once the far side is healthy (operator or root):
Terminal window
```
curl -s -X POST -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"all": true}' \
  "$A/v1/tenants/$T/connectors/$K/dlq/replay"          # or {"ids": ["<event_id>", …]}
```
Rows move back to connector_retry with attempts = 0 and are delivered on the next tick; the receiver dedupes on event_id (MSH-10 / FHIR resource id), so a re-delivery is safe. One connector.event_replayed audit event per row records who replayed what. If the events must not be re-sent (the far side already has them from an export), POST …/dlq/discard with the same body deletes the rows — audited as connector.event_discarded; the events themselves stay in the audit log. Never delete connector_dlq rows by hand. The console offers the same two actions on the connectors page whenever the DLQ is non-empty.

Notes

Every event has a stable UUIDv7 event_id (MSH-10 in HL7, resource id in FHIR), so re-delivery is dedupe-eligible at the far side. The HL7 mapping is untested against a live engine (B12).
The retry table is unbounded; for an outage longer than a day, watch the table size and coordinate with the Tavrik product team.

Procedure — First enablement and backfill

When

Enabling a connector for a tenant for the first time (Day 3 of the onboarding)
A customer asks for history to be delivered to a newly attached system

Steps

First enablement starts at the tail by default (ADR-0046 amendment Decision C): PUT …/connectors/{kind}/policy with {"mode": "enabled"} places the pair’s cursor at the current end of the audit file, so the first delivery is the next event, not months of history (#302). Pass "start_from": "beginning" only when the customer explicitly wants the full backfill on first enable. Re-enabling a pair never moves its cursor. The response and the connector.policy_updated event carry start_from and cursor_created.
Backfill an already-enabled pair with the explicit, audited rewind — never by editing connector_cursors:
Terminal window
```
curl -s -X POST -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"to_byte_offset": 0}' \
  "$A/v1/tenants/$T/connectors/$K/cursor/rewind"
```
The dispatcher re-reads the file from that offset on its next tick and re-delivers everything for the tenant; the receiver dedupes on event_id but still has to ingest the volume, so agree the window with the customer first. Audited as connector.cursor_rewound (from_byte_offset, to_byte_offset). GET …/status shows cursor.lag_bytes shrinking as the backfill drains.

Procedure — FHIR endpoint URL drift

When

The FHIR audit-aggregator endpoint moved
connector.fhir_endpoint_unreachable with failure_class: dns_failure or client_error (e.g. 404 from the old path)

Steps

Confirm the new URL with the aggregator’s operator.
Resend the full config with the new endpoint_url (https only).
Verify connector.event_forwarded resumes.

Procedure — FHIR bearer token rotation

When

The aggregator’s token is rotated on a schedule (the connector holds a static bearer token; SMART-on-FHIR client-credentials refresh is not implemented)
connector.fhir_endpoint_unreachable with failure_class: auth_rejected

Steps

Obtain the new token from the aggregator’s identity provider.
Resend the full config with the new bearer_token. The API envelope-encrypts it; plaintext exists only in process memory when the transport is built. The token is never logged.
Verify connector.event_forwarded resumes.

Procedure — Quarterly audit-log verification

Follow audit-log-tamper-investigation.md steps 1–4. There is no gateway-admin audit seal verify command and no daily seal; the shipped check is the per-event hash chain via POST /v1/audit/verify, which must be given an explicit, ordered list of event ids.

Verification — connector health is good

GET …/connectors/{kind}/status per enabled pair: retry.depth 0 or falling, dlq.depth 0, cursor.lag_bytes small and not growing, last_success_at recent, last_failure older than the last incident. The dashboard Connector DLQ tile reads 0.
connector.event_forwarded events flowing at the expected rate per tenant (console /audit, filter by event type; or pre-flight step 2)
connector_retry empty or draining; connector_dlq count unchanged since the last review (pre-flight step 3)
No connector.client_cert_expiring within the past 24 h
No connector.mllps_handshake_failed / connector.fhir_endpoint_unreachable within the past 24 h
Quarterly hash-chain verification clean per the tamper runbook

References

ADR-0043 — EHR-adjacent connector architecture
ADR-0046 — Connector dispatcher (audit file as outbox; connector_cursors / connector_retry / connector_dlq)
ADR-0036 — Connector framework (per-tenant configs and policies)
ADR-0035 — BYOK key resolution (envelope encryption of connector credentials)
ADR-0026 — Audit-read endpoints and the per-event hash chain
docs/operations/admin-api.md — endpoint reference
docs/operations/on-prem-vm-runbook.md — host layout, ports, environment
../healthcare/architecture.md — wire shapes and topologies
../healthcare/pilot-blockers.md — B5, B10a, B10b, B12
audit-log-tamper-investigation.md — escalation runbook for a hash-chain failure