Transport failure harness

Constructs network failures that a socket close cannot imitate, and drives apiary against them.

Everything here talks to a throwaway local enclave-server. Nothing in this directory should ever be pointed at a deployment anyone is using: the whole point is to break the link on purpose.

Why it exists

Apiary's unit tests cover the keepalive state machine, and smoke.sh covers a live connect. Neither can produce the failure that matters most in practice: a blackholed route, where bytes simply stop arriving and no FIN or RST is ever sent. The peer's kernel keeps the socket ESTABLISHED forever and every local write still succeeds. That is what a laptop changing networks looks like from the other end, and it is the condition behind both of the defects this harness measured.

Files

Apiary has no CLI — the binary speaks MCP JSON-RPC on stdin/stdout — so mcpdrv.py is how a test script calls a tool and observes what the bot received, without needing an MCP client.

Running

ENCLAVE_BIN=/path/to/enclave ./test/harness/env-up.sh
APIARY_BIN=$PWD/build/release/bin/apiary \
    python3 test/harness/exp_silent_loss.py
./test/harness/env-down.sh

SIGIL_LOG_LEVEL=debug makes apiary's per-post confirmation lines visible, which several experiments parse.

Always run an experiment as a matched PAIR

Point APIARY_BIN at the pre-change binary, then at the post-change one, and compare. A single reading tells you what a binary does; it cannot tell you what a change did. APIARY_BIN is deliberately required with no default, because a pair in which both arms silently ran the same binary is worse than no measurement — it looks like a result.

Watch the internal control in each experiment (the arm that should NOT change) as closely as the headline number. If it moves between arms, the two runs are not comparable and the delta is not attributable to the code.

THE TEARDOWN HAZARD — read this before stopping anything

Stop servers by the exact PID recorded in state.env. Never by pattern.

env-down.sh does this. Do not replace it with pkill -f enclave, pkill -f apiary, or anything similar, for two independent reasons:

1. Concurrent harnesses on one host are normal, not an accident. More than one of these environments has been found running at once, from the same binary path, belonging to different work. A pattern kill takes the other one down silently.
2. A pattern can match the killing process itself. An agent session's own command line typically contains its task name, its worktree path and its launch arguments. pkill -f <task-keyword> has terminated the very session that ran it, mid-measurement, with no output explaining why.

If you need to stop something whose PID you did not record, kill by port (fuser -k <port>/tcp) rather than by pattern.

Gotchas that will otherwise cost you an hour

The bot quota is 3 per owner and the leader bot counts. So one leader plus two workers is the ceiling, and exp_chatty_vs_silent sits exactly at it. Revoking a bot DISABLES the row rather than deleting it, so nicks stay burned and quota is only released when the row expires. Bring the environment up fresh between experiments rather than debugging quota-exceeded.

A single-token message body is dropped in transit. apiary writes PRIVMSG #chan :SPACELESS, and the server relays it as PRIVMSG #chan SPACELESS — no colon — so the receiving parser reads the body as a middle parameter and the text arrives EMPTY. Measured on the wire on both legs; two words relays intact in the same run. Every payload in this directory is multi-word for that reason. A harness using single-token tags would see its messages "lost" for a reason that has nothing to do with what it is testing.

Extra CAPs are lost on the reconnect path. After a reconnect, acked=sasl only, with a CAP REQ timed out — falling back warning. So anything you build on echo-message, draft/multiline or message-tags is unavailable on exactly the path that follows an outage. Verify against a reconnected connection, not just a fresh one.

sever_existing() and freeze_existing() are different failures. Severing closes cleanly and the peer sees EOF within milliseconds; freezing sends nothing at all and detection takes the keepalive's full window. Use severing only when you want the reconnect path itself, not the detection that precedes it.

The proxy still forwards the server's FIN in frozen mode. So a bot can be rescued by the server giving up on it, which a real blackhole would never deliver — that close travels the same dead path. Never cite a reconnected line as evidence that a bot detected anything itself; only its own PONG timeout is that evidence. The experiments here report the two separately for this reason.

The SYN-drop positive control — do not remove it

blackhole_new() stops new connections by filling the accept queue and never accepting, so the kernel drops further SYNs and a connect hangs for its full retransmit budget (~135s).

If the queue does not actually fill, the listening socket refuses instead, and a connect fails in about 5 seconds. Refused and dropped are different failures, not a strong and a weak version of one. An experiment that believes it is measuring the first while measuring the second still produces numbers, and the numbers still look like results.

So the mode carries two controls, and they answer different questions:

• assert_accepting() — the negative control. Proves the probe can see a HEALTHY port. Without it, a probe that reported "hung" unconditionally would pass the positive control for free.
• assert_blackholing() — the positive control. Proves new connections HANG, and raises loudly and specifically if they are merely refused.

python3 test/harness/proxy.py runs both and prints the classification either side of the switch:

  before blackhole_new(): fresh connect -> 'connected'
  [PASS] negative control: probe sees a HEALTHY port
  after  blackhole_new(): fresh connect -> 'hung'
  [PASS] positive control: new connections are DROPPED, not refused

python3 test/harness/proxy_control_test.py then breaks the mode three ways and asserts the control goes red for each — listener closed (→ refused), blackhole_new() a no-op, and a backlog large enough that the kernel completes the handshake by itself. A control that has only ever been seen passing is indistinguishable from one that always passes.

Both controls score their own single probe. An earlier version probed separately and compared the answers, which is a race rather than a check — two connects can land on different port states, and one did.