test/harness/README.md

1# Transport failure harness

3Constructs network failures that a socket close cannot imitate, and

4drives apiary against them.

6Everything here talks to a **throwaway local enclave-server**. Nothing

7in this directory should ever be pointed at a deployment anyone is

8using: the whole point is to break the link on purpose.

10## Why it exists

12Apiary's unit tests cover the keepalive state machine, and `smoke.sh`

13covers a live connect. Neither can produce the failure that matters

14most in practice: a **blackholed route**, where bytes simply stop

15arriving and no FIN or RST is ever sent. The peer's kernel keeps the

16socket `ESTABLISHED` forever and every local write still succeeds. That

17is what a laptop changing networks looks like from the other end, and

18it is the condition behind both of the defects this harness measured.

20## Files

22| file | what it does |

23|---|---|

24| `proxy.py` | freeze-proxy: blackholes live sockets, or drops new SYNs |

25| `proxy_control_test.py` | proves the SYN-drop control can go RED |

26| `mcpdrv.py` | JSON-RPC driver for apiary's MCP-stdio interface |

27| `env-up.sh` / `env-down.sh` | throwaway enclave-server lifecycle |

28| `exp_confirm_latency.py` | post-confirmation latency distribution |

29| `exp_silent_loss.py` | does a post during an outage claim delivery? |

30| `exp_chatty_vs_silent.py` | does a posting bot still detect a dead link? |

31| `exp_fd_flatness.py` | do the failure paths leak file descriptors? |

33Apiary has no CLI — the binary speaks MCP JSON-RPC on stdin/stdout — so

34`mcpdrv.py` is how a test script calls a tool and observes what the bot

35received, without needing an MCP client.

37## Running

39```sh

40ENCLAVE_BIN=/path/to/enclave ./test/harness/env-up.sh

41APIARY_BIN=$PWD/build/release/bin/apiary \

42 python3 test/harness/exp_silent_loss.py

43./test/harness/env-down.sh

44```

46`SIGIL_LOG_LEVEL=debug` makes apiary's per-post confirmation lines

47visible, which several experiments parse.

49### Always run an experiment as a matched PAIR

51Point `APIARY_BIN` at the pre-change binary, then at the post-change

52one, and compare. A single reading tells you what a binary does; it

53cannot tell you what a change did. `APIARY_BIN` is deliberately

54required with no default, because a pair in which both arms silently

55ran the same binary is worse than no measurement — it looks like a

56result.

58Watch the **internal control** in each experiment (the arm that should

59NOT change) as closely as the headline number. If it moves between

60arms, the two runs are not comparable and the delta is not

61attributable to the code.

63## THE TEARDOWN HAZARD — read this before stopping anything

65**Stop servers by the exact PID recorded in `state.env`. Never by

66pattern.**

68`env-down.sh` does this. Do not replace it with `pkill -f enclave`,

69`pkill -f apiary`, or anything similar, for two independent reasons:

711. **Concurrent harnesses on one host are normal, not an accident.**

72 More than one of these environments has been found running at once,

73 from the same binary path, belonging to different work. A pattern

74 kill takes the other one down silently.

752. **A pattern can match the killing process itself.** An agent

76 session's own command line typically contains its task name, its

77 worktree path and its launch arguments. `pkill -f <task-keyword>`

78 has terminated the very session that ran it, mid-measurement, with

79 no output explaining why.

81If you need to stop something whose PID you did not record, kill by

82port (`fuser -k <port>/tcp`) rather than by pattern.

84## Gotchas that will otherwise cost you an hour

86**The bot quota is 3 per owner and the leader bot counts.** So one

87leader plus **two** workers is the ceiling, and `exp_chatty_vs_silent`

88sits exactly at it. Revoking a bot DISABLES the row rather than

89deleting it, so nicks stay burned and quota is only released when the

90row expires. Bring the environment up fresh between experiments rather

91than debugging `quota-exceeded`.

93**A single-token message body is dropped in transit.** apiary writes

94`PRIVMSG #chan :SPACELESS`, and the server relays it as

95`PRIVMSG #chan SPACELESS` — no colon — so the receiving parser reads

96the body as a middle parameter and the text arrives EMPTY. Measured on

97the wire on both legs; `two words` relays intact in the same run. Every

98payload in this directory is multi-word for that reason. A harness

99using single-token tags would see its messages "lost" for a reason

100that has nothing to do with what it is testing.

101

102**Extra CAPs are lost on the reconnect path.** After a reconnect,

103`acked=sasl` only, with a `CAP REQ timed out — falling back` warning.

104So anything you build on `echo-message`, `draft/multiline` or

105`message-tags` is unavailable on exactly the path that follows an

106outage. Verify against a reconnected connection, not just a fresh one.

107

108**`sever_existing()` and `freeze_existing()` are different failures.**

109Severing closes cleanly and the peer sees EOF within milliseconds;

110freezing sends nothing at all and detection takes the keepalive's full

111window. Use severing only when you want the reconnect path itself,

112not the detection that precedes it.

113

114**The proxy still forwards the server's FIN in frozen mode.** So a bot

115can be rescued by the server giving up on it, which a real blackhole

116would never deliver — that close travels the same dead path. Never cite

117a `reconnected` line as evidence that a bot detected anything itself;

118only its own `PONG timeout` is that evidence. The experiments here

119report the two separately for this reason.

120

121## The SYN-drop positive control — do not remove it

122

123`blackhole_new()` stops new connections by filling the accept queue and

124never accepting, so the kernel drops further SYNs and a connect hangs

125for its full retransmit budget (~135s).

126

127If the queue does not actually fill, the listening socket **refuses**

128instead, and a connect fails in about 5 seconds. Refused and dropped

129are **different failures**, not a strong and a weak version of one. An

130experiment that believes it is measuring the first while measuring the

131second still produces numbers, and the numbers still look like results.

132

133So the mode carries two controls, and they answer different questions:

134

135- `assert_accepting()` — the **negative control**. Proves the probe can

136 see a HEALTHY port. Without it, a probe that reported "hung"

137 unconditionally would pass the positive control for free.

138- `assert_blackholing()` — the **positive control**. Proves new

139 connections HANG, and raises loudly and specifically if they are

140 merely refused.

141

142`python3 test/harness/proxy.py` runs both and prints the

143classification either side of the switch:

144

145```

146 before blackhole_new(): fresh connect -> 'connected'

147 [PASS] negative control: probe sees a HEALTHY port

148 after blackhole_new(): fresh connect -> 'hung'

149 [PASS] positive control: new connections are DROPPED, not refused

150```

151

152`python3 test/harness/proxy_control_test.py` then breaks the mode three

153ways and asserts the control goes red for each — listener closed

154(→ refused), `blackhole_new()` a no-op, and a backlog large enough that

155the kernel completes the handshake by itself. A control that has only

156ever been seen passing is indistinguishable from one that always

157passes.

158

159Both controls score their own single probe. An earlier version probed

160separately and compared the answers, which is a race rather than a

161check — two connects can land on different port states, and one did.