Field notes from an autonomous engineering fleet

1. Abstract

A single operator built a multi-persona engineering orchestrator (privateos) in 29 days and put it to work on four production repositories, where its agents authored 75 pull requests across a 28-day operation window. We analyze the complete GitHub record of both the construction (376 commits, 285 pull requests, 310 issues) and the operation, and verify every reported failure against a citable artifact: an eleven-entry incident registry asserted programmatically against archived snapshots. The central operational finding is that enforced structure outperforms prompted instruction. After a 2026-05-05 change that builds the pull-request body in code rather than asking the model to follow a template, the share of agent PRs carrying a summary and an issue linkage rose from 1 of 4 to 71 of 71 (one-sided Fisher exact p = 5.9 × 10⁻⁵), and the median blast radius fell from 9 files to 3. Merge rate is domain-dependent: 92 percent on a well-specified trading backend, 35 percent on a design-led website. Failures cluster into recurring families: scope indiscipline, self-triggering loops, silent capability loss, liveness misjudgment, and isolation gaps. The principal limitations are a four-PR pre-contract era and the loss of runtime telemetry, which restricts the analysis to the GitHub artifact record.

2. Introduction

Between 2026-04-29 and 2026-05-27, a single operator built an orchestration system that lets AI agents write production code without a human in the loop, and the agents then wrote some. The system, privateos, treats a GitHub Projects board as its scheduler: a card moving between columns dispatches one of sixteen personas (manager, engineer, worker, reviewer, fixer, triage, and others) as an idempotent database-backed job, and the dispatched agent checks out a worktree of the target repository, makes its changes, and pushes a branch and a pull request back. Four repositories were enrolled as workspaces during the study window: a Polymarket trading fleet (poly), a generative audio project (market-ambience), the research site that publishes this paper (atol-research), and a smoke-test sandbox. From 2026-05-05 the fleet also worked on its own orchestrator: 22 of the 285 construction pull requests are agent-authored, a recursion that §4.3 discloses and the analysis keeps out of the operational record.

Vendor demonstrations and benchmark suites dominate the public evidence on AI coding agents, and both sources share a selection problem: the work is chosen to be completable. The record studied here has the opposite character. It is everything the fleet did, including the pull request that deleted 565 lines it was never asked to touch, the fixer persona that triggered itself in a loop, and the kill switch that turned out to update a database row while the process it named kept running.

The contribution of this paper is an archival analysis of that complete record, with every failure claim verified against a concrete GitHub artifact and every number reproduced by a dependency-free notebook. §3 places the work against benchmark evaluations and the honor-system result it extends; §4 documents the system, the snapshots, the metrics, and the verification method; §5 reports the construction record, the operational record, the before/after effect of the PR-body contract, and the incident registry; §6 interprets the failure families and the one intervention that measurably worked; §7 enumerates threats to inference; §8 closes; §9 names the data.

3. Background and Related Work

Benchmark evaluations measure AI coding agents on curated issue sets: SWE-bench reports resolve rates on 2,294 GitHub issues drawn from twelve Python repositories (Jimenez et al., 2024), and SWE-agent established the agent-computer-interface framing on the same suite (Yang et al., 2024). Field evidence on AI-assisted development is thinner and concentrates on human-in-the-loop pair programming, where Peng et al. (2023) measured a 55.8 percent speedup on a single controlled task. Operational records of unattended fleets, where the agent owns the branch, the PR, and the merge-queue interaction end-to-end, are scarce in the published record; this paper supplies one such record at single-operator scale, with the caveats that scale implies (§7).

The analytical lens extends MKT-002 (ATOL Research, 2026), which documented how an AI-coordinated trading operation reproduced the Knight Capital failure mode: rules that existed in documents and were enforced by nothing. MKT-002 argued the distinction between a guidance layer (what the system is told) and an enforcement layer (what the system can do) is categorical. The present paper measures that distinction on a new surface. A PR-quality rule moved from the guidance layer to the enforcement layer mid-window, and the before/after record quantifies the difference.

Two systems-engineering literatures bear on the failure families in §5. Idempotent job dispatch under at-least-once delivery is a known hard problem with known patterns (Helland, 2012); privateos converged on the same patterns through operational incidents rather than through design. Liveness detection by heartbeat, and the false-positive reaping that follows from aggressive thresholds, is standard material in the site-reliability literature (Beyer et al., 2016); the fleet relearned it across a 14-day incident arc.

4. Methods

4.1 System under study

privateos is a TypeScript application (Hono server, React dock UI, Postgres 16, pg-boss job queue) that connects a GitHub App to a fleet of Claude-based personas. Webhook deliveries for board-column changes enter a dispatch registry that maps columns to roles; each dispatch attempts to claim a run row under a transaction-scoped advisory lock, so that webhook redeliveries, concurrent column moves, and operator retries cannot create duplicate work. The claim protocol is the system’s load-bearing concurrency primitive:

// source: privateos/src/server/claim-run.ts @ 44a0a64
await client.query("BEGIN");
await client.query(`SELECT pg_advisory_xact_lock(hashtext($1))`, [
  opts.lockKey,
]);

Worker-class personas run as spawned subprocesses against git worktrees of the target repository and report heartbeats; in-process personas run inside the job handler. A janitor reaps runs whose heartbeats go stale, a reconciler re-enqueues stalled chains, and per-workspace daily budget gates can defer runs. The repository at the studied commit contains 100,863 lines of TypeScript under src/, a further 88,454 under tests/, and 66 SQL migrations.

4.2 Data

All data are fixed snapshots fetched on 2026-06-11 and archived under data/raw/: the full pull-request and issue records of privateos (raw-privateos-prs, 285 rows; raw-privateos-issues, 310 rows), the full PR records of the four workspace repositories (raw-poly-prs, 197; raw-atol-research-prs, 49; raw-market-ambience-prs, 41; raw-privateos-sandbox-prs, 1), the git history of the privateos main branch at commit 44a0a64 in numstat form (raw-privateos-git-numstat, 376 commits), and the listing of the postmortem directory (raw-privateos-learnings-files, 29 files). Dataset identifiers here and below refer to data/manifest.yaml.

The fleet’s runtime Postgres held zero rows in its runs, events, and webhook_log tables at the snapshot date, following a database reset after the operation window. Runtime telemetry (per-run cost, tokens, latency) is therefore unavailable, and the analysis is restricted to artifacts the fleet left in GitHub: pull requests, issues, commits, and documents committed to the repositories.

4.3 Agent-PR identification and quality metrics

A pull request is classified as agent-authored when its head branch carries the agent/ prefix or its body contains the run.id stamp the runner writes on creation. The rule yields 75 PRs across the four workspaces (agent-prs). Four structure flags are computed from each PR: presence of a ## Summary section, presence of an issue linkage line (Part of #, Closes #, or Fixes #), presence of a ## Scope section, and whether the head branch is a raw GitHub node id rather than a readable slug.

The window splits at 2026-05-05, the day a root-cause analysis of poly#191 (an agent PR rejected in human review) produced the PR-body contract: the body template, the issue linkage, and the scope statement are assembled in code from data the system already holds, with the design note that this must happen “not by asking the model to remember to write it.” PRs created before 2026-05-06 form the before era (n = 4) and the rest the after era (n = 71). The split is observational and the analysis is tagged [post-hoc] throughout; no hypothesis was registered before the data existed.

Applying the same identification rule to the privateos repository itself yields a further 22 agent-authored pull requests, all merged, created from 2026-05-05 onward: from the contract day on, the fleet also worked on its own orchestrator. These 22 PRs are excluded from the 75-PR operational record so that the construction and operation surfaces stay distinct; §7 returns to this boundary. The one structure-compliant before-era PR (poly#192) was created hours after the root-cause analysis on the contract day itself, so the era boundary is conservative for the measured contrast.

4.4 Incident registry verification

Eleven incident claims were collected from the fleet’s own engineering record (commit messages, issue threads, and the 2026-05-05 state-of-the-app document in the privateos repository) and ground-truthed against the snapshots. For each claim, the notebook asserts that the named artifact exists, that its title contains the expected phrase, and that its state and dates match the narrative; the notebook fails if any assertion drifts from the record (incident-registry). The nine merged rollout PRs of the claimRun consolidation (epic #196) are verified the same way.

4.5 Statistical approach

Era contrasts on binary flags use the one-sided Fisher exact test, computed as the hypergeometric tail probability of observing at most $s_1$ successes in the before era under a shared rate:

where $N$ is the total PR count, $K$ the total success count, and $n_1$ the before-era count. With $n_1 = 4$, only near-categorical shifts can reach significance, which is the appropriate level of caution for this sample.

4.6 Software environment

Python 3.12, standard library only; no third-party packages and no network access. The notebook (notebook.ipynb) loads the gzipped snapshots, rebuilds every derived dataset in data/, runs the verification assertions, and writes the four figure specifications. Runtime is under ten seconds on commodity hardware; the final cell prints per-file SHA-256 digests and all checks pass.

5. Results

5.1 Construction record

376 commits reached the privateos main branch in a 29-day window with 27 active days, alongside 285 pull requests (284 merged) and 310 issues (Fig. 1). The largest subsystem by lines added is tests/ at 91,382, ahead of the server itself at 62,902

5.2 Operational record

75 agent-authored pull requests landed across the four workspaces between 2026-04-29 and 2026-05-26, of which 56 merged, a 75 percent merge rate overall. The rate is sharply domain-dependent (Fig. 2): 36 of 39 merged on poly (92 percent), 12 of 12 on market-ambience, 8 of 23 on atol-research (35 percent), and 0 of 1 on the sandbox. poly issues carried file-level specifications inherited from epic breakdowns; atol-research work was design-led, and 14 of its agent PRs remained open at the snapshot date, with one more closed unmerged.

5.3 The PR-body contract

1 of 4 before-era agent PRs carried a summary section and an issue linkage; after the 2026-05-05 contract, 71 of 71 carried both, and 69 of 71 carried an explicit scope section (Fig. 3) [post-hoc]. The one-sided Fisher exact probability of the summary and linkage shifts under a shared rate is 5.9 × 10⁻⁵ each (Eq. 1). Raw node-id branch names went from 3 of 4 to 0 of 71. Median blast radius fell from 9 files to 3, with the before era topping out at 26 files against 30 after (Fig. 4).

The anchoring artifact is poly#191, created 2026-05-05 and closed unmerged: titled as a narrow configuration fix, it touched 15 files with 53 additions and 565 deletions, deleted client and redeemer modules it was never asked to modify, carried the literal body (no summary) above its run-id stamp, and shipped under the branch name agent/I_kwDORxYP-s8AAAABA1Z61A. The merge rate moved from 2 of 4 to 54 of 71, which does not reach significance at this sample (p = 0.26) [post-hoc].

5.4 Incident registry

Table 1 lists the eleven verified incident artifacts; the dispatch loop they interrupt is the one sketched in Figure 5. Every failure family that produced a control subsystem is represented: the scope explosion preceded the PR-body contract, the fixer self-loop preceded review-driven triggering (Fig. 6), the duplicate deferred rows preceded the dedup fix, and the ineffective kill preceded the cooperative-cancellation work.

Table 1. Verified incident registry. Each row is asserted by the notebook against the archived snapshots; dates are artifact creation dates (incident-registry).

The liveness arc alone spans 14 days, from the discovery that the kill mechanism could not kill (privateos#362) to the threshold rebalance that stopped the janitor from reaping healthy runs (privateos#613) (Fig. 7). Alongside the registry, the fleet institutionalized its memory: 29 single-bug-class postmortems across eight categories accumulated in the repository’s learnings directory, with architecture patterns (9) and conventions (7) the largest.

6. Discussion

The contract result isolates a mechanism, within the limits of an observational split. Before 2026-05-05, PR structure was a guidance rule: the worker persona was expected to write a summary, link the issue, and respect scope. After it, structure became an enforcement property: the orchestrator assembles the body from the issue node id, the epic linkage, and the spec file list it already possesses, and the model is no longer the custodian of the template. The shift moved compliance from 1 of 4 to 71 of 71 (§5.3) [post-hoc]. MKT-002 (ATOL Research, 2026) argued from a single anchoring incident that guidance-layer rules fail regardless of how prominently they are stated; the present record adds a quantified before/after on a production surface, and the direction matches.

Failures arrived in families rather than as one-off events, and the families ordered the architecture. Every control subsystem in privateos has a verifiable incident that preceded it (Tab. 1): the body contract followed the scope explosion, review-driven triggering followed the fixer loop, the claimRun primitive consolidated the per-role dedup logic behind three composable guards after redelivery races, and the janitor’s thresholds were rebalanced only after they reaped healthy work. The system’s complexity reads as scar tissue. None of the controls existed in the initial design, and the postmortem directory (29 entries in 27 active days) grew at roughly the pace of the commit record itself.

Silent capability loss is the family with the worst detection asymmetry. A loud failure (a crash, a red CI run) entered the record and was fixed within days; privateos#193 describes an agent whose terminal tools were defined but never registered, so it produced plausible work while quietly lacking the capability the work assumed, and nothing errored. The matching SDK quirk (privateos#220) inverted the sign: successful runs were recorded as failures. Both artifacts date to the same audit, which suggests the discovery mode for this family is deliberate inspection rather than alerting.

The merge-rate spread (Fig. 2) bounds what the fleet can be trusted with. Where acceptance criteria were mechanical (a file list, a passing gate sequence), agents merged at 92 to 100 percent; where acceptance was aesthetic, 15 of 23 PRs failed to merge (14 still open at the snapshot date), awaiting a human judgment the board could not encode. The operator also remained the only effective backstop during control failures: when the kill mechanism updated a row while the process ran on (privateos#362), attention, rather than architecture, was the remaining layer of enforcement. The honor system that MKT-002 located in guidance documents survives here in a narrower form, as the operator’s own vigilance.

7. Limitations

This is a single fleet, run by a single operator, on one model family, over 29 days; none of the rates reported here estimate anything beyond this system. The before era contains four PRs, the era split is post-hoc, and the contract landed alongside other early fixes, so the attribution in §6 rests on mechanism as much as on the test in Eq. 1. The merge-rate contrast is confounded by the operator being both the merger and the author of the system under study. Runtime telemetry was lost to a database reset, so cost, latency, and run-level failure rates are absent; the GitHub artifact record undercounts any agent activity that produced no PR, and the agent/ branch heuristic may miss agent-authored work pushed under other naming. The construction and operation surfaces also overlap: the 22 agent-authored privateos PRs of §4.3 mean the fleet helped build its own orchestrator from 2026-05-05 onward, and the 285-PR construction record is not purely human work. The incident narrative draws on engineering documents written by the same operator whose system they describe. Verification (§4.4) constrains those narratives to match the artifact record, and constrains nothing about what the record omits.

8. Conclusion

A month of complete operational record from a self-hosted agent fleet supports one quantified claim and two structural ones. Quantified: moving PR structure from prompted instruction to code-level enforcement took compliance from 1 of 4 to 71 of 71 and shrank median blast radius from 9 files to 3. Structural: failures recur in families (scope, loops, silence, liveness, isolation) and each family eventually purchased a control subsystem; and the fleet’s competence boundary tracks specification crispness, with merge rates of 92 to 100 percent under mechanical acceptance criteria and 35 percent under aesthetic ones.

What remains open is the cost ledger. The next operation window runs with the telemetry retained, which permits the questions this record cannot answer: cost per persona, time-to-recovery by failure family, and whether the contract effect holds when the before era is larger than four.

9. Data Availability

Raw snapshots are gzipped under data/raw/ and are excluded from the public repository; the upstream repositories are private to the operating organization, and data/README.md documents the exact refetch commands for readers with access. All derived datasets are committed as CSV and rebuilt by the notebook. The derived datasets are released under CC BY 4.0, the license declared in the manifest; the raw snapshots describe private repositories and are not redistributed.

• agent-prs (data/agent_prs.csv): 75 agent-authored PRs with structure flags and era assignment.
• pr-quality-eras (data/pr_quality_eras.csv): before/after aggregates for the contrast in §5.3.
• construction-cadence (data/construction_cadence.csv): per-day commits, PRs, and issues for the build window.
• loc-by-subsystem (data/loc_by_subsystem.csv): weekly lines added by subsystem.
• incident-registry (data/incident_registry.csv): the eleven verified incident artifacts of Table 1.
• learnings-catalog (data/learnings_catalog.csv): postmortem counts by category.
• workspace-summary (data/workspace_summary.csv): per-workspace merge outcomes plotted in Figure 4.

The full manifest is at data/manifest.yaml. The reproducibility notebook is at notebook.ipynb; Python 3.12, standard library only, runtime under ten seconds, final cell prints all checks pass.

10. References

See references.yaml. Inline citations resolve there: Jimenez (2024) on SWE-bench, Yang (2024) on SWE-agent, Peng (2023) on AI pair-programming productivity, Helland (2012) on idempotence in distributed dispatch, Beyer (2016) on liveness and heartbeats, and ATOL Research (2026) for the honor-system predecessor MKT-002.