How It's Built — Ben Merrill

The shape of it

Two systems, one house.

Sparky

My personal AI. One Python process (FastAPI), reachable from my phone, my desktop, and a tablet display on my desk, over a private network. Nothing exposed to the internet.

The Lab

My AI coding harness. A real workspace where agents read files, run checks, and produce things I actually use, with the work visible as it happens.

They have different jobs and a lane between them: I fire a half-formed thought at Sparky, she helps me find its shape and writes the brief, the Lab builds it, and the result comes back to the conversation.

The numbers

Small on purpose.

From the current build — the one that works — as of October 2026:

Part	Size
The assistant	~5,000 lines of Python, 22 files
Web front end	~2,300 lines — plain HTML/CSS/JS, no framework
iPhone app	~850 lines of Swift
House commands	~1,400 lines, 10 small scripts
Tests	~4,000 lines — 374 tests, run offline with a scripted model

Brain: a local open-weight model served on the same box. No cloud model in the loop.
Storage: one SQLite file. No vector database, no embeddings — search is full-text.
Twelve tools, and a budget that keeps it near twelve. New abilities are small shell scripts the assistant runs through its own shell, so the tool list never grows.
Identity is one prose file — the voice document loaded at startup. It's the most important file in the repo.
The current build started September 22, 2026: about 250 commits in its first three weeks, 270 in its first month.

The loop

How one message works.

The message is saved with a small note (time, thread, any new facts).
Context is assembled: the voice file, my profile card, saved facts, and a bounded window of recent conversation. Older history is never replayed or summarized — it's searched on demand.
The model answers or calls tools. Every tool call and result is saved as a row and shown as a receipt. A receipt that isn't stored is one a model can later invent.
Read-only actions run immediately. Anything that changes, moves, sends, or deletes becomes a card — Do it / Nope — and the agent waits.
The loop repeats until there's an answer. A message I send mid-work is read at the next step, so I can steer without stopping it.

Safety is one rule: looking is free, changing asks. An allowlist decides what "only looks"; everything else needs my tap. The agent is never allowed to change its own code, voice file, settings, or database — even with a card.

War stories

What it taught me the expensive way.

01

2,500 commits of mess.

The first iteration of Sparky grew to about 80,000 lines and 31 tools across roughly 2,500 commits. It worked, in the way a pile of cardboard works as furniture — until you sit on it. Every feature had added a mode, a setting, or a list to maintain.

I retired it. The rebuild — the Sparky you're reading about — is about 5,000 lines and works better than the mess ever did. The rule now for every feature: does it remove a layer from my life? If it adds one, it doesn't ship.

02

One agent doing everything.

Early on, I had a single agent do all the work — design, scope, build, review. It was fast, and it was building me a cathedral I'd never use.

The fix was an AI team, and it's the biggest lesson in this whole project:

An architect — a dedicated agent on the strongest frontier model available — scopes the backend and holds the shape of the system.
Independent builders get fired off, one per slice of work, each with its own brief.
The architect reviews what the builders produce.
Every few slices, a second, separate frontier model audits the accumulated work — a reviewer with no stake in any of it.

One human decides what gets built. The agents write, test, and check each other's work. Building an AI team to build for me — instead of asking one AI to build with me — has been the big win.

03

Fifty features I never used.

The first version had about 50 features, because I was excited and every idea sounded essential. I really only ended up needing about 10.

Now features only get built after daily use hits a real wall. One change at a time, then live with it for a while. The 10 features I actually needed are better than the 50 I imagined needing — and they're the only ones I have to maintain.

If you're asking the same question I was

What I'd tell an aspiring agentic AI engineer.

Build something you'll actually use every day. A demo that works once teaches you nothing. Software you live with for a month teaches you everything — including which of your 50 features were really 10.
Make proof a first-class feature. If your agent can't show a receipt for what it did, you don't have a system, you have a story.
Use agents as a team, not a genie. Separate the roles — architect, builder, reviewer — and never let the builder grade its own homework.
Stay small on purpose. Every feature is something you maintain forever. The rebuild beat the original by being a sixteenth the size.

Still open

What it can't do yet.

Honest list: read my calendar, send texts, see old photos again, or touch anything outside the house without a card. It's told to say so plainly rather than improvise — which is the whole philosophy in one sentence: no hidden magic, no fake confidence, and a receipt for everything.