Seven things, one person

Over the last few months I built a lot of things at work. An agent platform that turns work items into code. An SRE dashboard that grew into something much bigger than a dashboard. A meeting notes app. A sales CRM. An AI gateway that every one of those tools now talks through. A bridge between our support desk and our issue tracker. And, on the side, a few small fine-tuned models.

That is a lot of surface area for one person, and I do not say that to brag. I say it because building that many things back to back taught me something different from building one thing well. Patterns show up. The same mistake shows up in three repos. The same good decision saves you in four. This post is me writing those down before I forget them.

The agent platform: the agent never touches the outside world

The agent platform takes a work item, plans it, implements it in a sandbox, checks its own work, and hands back a draft change for review. The most important line in its design doc is not about the model at all. It is this: the agent never writes to the repo or the issue tracker. Pushing code, opening the review, posting a comment, all of that is done by plain server code after a human clicks a button.

The agent works inside a sandbox with no write access outside it The agent plans, implements and self-checks inside a sandbox. It can only hand a draft across the boundary. A human approves, and plain server code does the push to the repo and issue tracker. sandbox: the agent plan implement self-check no write access draft human approves server code repo + tracker
The agent can be ambitious inside the box because nothing it does can leave the box on its own.

That one rule made everything else easier. I could let the agent be ambitious inside the sandbox because nothing it did inside the sandbox could leak out on its own. The blast radius was decided by the architecture, not by how careful the prompt was.

The other lesson was about failure. When we retired one model provider, I had the choice to quietly fall back to another one or to refuse loudly. I chose the loud refusal. A quiet fallback would spend money on a provider nobody picked and would probably fail anyway, just later and more confusingly. I keep coming back to that. A loud failure is a gift. A quiet fallback is a debt.

And containers lie. I had one that reported itself as up, with nothing listening and no logs, because a single line of Python used syntax newer than the image’s interpreter and the start command hung forever. “Up” is not the same as “working”. Now I check the port, not the status.

The SRE dashboard: tests that agree with the code prove nothing

This one started small. Compare how many queue consumers are live against how many there should be, and send a message when they do not match. It grew into incident intelligence, an alert noise classifier, an agent, and a handful of other lanes.

The thing I learned here changed how I work with AI coding tools in general. When an agent writes the code and then writes the test, the test and the code are the same interpretation of intent. If the interpretation is wrong, both are wrong together, and the test passes. I had an outage slip through exactly like that: the code called a method on a value that, in production, was a different type than the mocked one the agent had invented. The mocked test was green. Production was not.

One reading of intent versus two On the left, the agent reads the intent once and writes both the code and the test from that reading, so the test passes even when the reading is wrong. On the right, the human owns the contract test and the agent writes the code, so a wrong reading shows up as a failure. one mind agent's reading code test green, even when wrong two minds my reading agent's contract code disagreement surfaces
A test written from the same reading as the code is a second copy of the same opinion.

So I split ownership. There is now a contract test file that I own, and a hook that stops agents from editing it. When a contract fails, the code is wrong. Not the contract. That single file catches more than every agent-written test combined.

The second lesson was about being polite to expensive things. The AI alert classifier used to re-ask the model every time an input moved slightly. Now inputs are bucketed so small wobbles do not count as change, and a failed call backs that alert off for half an hour instead of retrying in a loop. Most of the cost of an AI feature is the feature asking the same question again.

The meeting notes app: the boring parameters are where the money is

The notes app records a meeting, transcribes it, and merges your scribbles with the transcript into proper minutes. It is the most “AI product” thing I built, and almost everything I learned from it was about knobs nobody talks about.

Setting reasoning effort to low on a summarisation call cut token use by roughly five times with no visible drop in quality, because a summary does not need the model to think hard. One step lower than that returned an error. An hour of audio is around a hundred thousand tokens, which is fine until you multiply it by a whole company’s calendar.

Prompt caching taught me the most. Identical requests hit the cache almost every time. Requests with the same beginning and a different ending hit it far less than I expected. And anything dynamic in the system prompt, a timestamp, a request id, quietly breaks caching for every call.

Cache hits by request shape In a small test, identical requests hit the cache 7 times out of 8. Requests with the same prefix and a different suffix hit only 2 out of 8. A timestamp in the system prompt means every request is new, so it never hits. cache hits, out of 8 calls identical request 7 same prefix, new suffix 2 timestamp in system every call is a new prompt
Small test, not a benchmark. The shape of the request decides the bill more than the model does.

I now treat the system prompt like a build artifact: frozen, and nothing in it that changes per request.

The privacy decision was the easy one. Audio is not stored. The login token is thrown away right after we read the profile. If you do not keep it, you cannot leak it.

The CRM: decide the dull things first

The CRM is the least glamorous thing on this list and the one I learned the oldest lessons from. Money is stored as an integer number of the smallest unit, never as a float. Permissions are checked on the server, not only hidden in the UI, because a hidden button is not a permission.

None of this is new. I knew all of it before. But it was a good reminder that the decisions that hurt most later are the dull ones you make in the first hour, and they are the ones you are most tempted to skip because they do not feel like progress.

The AI gateway: one place to turn things off

Before the gateway, every tool had its own keys to its own providers. Nobody could answer simple questions. Who is spending what? Which key do I revoke if a laptop goes missing? What happens when a provider has an outage?

The gateway is one endpoint in front of every provider. Every person and every app gets its own key with a list of allowed models and a budget. The agent platform, the SRE agent and the notes app all talk through it. Swapping a provider is now a config change, not a code change in seven repos.

Every tool talks to every provider through one gateway Four internal tools on the left all connect to a single gateway in the middle, which holds per-key budgets, model allow-lists and one place to revoke access. The gateway connects to four model providers on the right. agent platform SRE agent notes app support bridge provider A provider B provider C self-hosted gateway per-key budgets model allow-list usage per person one revoke switch
Before: a key per tool per provider. After: one door, with a lock you actually control.

What I learned: build the control point before you need it. The gateway felt like overhead while I was building it. Every week since, it has paid that back, usually in the form of a question I could answer in thirty seconds instead of a day. I also learned that “open source with a paid tier” often means the feature you need is in the paid tier, so I ended up building a small dashboard of my own for the part that was capped.

The support bridge: let the existing system do its job

The support bridge pulls tickets from our support desk, triages them with a model, and creates work items, with a human reviewing everything before anything goes outbound.

The lesson I keep thinking about came from SLA tracking. I wrote my own priority tier calculation to work out when a ticket was due. It was clever. Then I reverted it, because the support desk already had a due date field that implemented the real policy, including the cases where the clock pauses. My version was a second source of truth that would drift from the first one the moment someone changed the policy. If the system you integrate with already knows the answer, ask it. Do not re-derive it.

Smaller one: API docs lie by omission. One search filter that looked obviously correct returned a 400 error, and the field that actually worked had a different name. The only documentation that is never wrong is a request you have actually sent.

The small models: the loss curve is not the truth

The fine-tuning work was about one question: can a small model, running on a normal laptop, answer questions about our internal documentation well enough to be useful?

Short answer: a 4B model was not reliable enough. Around 30B started to be. That was the honest result and it was worth getting, because it set the real hardware bar instead of the hoped-for one.

The longer lessons were about trusting numbers. On one run, training loss fell beautifully from almost 2 to under 0.2 while evaluation loss stayed flat and then got worse. The model was memorising, not learning, and it started after about one and a half epochs.

Training loss falls while evaluation loss does not Over three epochs, training loss drops steeply from about 1.9 to about 0.2. Evaluation loss stays roughly flat, then starts rising after about one and a half epochs, which is where the model begins memorising instead of learning. loss 0123 epochs ~1.5 epochs: memorising starts train loss eval loss
Shape of the run, redrawn by hand. The green line looked like success. The red line was the truth.

On another comparison, two models had very different loss numbers that turned out not to be comparable at all, because one run measured loss on the whole sequence and the other only on the assistant’s answers. Same word, different measurement.

And my favourite one: the training tool I used had a local backend option that, it turned out, did nothing. Setting it silently fell through to a different backend. Everything ran. Nothing errored. It just was not doing what the config said. I patched it, and then an upgrade wiped my patch. Silent fallbacks again.

The threads that run through all of it

Looking back across all seven, a few things repeat often enough that I believe them now.

Silent is the enemy. The container that said it was up. The backend flag that did nothing. The mocked test that passed. The quiet provider fallback I chose not to build. Nearly every bad day came from something that failed without saying so. I would rather have a tool that crashes loudly than one that keeps going wrong politely.

Put the boundary in the architecture, not the prompt. The agent cannot push code because the agent does not have the ability, not because I asked it nicely. The gateway can revoke a key because it is the only place keys live. Rules that depend on good behaviour are suggestions.

Own the thing that checks the work. If the same mind writes the code and the test, you have one opinion, not two. Keep one layer of verification that is yours.

Most AI cost is repetition. Re-asking the same question, re-sending the same prompt with a timestamp in it, re-thinking a summary that did not need thinking. The model is rarely the expensive part. The loop around it is.

Do not rebuild what already knows the answer. The due date field. The provider’s own policy. The existing system of record. My cleverness is a liability when it creates a second version of the truth.

None of these are new ideas. I had read most of them before. The difference is that now each one comes with a specific bad afternoon attached, and that is the version of a lesson that actually sticks.