v3Latest
How to Add Agent Skills Support to a Custom Agent Harness
Add Agent Skills support to your custom AI agent: SKILL.md discovery, progressive disclosure, tool loading, and evals, with TypeScript and Python examples.
• 18 min read
The case study
Cameron
Building a local-first AI finance agent, in public — one release at a time.
A local-first personal finance agent you own and run yourself — an AI agent you can trust with real money because it never acts without your approval. Built in public, one tagged release per article: each entry adds a capability to the same agent instead of starting over, so you learn agent engineering by watching one real system grow.
Most agent tutorials build a throwaway toy and move on. Cameron is the opposite: one real agent that grows across the series, so you see how tools, memory, and safety actually compound into a system you can trust. And it is local-first by design — your financial data never leaves infrastructure you control, which is exactly the constraint that makes the engineering interesting.
Design invariants
These hold every version and explain why the code is shaped the way it is.
- 01Cameron never moves money or mutates financial records without explicit human approval.
- 02Every capability — built-in, skill, connector, or self-written — passes through the same approval gate.
- 03All financial data stays on infrastructure the owner controls.
Cameron in action
Two real sessions against a live statement. Both show the same invariant from different angles: you see what Cameron is about to do before it does it.
cameron · can you load these expenses
Step 1 of 3
The roadmap
Each version is roughly one or two articles. The ecosystem may reorder or insert versions along the way — that’s the point.
- v0FoundationShipped
The starter as-is: chat loop, streaming, persistence, dynamic MCP tool loading, human-in-the-loop approvals, and multi-model support that everything else builds on.
- v1Cameron is bornShipped
Identity and persona, a transaction store you own, and approval-gated finance tools (log, query, guarded SQL, fail-loud CSV import). "Meet Cameron."
- v2EvalsShipped
A test layer for the tools and an eval harness with gradeable ground truth, so every capability after this one ships with evidence it works instead of an assurance that it does.
- v3SkillsShipped
A skill loader built on the standard SKILL.md format, with progressive disclosure: names and descriptions sit in the prompt, and the agent loads a skill’s full instructions only when it decides it needs them. The first skill turns questions into visual spending reports, rendered by UI tools the agent calls with data — no code sandbox required.
- v4Owning the tracesUpcoming
Cameron inherited Langfuse tracing from the starter: a vendor callback pointed at a hosted backend by default. That quietly breaks the third rule, because a trace carries every prompt, tool argument and transaction. Tracing moves to OpenTelemetry’s GenAI conventions, financial data is redacted before anything leaves the process, and the backend is self-hosted next to Cameron, so the traces are as owned as the data. Any OTLP backend becomes a one-line swap.
Show the rest of the roadmap (v5–v13)Hide the later roadmap
- v5MCP v2Upcoming
The MCP client Cameron inherited from the starter moves from the v1 SDK to v2, and stops binding every connected tool into the context up front. Tools get the same progressive disclosure skills got in v3: the agent sees each tool’s name and a one-line description, and loads its full schema only when it decides to call it. Context stays lean as connectors pile up, starting with DemoBank in v7.
- v6DocumentsUpcoming
Receipt and statement ingestion: extract a proposed expense from a photo, then confirm it through the same approval gate — the model reads, the owner decides.
- v7External dataUpcoming
DemoBank as a standalone MCP server — build, secure with OAuth, deploy; Cameron consumes it as a client.
- v8MemoryUpcoming
Budgets and preferences as long-term memory, plus summarization of aging history — once there is enough expense history to make them worth constraining.
- v9TrustUpcoming
A release regression suite built on the v2 eval harness and the v4 traces: production runs that went wrong become eval cases that keep them from going wrong again.
- v10AutonomyUpcoming
A scheduler, weekly digest, subscription-creep detection, and an approval queue for actions taken while unattended.
- v11ChannelsUpcoming
A messaging gateway for alerts and digests over Telegram / WhatsApp.
- v12DelegationUpcoming
Subagents — a tax-season agent, a big-purchase research agent — and context flow between them.
- v13Self-extensionUpcoming
Cameron detects capability gaps, drafts its own skills, tests them in a sandbox, and requests approval through the same gate. The payoff.
Entries
Devlog
When Your Eval Script Answers a Question Nobody Asked
A scripted multi-turn eval replays fixed user turns regardless of what the agent said. The moment the agent asks a question the case author did not anticipate, the script answers something else and the case fails for the wrong reason - rewarding agents that guess and punishing agents that ask.
• 9 min read
Devlog
How to Write Your First AI Agent Evals
Where the first cases come from, which ones earn a place in the suite, why every case that pushes a behavior needs one that bounds it, how to grade without an LLM judge, and what the first honest run surfaced.
• 18 min read
Devlog
Testing Tools for AI Agents: catching the bugs your agent can't tell you about
You designed the tool well. Nothing verifies it still does what its description promises. An audit of eight tools found four contract bugs, including one that made a finance agent report $2,812 for a $2,752 bill, and the cheap test layer that catches them.
• 13 min read
Devlog
Meet Cameron: a personal finance agent you own and run
Cameron is a local-first personal finance agent you own and run yourself. v1 shows how an LLM loop, tools, and an approval gate compose into an agent you can trust with real money.
• 3 min read