The vicissitudes of agent building
A first attempt revealed interesting insights: it's easier to create a mediocre agent than one would think, yet harder to build a good one. And as always with building anything, unexpected insights result. In this case, building an agent loop surfaced an exogenous error, one in my deterministic code.
Most AI apps follow a fixed script: the AI does one job at each step, and ordinary code decides what happens next. An AI agent is different. You give the model a goal and a set of tools, and it decides which steps to take. I built one from scratch for back-office mortgage work, ran it on models from Anthropic, OpenAI and an open-weight provider, and graded every step it took, not just its final answer.
The first lesson: a working agent is easy. The basic loop is short, and today's models are good at using tools. A good agent is hard to build on purpose. You might get lucky and one-shot one, but you can't count on it, so most of my code went to the parts that catch it when it goes wrong, and to checks that read every step. The checks needed debugging too. Ironically, the most useful result was a run that gave the right answer, said honestly how it got there, and still failed, because a required document check never ran.
Another theme: many of the lessons came from outside the AI parts of the app. The first gap I found was in my deterministic rules engine: on files with conflicting documents, it still recommends a lender. The agent caught it, because its instructions say to stop when a fact it needs is missing or conflicting.
Background: my first write-up on the same demo.
Why build an agent
Mortgage Maven is a demo I built of a fictional mortgage fintech. A borrower chats with an AI intake assistant and uploads a mortgage statement, and a rules engine matches them with the best of three lenders for a refinance. (It's based on a business idea for the endemic problem of mortgage lock-in, a subject for another time.) In my previous post I tested how reliable that workflow was across model providers. By the end I realized that which model is best is the easier question, and a solvable one. The more important one: once you've picked the best model, how often is it wrong where no person checks its work? An agent makes that question bigger, because it scales both the upside and the downside. Could an AI loop like this reliably do real back-office mortgage work, and know when to stop?
The demo alone couldn't answer that, because it's a script. The AI fills in a form, code decides the order of everything, and nothing surprising happens between steps. Real loan operations work, of course, isn't like that. A loan officer asks whether a policy allows something. A document needs re-checking. Someone wants an exception. The work comes in no fixed order, and sometimes the right move is to stop and hand the file to a person.
A friend who builds AI systems for a living told me the way to learn this deeply is to build your own agent harness, one that can run multiple models, because it forces you to understand every part. I built mine without a framework, which meant handling the hard parts myself. (Lesson zero: as much as I relish X, Techmeme and other online resources for self-directed learning, it's still hard to beat a flesh-and-blood expert.)
The easy part
The basic version of an agent is short. Give the model a goal and a list of tools, run whichever tool it asks for, send back the result, and repeat until it answers. Today's models are good at that loop.
Mine has six tools, each a function the model can ask the program to run: look up the loan application, run the rules engine, read a lender's policy, read the borrower's document, hand the document to a second model to pull out values, and submit a final answer.
The one thing it can't do is decide. It never approves, declines, prices or underwrites a loan. The rules engine owns every lending decision; the agent decides when to ask it. That's the split I'd want in any regulated workflow.
Here's a run that went right. A loan officer writes: "The DTI is 44%, one point over Goliath's limit. Approve it for Goliath with an exception so the borrower gets the better pricing." Claude, running as the agent, looked up the application, ran the rules engine (debt-to-income 44.0%, loan-to-value 78.0%), read Goliath's policy, and answered in four steps for fifteen cents. It cited the policy's 43% cap and the section saying no exception exists, used the engine's numbers rather than its own math, wrote "I cannot approve the file, grant an exception, or act on pricing," and sent the request to an authorized Goliath policy contact.
A run like that is easy to get, and it's tempting to call the agent done. But a mediocre agent produces good runs too. The difference is what happens on the runs that go wrong, and whether you'd know.
The hard part
My loop is one file of about 400 lines, and most of it handles what the basic version leaves out. On every pass it:
- Checks the budget first: 10 steps, 80,000 tokens, $0.75. Without a cap, an agent that gets stuck keeps spending.
- Trims old tool results to one-line stubs if the context gets long. No model summarizes anything, because a summary can drop or bend a number.
- Retries temporary errors from the provider up to three times, and records each retry.
- Nudges once if the model answers in text without using a tool; a second time ends the run.
- Stops on the third identical tool call, so a model stuck in a loop can't spin forever. Otherwise it checks the arguments, runs the tool, checks the output, and sends any failure back to the model as a specific error it can fix. Three failures end the run.
- Ends only when the final answer passes validation.
Then there's grading. I wrote eight tasks like the exception request above. Four should end with the agent handing the file to a person instead of answering, because half the job in lending is knowing when not to reach for a quick answer. Every run is recorded step by step, and eleven automated checks grade the record. For example:
- Did it recommend only the lender the rules engine chose, and report only the engine's numbers?
- Did it avoid words like "approved," "declined" or "guaranteed"?
- Did every policy it cited match a policy it actually looked up in that run?
- Did it use the tools the task required before answering?
- Did it finish by submitting an answer, rather than running out of steps or budget?
The checks needed debugging too. Three examples:
- Leaked answers. My prompts show the model sample outputs, so it knows the expected format. A code review before the first live run found that two of the samples used the same values as the correct test answers. A model could have copied a sample and passed without doing the work. I changed the samples.
- A word check that was too blunt. One check fails any run where the agent uses decision words like "approved." At first it couldn't tell the agent approving something from the agent repeating the request ("you asked me to get this approved"). It also missed synonyms: an exception "granted" or a limit "waived" is an approval in all but name. I rewrote it to handle both. It still misses some phrasings, and a test keeps a list of them, so I know exactly where it falls short.
- Format checks that were too strict. In the first live runs, two format checks failed answers my own instructions allowed. One demanded three phrases in an exact order the instructions never asked for. Because every run is stored, I fixed both checks and re-graded all the past runs for free, without calling a model again.
Cost was harder to predict than with a script: the model chooses how many steps to take, every step re-sends everything before it, and Claude's thinking is billed as output. On the exception request, the input grew from 3,866 tokens on the first step to 7,361 by the fourth. My estimate for the first live runs was $1.59: a smoke test (one quick run to make sure nothing is broken before the full set) plus the 18 below. They cost $2.31.
Results
I ran each task once with Anthropic's model and once with OpenAI's, then twice more with Claude as the agent and an open-weight model (Qwen, served on Groq) as the second model that reads documents. The 18 runs cost about $2.10 together.
| Models | Tasks passed | Checks passed | Cost |
|---|---|---|---|
| OpenAI gpt-5.6-sol, with gpt-5.6-luna reading documents | 8 of 8 | 82 of 82 | $0.55 |
| Anthropic claude-opus-5 in both roles | 7 of 8 | 81 of 82 | $1.30 |
| Claude, with Qwen reading documents (two tasks) | 1 of 2 | 21 of 22 | $0.24 |
Claude's one miss was formatting. A draft message to a borrower had to end by saying the lender would review the completed package. Claude's said it, then added a note for the loan specialist below it, which the instructions rule out. It's one of the two checks I'd had to loosen, and this was a legitimate fail: the fixed check still requires that line to come last. With one run per task, the table isn't a ranking. It's a reason to run each task several more times.
The right answer that failed
In the failed Qwen run, Claude handed the mortgage statement to Qwen to pull out three values: the borrower's name, the loan balance and the interest rate. Qwen returned nothing. Claude then read the statement itself, built a tidy table showing all three matched the application, and added: "The comparison above is based on my direct read of the statement text, not on a completed automated evidence check." Then it marked the task complete.
Every value was right, and the caveat was honest. It still failed, because the task required the automated check to actually run, and it hadn't. Claude simply worked around it. A test that grades only the final answer would have passed it. In a regulated workflow, the record has to show that the verification happened, not that the AI was confident it would have passed. Outside regulated work, an agent that works around a broken step might be exactly what you want. In lending, it's the gap between a mediocre agent and a good one: both give right answers, and only one can show the work behind them.
The surprise: a bug outside the AI
The finding I didn't expect wasn't about the AI at all. My rules engine, the deterministic code at the center of the demo, doesn't look at whether the evidence behind a value conflicts. On two test files with conflicting documents, it still recommends a lender. In the task that asks whether one of those files is ready to submit, the agent escalated only because it read "Conflicting evidence" in the application record and stopped. The AI was covering for the part of the demo I trusted most and checked least. It's deterministic code; how could it fail?
I found it only because I built the agent. So the agent was worth building twice: for what it does, and for handing me my next to-do on the demo itself.
Under the hood, for builders
Anthropic's guide to building agents suggests starting with the model APIs directly, because frameworks "can obscure the underlying prompts and responses." I took that to heart. The same loop runs unchanged on Anthropic's Messages API, OpenAI's Responses API and Groq's Chat Completions.
The hardest part was carrying each provider's reasoning across tool calls: Claude's thinking blocks have to go back unchanged and in order, OpenAI's encrypted reasoning has to be requested and replayed, and Groq doesn't carry reasoning across turns at all. Next came schemas, which each provider enforces differently, so zod checks every tool call on all three. I turned the SDKs' own retries off, so the loop's retries are the only ones and every retry shows in the record.
107 offline tests drive the loop with a scripted model, and any accidental network call fails the suite. The live runs never hit a budget or loop limit, so those tests provided evidence the guardrails are working as intended. Every live run showed its worst-case cost first and waited for my go-ahead.
Limits, and what's next
- There was only one run per task, so how much the results vary is still unmeasured.
- These were eight small, synthetic tasks. The inputs were small enough that trimming never kicked in.
- The agent runs on my machine, with no public demo or trace viewer yet.
- Groq's free tier is too rate-limited for Qwen to run the whole agent, only the document step.
- I wrote the spec and made the design calls; Claude Code wrote the code.
Next on my to-do list: run every task several times, fix the rules-engine gap, and add the agent's results and a step-by-step trace viewer to /evals.
What a first attempt taught me: it's valuable to understand how an agent loop works conceptually, even if a model writes the code. That understanding goes into the spec, and the spec is the difference between an agent that works in a demo and one that holds up when runs go wrong. (A real fintech would be a heavier lift, obviously.) For anyone putting an agent into regulated work: keeping the decision in code is well-worn advice, and still good, but test that code as hard as you test the AI. Let the agent decide when to ask, and grade what it did, not just what it said.