The short version. Horus is a lead-vendor intelligence platform for insurance agencies. Its chat agent answers questions like "which vendor should get more budget next month" in plain English, off live production data, and answers them well enough to spend against. The analysis behind that is genuinely cumbersome: several marketplaces, a different export format from each, costs that change after the fact, and product tiers that are named differently everywhere. Horus does that reconciliation underneath, and every figure is calculated by the platform from a single definition rather than worked out by the model on the spot, so an answer only moves when the data moves and every recommendation comes back with the numbers and reasons behind it. A graded evaluation suite runs the real model before each release to keep it that way.
At a glance
- Sector: insurance lead buying, multi-tenant across agencies
- Problem: agencies buy leads from several marketplaces at once and could not tell which were actually profitable, because the data arrived inconsistent and the analysis needed to drive budget decisions
- Approach: an assisted import that proposes its own column mapping, a reconciled data layer, a deterministic metrics and classification engine, and a tool-calling agent on top that answers questions and explains the reasoning
- Tech: Next.js on Vercel, Postgres with row-level security, Vercel AI SDK, Claude Sonnet 5
- Outcome: live in production, multi-tenant across agencies; the analysis that used to be a monthly spreadsheet exercise is available whenever someone asks for it
The challenge: budget decisions made on unreliable arithmetic
An insurance agency buys leads from several marketplaces simultaneously. Each one sends data in its own shape, on its own schedule, with its own definition of what a lead is and what it cost.
That creates a problem that looks like reporting and is actually accounting. Before anyone can ask which vendor performs best, four things have to be true:
- Identity. Leads arrive more than once, across files and across months. Without a stable identity per lead, every re-import inflates the numbers.
- Cost. What a lead cost is not what the export says it cost. Returns, credits and billing lag mean the real figure only exists once the lead is reconciled against the billing ledger, which arrives separately and later.
- Labels. The same product tier appears under different names in different exports, so like is not being compared with like.
- Definitions. Cost per lead is easy. Cost per sold household is the number that decides budget, and it has to mean exactly one thing everywhere it appears.
All four assume the data arrived in a usable shape, and it does not. Every month someone has to pull a fresh export from each marketplace portal, each portal names its columns whatever it likes, those names change between exports, and the person doing the upload is an agency operator rather than a data engineer.
Do this by hand and it is a spreadsheet exercise that takes days, has to be redone every month, and is wrong in a way nobody notices if a single step is skipped. Skip it and the ranking is confidently wrong: a vendor looks cheap because its credits have not landed yet.
Only then does the actual question become answerable: given all that, which vendors should get more money next month and which should be paused?
The people who need that answer are agency owners and their staff. They are not analysts, and they should not have to be.
The approach: solve the analysis once, then let people ask
What it does
Someone asks, in plain English, which vendors are worth more budget next month. They get a ranked answer with the reasoning shown: how each vendor performs on cost per sold household, how that compares to its peers, which direction it is trending, and whether the data behind it is complete enough to act on.
The same conversation covers the rest of the analysis an agency would otherwise commission: performance by vendor, by product tier and by lead kind, geography down to county and ZIP including where spend is being wasted, and a budget allocation scaffold with floors and ceilings per vendor, rendered against a stated monthly budget.
Each answer is bounded honestly. If the current period is partial or a vendor has gone stale, the answer says so instead of presenting a thin month as a finished one.
Getting the data in, whatever shape it arrives in
The upload is where the month's numbers are usually lost, because one column mapped to the wrong field makes everything downstream quietly wrong. So Horus does the mapping itself and asks the operator to confirm rather than compose.
When a file is scanned, a profiler reads the values in each column, not just the headers, and a model uses that evidence to propose which column is which and why. What makes that safe is what happens to the proposal next. It is checked back against the file, and a column that does not exist, a duplicate assignment or an unrecognised field rejects the whole suggestion rather than part of it, so an invented column can never reach the screen. Personal data and a fixed list of columns that must never be imported are stripped afterwards, whatever confidence the model claimed. Anything the column evidence contradicts is flagged for a human to confirm instead of being filled in. Underneath it all, a deterministic matcher fills whatever is left and takes over completely if the model call fails, times out or is rejected.
The result is an import that is quicker and much harder to get wrong, and that stays deterministic. The model is a convenience layer over a strict importer, never in a position to make the import incorrect. The rules that operators used to have to remember are in the platform too: which fields each vendor requires, which vendors need a billing statement alongside the lead report before their spend means anything, and which exports are live transfers rather than data leads.
The layer underneath the agent
Most of the engineering is below the chat interface, and it is what makes the answers true.
Ingestion and reconciliation. Each marketplace gets its own parser for its own export format. Leads are deduplicated on a stable identity token rather than a display field, because display fields get corrupted in transit, so re-uploading a file or importing two exports that overlap by a fortnight does not inflate anything. Cost is reconciled against the billing ledger on every ingest path, without exception: that rule is enforced by a test that fails the build if any import route forgets it, because a single unreconciled lane is enough to make a whole vendor look cheaper than it is.
Label reconciliation. Raw vendor labels are mapped to a canonical product taxonomy, so tiers compare across sources.
One definition per metric. Cost per sold household is calculated in exactly one place, and everything that reports it calls that. The agent's glossary is generated from the same source, so what the model tells a user a number means and what the system actually computed cannot drift apart.
A deterministic classifier. The recommendation logic, which sorts each vendor into scale, test, hold, reduce or pause, is code, not model reasoning. It reads several time windows, compares each vendor against a peer baseline, folds in trend direction and data staleness, and returns a bucket with reason codes. The agent presents that result and is explicitly instructed not to re-derive it. The same inputs produce the same recommendation every time, which is not a guarantee a reasoning model can make on its own.
An allocation scaffold. Budget floors and ceilings per vendor are derived from the bucket assignment by a fixed weighting, optionally rendered into pounds or dollars against a stated monthly budget. Also pure code.
Where the intelligence sits
Next to all of that, the agent's job is deliberately small: understand the question, pull the relevant parts of the engine, and explain the result in the terms the question was asked in. It performs no arithmetic of its own, so an answer is a property of the platform rather than of the model that happened to serve it, and the model can be upgraded without anything an agency sees changing.
That makes this a different shape from a text-to-SQL agent, which composes its own queries against the warehouse. Both are valid, and the choice follows the question space rather than taste. Here the questions are a known set and the difficulty is underneath them, so the analysis is encoded once and the agent reads it. Where users need to slice data in ways nobody can list in advance, generating SQL is the right call, and that architecture is covered in our text-to-SQL case study.
The agent layer
On top of that sits a single agent loop with the eleven tools, and a set of rules about how it is allowed to answer.
Scope is not a parameter. Which agency's data a request can see is bound into the tools as a closure, not passed as an argument the model fills in. There is no field for the model to get wrong and no injected instruction that can widen it. Underneath, every database connection is row-level-security scoped, so the boundary is enforced twice, once in the application and once by the database.
The rules live in the prompt, and each one is tested. A minimum window of data before it will recommend an action. Minimum sample sizes before it will quote a rate. Cite only values returned by a tool. Caveat the answer when the data-readiness tool reports the current period is thin. These are behavioural rules, so they belong in the system prompt rather than scattered through code, but each one has an evaluation case behind it.
Data readiness is checked, not assumed. A dedicated tool reports whether the current period is complete, partial or sparse, and which vendors have gone stale. Mid-month, that is the difference between a useful answer and a confidently misleading one.
Evaluation: knowing it still works next month
The evaluation suite runs the real production model against a seeded database and grades three things per case: whether the right tools were called in the right order, whether the answer contains what it should and avoids what it should not, and whether a separate model judges the answer grounded in the data actually returned.
That third check catches the failure mode everyone worries about. An answer can call the right tools and still drift into a claim the data does not support; a groundedness judge catches it before a user does.
The suite runs before agent releases, and it is what makes the system safe to change. Prompts get edited, models get upgraded, tools get added. Any one of those can quietly alter an answer that used to be right, and an analytics tool has no natural error signal: a wrong number looks exactly like a right one. We build an evaluation suite into every agentic system we ship for that reason, not just this one.
The same principle covered the rebuild. When the platform migrated from its earlier architecture to this one, the change was locked by byte-identical before-and-after output comparison, so it could be proven not to have altered a single number an agency sees.
What we delivered
Horus runs in production, multi-tenant across agencies, with the chat agent live on the current Claude model.
What changed for an agency is when the analysis exists. It used to be a monthly spreadsheet exercise, done by whoever had the patience for it and out of date within a week. Now it is there whenever someone asks, on data that reconciles itself on the way in, and it arrives with its reasoning attached, so a recommendation can be argued with rather than only accepted or ignored.
Where this approach applies
This generalises to any business where the numbers people act on have to be assembled rather than read: several sources, definitions that need agreeing, and a judgement step in the middle that decides what the data means. Lead buying, media spend, channel performance, supplier or route economics all have the same shape.
What decides whether the result is trusted is mostly not the agent:
- Do the accounting before the AI. If costs, identities and labels are not reconciled, a conversational interface just makes it faster to get the wrong number. The reconciliation is the project; the chat is the last mile.
- Put the model where it saves time, not where it decides. Horus uses a model twice: to propose an import mapping and to answer questions. Neither gets the final word. One is validated against the file it just read, the other against the metrics engine. That is the general rule, and it is usually more useful than deciding which model to use.
- One definition per metric, in one place. Every metric that appears in an answer should resolve to a single implementation, and the model's own glossary should be generated from it rather than written alongside it.
- Encode the judgement. The step that turns numbers into a recommendation is domain logic. Put it in the system where it can be inspected and disagreed with, not in a prompt where it changes shape between answers.
- Be honest about thin data. Knowing when the current period is too partial to act on is worth more than any extra breakdown, because that is where confident, wrong advice comes from.
- Bind the tenant boundary outside the model. Isolation should be structural and enforced again at the database, never a parameter the model fills in.
- Evaluate the real system before every release. Not a spot check on a good day.
Whether the model composes its own queries or selects from a set the platform exposes is a smaller decision than any of these, and it falls out of the question space once the rest is done properly.
Frequently asked questions
What can an AI data analyst actually do for a business? Answer the questions that currently need someone to build a spreadsheet. In Horus that means vendor rankings, cost per sold household, performance by county and ZIP, product-tier breakdowns and budget recommendations, all asked in plain English and answered off current data. The work it removes is not the thinking, it is the assembly: reconciling sources, applying the definitions consistently and redoing it every month.
Can AI help with messy data imports? Yes, and it is one of the better places to use it, as long as it proposes rather than decides. A model reading the actual values in each column can suggest a mapping far faster than a person hunting through a portal export, which is where import mistakes come from. Then validate every suggestion against the file, drop anything sensitive regardless of confidence, flag whatever the data contradicts for a human to confirm, and keep a deterministic matcher underneath that works when the model does not. You get the speed without letting the import become non-deterministic.
Can an AI agent be trusted to report business metrics accurately? Yes, if it does not calculate them. Fix every metric definition in code and let the model retrieve the result and explain it. A definition that lives in a prompt will drift, because the model re-reads it fresh every time; one that lives in a query does not. In Horus the model has no arithmetic to do: it selects a query and explains what comes back.
What is the alternative to text-to-SQL for natural-language analytics? Constrained tool calling: the platform exposes a fixed set of parameterized queries and the model chooses which to call and with what filters, instead of composing SQL itself. Which one fits depends on where the difficulty in your data actually is. If it is query construction across a wide, open-ended schema, generate the SQL and guard it properly. If it is reconciliation, definitions and domain judgement, that work has to be built either way, and once it exists the agent is better off reading it than re-deriving it per question.
How do you stop an AI agent leaking one customer's data to another? Do not let the model address the boundary. Bind the tenant scope into the tools as a closure rather than passing it as an argument the model fills in, so there is no parameter to get wrong and no prompt injection that can widen it. Then enforce the same boundary independently at the database with row-level security.
How do you know an AI analyst's answers stay right over time? An evaluation suite that runs the real model, not a spot check. Horus grades every case on three axes: the tools called, what the answer does and does not say, and whether a separate model judges it grounded in the data returned. It runs before releases, so a prompt edit or a model upgrade that changes an answer shows up before a user acts on it.
Does an AI analyst replace a data team? No. It removes the queue for questions that already have a defined answer. Someone still has to decide what the metrics mean, reconcile the data, and encode the domain logic. That work does not disappear, it just stops being done one ticket at a time.
Build it properly
If you want people in your business asking questions of your data instead of waiting on a report, the work that decides whether it lands is underneath the chat: getting the sources reconciled, settling what each number means, and encoding the judgement that turns those numbers into a decision. Our guide to AI agent development for business covers how that work is scoped, and an AI roadmap is the cheapest way to find out what your data needs before committing a build budget.
Dot Square Lab builds production AI systems end to end, from ingestion and reconciliation to the agent and the evaluation suite that keeps it honest. Tell us your challenge and we will tell you what your data needs first, including when the answer is that you do not need an agent at all.
