Last time I argued that the hardest part of building AI services is making enterprise data legible, and that the agent is the thin, replaceable layer on top of a much harder foundation. This is about the layer just above that one. It is the difference between an impressive demo and something a business will pay for by the result.

Here is the shift underneath all of it. A traditional services business bills for effort: hours, headcount, time and materials. You sell the work. An AI-native services business wants to sell the result instead: the migrated system, the resolved ticket, the validated dataset. It charges for that rather than for the time it took. That is a much better business. It is also a much harder promise to keep, because the moment you are paid for outcomes, every wrong answer is your problem, not the client's.

So the question that decides whether you can sell an outcome is not "how good is the agent on average." It is "what does the agent do when it is wrong, and does it know."

You cannot just ask the model how sure it is

The naive version of confidence is to ask the model: have it rate its own certainty, or read the token probabilities, and route on that. This does not work, and it fails in the most dangerous way, because language models are confidently wrong. Their self-reported certainty is poorly calibrated, and their token probabilities measure fluency, not correctness. A model will happily assign high probability to a smooth, plausible, completely fabricated answer.

Route on that signal and you will auto-approve exactly the errors that look most convincing. The calibration has to come from somewhere outside the model's own opinion of itself.

What confidence actually is in production

We treat confidence as a composite signal, assembled from sources that are allowed to disagree with the model:

• A verifier. A second model whose only job is to check the first one's answer against the evidence, prompted to look for reasons it is wrong rather than reasons it is right. Agreement between a generator and an adversarial verifier is a far stronger signal than either alone.

• Self-consistency. Sample the answer several times. If the agent lands in the same place across independent attempts, that stability is evidence; if it scatters, that is a flag, no matter how confident any single run sounded.

• Grounding. Is the answer actually supported by the records the agent retrieved, or did it drift off the source? An answer that cites and matches real data beats one that merely reads well.

• Rule validation. Cheap, deterministic checks the answer must pass: types, ranges, referential integrity, business constraints. These catch a surprising amount of confident nonsense for almost no cost.

None of these is sufficient alone. Combined, they produce a number that means something, because it is correlated with being right rather than with sounding right.

The routing

Once you have a signal you trust, the behavior is simple to state and hard to tune. Three bands:

• High confidence: act. The agent completes the work and moves on. No human touches it.

• Medium confidence: review. The work is done but held; a person confirms or corrects it before it takes effect.

• Low confidence: escalate. The agent stops and hands the problem to someone, usually with a note on what it was unsure about.

I think of the line between act and review as the trust boundary. Everything above it runs as software. Everything below it is still a service delivered by people, with the agent assisting. The whole game is moving that boundary up, safely, over time.

Setting the thresholds

The thresholds are where the engineering lives, because they encode an economic decision, not just a technical one.

Set the act threshold too low and the agent auto-completes work it should have flagged. You get speed, but errors leak into production and you are paying for them. Set it too high and almost everything routes to a human. The system is safe, but it is barely software, and your margins start to look like a staffing agency's. The threshold is the dial between those two failure modes.

You choose the setting with data, not taste. Run the agent against a labeled set where you know the right answers, and at each candidate threshold you can measure two things: how often an auto-approved answer is actually wrong (the error that reaches the client) and how much work is being sent to humans (the cost that eats the margin). That gives you a curve, and you pick the point on it that matches the cost of being wrong in that specific workflow.

That cost varies wildly, which is why there is no global threshold. An agent suggesting a category for an internal ticket can be wrong sometimes; the cost is a small correction. An agent posting a financial transaction cannot; one wrong auto-action can outweigh a hundred needless escalations. Same architecture, completely different boundary, because the asymmetry between a false act and an unnecessary review is different. You tune to the asymmetry.

The human-in-the-loop earns the agent its autonomy

The reviews and escalations are not only a safety net. They are the training signal. Every time a person confirms, corrects, or overrides an answer, you learn whether the confidence score was telling the truth, and you feed that back.

That feedback is what lets the trust boundary move. As the agent accumulates corrected examples on a kind of work, two things improve together: the agent gets better at the task, and the confidence signal gets better calibrated, so you can safely lower the act threshold and let more run as software. The system becomes more autonomous not by decree but by earning it, one verified batch at a time. Week one, most of a new workflow sits in the review band. Month three, the bulk of it is above the trust boundary and a small team handles the long tail.

Why this is what lets you sell an outcome

Bring it back to the business. A client will not pay for an outcome they cannot trust, and they will not trust a system whose failure mode is silent, confident error. Confidence routing is what makes the failure mode safe: the system does the work it is sure about and visibly flags the rest, so the worst case is a review queue rather than a wrong answer shipped at scale. That is a bounded, inspectable risk, which is the kind a business can actually sign off on.

So confidence routing is not really a model feature. It is the thing that turns "an agent that is usually right" into "a service you can put your name on and bill by the result." The data layer makes the agent possible. The confidence layer makes it sellable.

Next in the playbook: the people. When the agent does most of the work and routes the rest, what is the small human team actually for, and how do you organize it: the pod around the fleet.

— Nikhil

Two years ago, growing a services business meant hiring people. The work was done by humans, so more work meant more humans, and the margins were whatever the labor math allowed. We are now building that same kind of business, enterprise system integration, to run mostly on agents instead of people. Engagements that used to take a team of twenty most of a year, we deliver with a small pod and a fleet of agents in weeks.

When people hear that, they assume the hard part is the agents. It isn't. The agents are the easy part. The hard part, the part that decides whether a project succeeds or quietly falls apart three weeks in, is everything underneath them: making the customer's data legible enough that an agent can reason over it without producing confident nonsense.

This is the least glamorous engineering I know of, and it is most of the job. Here is what it actually involves.

Semantic debt

Every enterprise is carrying what I have started calling semantic debt. It is like technical debt, except instead of shortcuts in the code, it is meaning that was never written down.

A schema tells you a table is called CUST_MASTER and has a column status_2 of type VARCHAR. It does not tell you that status_2 is the field people actually trust, that status was deprecated in 2017 but never dropped, that the values A/I/H mean active, inactive, and a third state nobody can define anymore, or that a quarter of the rows are test records from a migration that half-finished. None of that is in the database. It lives in the heads of people, some of whom have left.

Semantic debt shows up in a few consistent shapes:

• Schema without semantics. You have column names and types, but not what they mean, which are authoritative, or how they relate.

• Entity fragmentation. The same real-world thing is recorded several ways across systems that were bought rather than built, with no shared key. "Acme Corp," "ACME Corporation," and "Acme Inc." are three rows and one company.

• Lineage loss. A number on a report exists, but how it was produced (which tables, which filters, which business rules) is gone. Two reports show different revenue and both are "right," because they define revenue differently and nobody wrote it down.

• Silent contradiction. The data disagrees with itself in ways nothing flags: a fact row whose foreign key points at a dimension that does not exist, a date that is sometimes the order date and sometimes the ship date depending on the source.

An agent pointed at this does not see a mess. It sees confident inputs and produces confident outputs, and the outputs are wrong in ways that are expensive to catch later. Garbage that looks like an answer is worse than no answer.

Why the obvious approaches fail

The first instinct is to hand the model the raw schema: dump the DDL into the context, add a few sample rows, let it figure things out. This works in a demo and fails in production. The model invents joins that are not valid, picks the wrong grain, and chooses whichever of the three revenue definitions sounds most plausible. It has no way to know which status field is real, because that information is not in what you gave it.

Retrieval over the raw tables has the same problem one layer up. You can embed and search the data all you like; if the meaning was never captured, similarity search just retrieves the same ambiguity faster.

The thing the agent needs does not exist yet when you arrive. You have to build it.

Building the legibility layer

The legibility layer is an explicit, verified model of what the customer's data means. It is produced by a pipeline of six stages, with a human-in-the-loop wrapped around all of them.

1. Profile before you interpret. Before a model is allowed an opinion about a column, we profile it directly against the data. For each column we compute cardinality and null rate, the distribution of values, and a format signature. Do the values look like emails, ISO dates, currency, a short coded enum? We infer candidate primary keys from uniqueness, and candidate foreign keys from value containment: if 98% of orders.cust_ref values appear in cust_master.id, that is a join the declared schema never told you about. Statistics come first, because they ground everything downstream and catch the model the moment it starts to guess.

2. Infer meaning, then have a second model attack it. Each table and column goes to an agent along with its profile, a sample of real rows, and the names of its neighbors. The agent proposes what the field means, its semantic type, whether it is authoritative or a deprecated remnant, and how it relates to other tables, attaching a confidence to every claim. A second model runs as an adversarial check: same evidence, but its job is to falsify the first proposal. We route on the result in three bands. Above 0.9 with agreement auto-accepts. Between 0.7 and 0.9 is queued for human review. Below 0.7 escalates. The bands exist because a wrong semantic label here is not a local error; it is paid back, with interest, by everything built on top of it.

3. Resolve entities. The three-different-Acmes problem is its own discipline. Comparing every record to every other record is quadratic and hopeless at enterprise scale, so we block first, generating cheap candidate keys (a normalized name plus a postal code, say) so that only plausibly-matching records are ever compared. Each candidate pair is then scored with a mix of deterministic rules and embedding similarity, the scores are thresholded, and the surviving links are clustered with union-find so that "Acme Corp ↔ ACME Corporation ↔ Acme Inc." collapse into one entity with one key. Pairs near the threshold, the genuinely ambiguous ones, go to a person. The output is a resolved entity the rest of the system can rely on.

4. Recover lineage. Where a metric is produced by transformations (SQL views, stored procedures, pipeline code), we parse them into syntax trees and build a column-level lineage graph: this report field comes from that view, which sums this column filtered by that flag. The business rules buried in the WHERE clause become part of the definition. "Revenue" stops being a word and becomes a specific, traceable computation everyone can point at and argue with.

5. Assemble the model. The verified pieces are assembled into a layered model. Raw landing data (bronze) is cleaned and conformed (silver) and then modeled (gold). The gold layer is shaped as a star schema: fact tables declared at an explicit grain, surrounded by conformed dimensions, so that a question like "revenue by region by quarter" has exactly one correct way to be answered. The grain is the contract. Most of the silent contradictions in enterprise reporting are really two numbers computed at two different grains and compared as if they were the same.

6. Detect what is broken. The same machinery that builds the model is run as a set of assertions against it: referential integrity (orphaned facts pointing at missing dimensions), grain violations (a "unique" business key that is duplicated), cross-source reconciliation (the same metric from two systems, and the delta between them), and temporal anomalies. A lot of the value we deliver in the first weeks of an engagement is simply telling a customer the truth about their own data, with the specific offending rows attached.

At the end of this you have something the customer never had: a written, verified account of what their data means. That artifact is the product. The agents that do the actual delivery work (migration, validation, answering questions, monitoring for drift) all stand on top of it.

Why we let agents do it, and why we watch them

Two things make agents well-suited to this. The work is enormous and repetitive, far too much for a human team to do by hand across thousands of columns and hundreds of tables. And it is exactly the judgment-under-ambiguity that language models are good at, given enough grounding.

Two things make it dangerous. Models are confident when they are wrong, and this is precisely the place where a wrong answer gets baked into everything downstream. So the agents never run unsupervised. Every inference is grounded in real profiles and samples rather than the model's prior, every output carries a confidence, a second model checks the first, and a human owns the thresholds where the system stops and asks. The goal is not full autonomy. It is a system that does the overwhelming bulk of the work, knows the exact boundary of what it is sure about, and widens that boundary as it earns trust.

Why this is the whole game

A services business can run on software not because the agents are clever, but because this layer, once built, is leverage. The semantic model is reusable. The entity resolution compounds. The lineage stays recovered. The next engagement at a similar customer, on a similar source system, starts further along than the last one did. The labor you would have spent on twenty people for six months collapses into the work of building and verifying this artifact once, after which everything you deliver runs on it.

That is why services as software is an engineering story and not a margin story. The margin is a consequence. The cause is whether you can take an illegible enterprise data estate and make it legible faster and more reliably than a room full of consultants could.

This is the first piece of the playbook. The next ones go up the stack: the delivery agents that run on top of this layer, how confidence routing lets you sell an outcome instead of hours, and how a small pod of people is organized around a fleet of agents. The foundation comes first, because nothing above it works without it.

— Nikhil