Imagine three people asking their AI assistant how many customers the business has. Marketing gets one answer, finance gets another, and operations gets a third. Each comes with a query, an explanation and a reassuring amount of confidence.
The tempting response is to work out which assistant got it wrong. Sometimes that will be exactly the right question. A bad join or a misunderstood field can produce a very convincing piece of nonsense.
But suppose you’ve checked all three queries and they do what their authors intended. You can still have three different answers, because the disagreement was about what a customer means in the first place.
Correct for which question?#
“How many customers do we have?” sounds like a simple factual question. It leaves quite a lot unsaid:
| You might count… | Which could help you understand… |
|---|---|
| Everyone who has ever bought something | The size of your historical customer base |
| Accounts that bought in the last 30 days | A particular definition of current activity |
| Households receiving deliveries | How many homes your operation serves |
| Paying business accounts | How many commercial relationships you manage |
Even after choosing one, there are more decisions. Do trial customers count? What about someone who ordered and then cancelled? Is one person with two accounts one customer or two? When does an account become inactive?
Several answers can be useful. A count of people who have ever bought something is a poor substitute for a forecast of next week’s delivery workload, but that doesn’t make the historical count incorrect.
An AI can choose a reasonable interpretation of an ambiguous question. That doesn’t mean it chose the interpretation everyone else in the organisation is using.
Agree which definitions need to be shared#
I don’t think the answer is to insist that every team uses one customer count for every purpose. Different decisions need different views of the business. The useful agreement is about what each measure means and when to use it.
If marketing reports “customers acquired”, operations reports “active delivery households”, and both are clearly defined, the difference can be perfectly healthy. The trouble starts when both numbers are called “customers” and get used interchangeably in a leadership meeting.
Someone needs to get the relevant people to agree the unit being counted, the inclusion rules, the time window and the purpose of the measure. The business owners need to accept those choices. That means making the consequences visible, helping resolve ambiguity, and getting a commitment to use the agreement consistently.
That might involve an uncomfortable conversation about targets. Changing who counts can make one team’s performance look better and another’s look worse. An analyst quietly choosing the most defensible SQL hasn’t resolved that conversation.
Keep the measuring stick still#
Agreement also needs to last long enough to make comparisons useful.
Suppose you approve a retention initiative based on “active customers” meaning people who bought in the last 30 days. Three months later, someone reports progress using a 90-day window. Both definitions might be reasonable, but a comparison between them won’t tell you whether the initiative worked.
The bar has moved since the decision was made. You could announce success without having retained a single additional customer.
This matters even if every new number is defensible in isolation. A business continues to exist, set targets and make decisions. Its measurement needs enough continuity for those decisions to mean something over time.
Definitions can change. Sometimes they absolutely should: the product changes, you discover a mistake, or the old measure turns out to be a poor guide. But make that change explicit. Record what changed, why, when, and who agreed it. Where possible, calculate the old and new measures over the same period so people can see the difference.
If you restate history under the new definition, update the comparison and explain what that means for the target. Keep a record of the original basis on which the decision was made, too. Quietly comparing a new measure with an old target is a particularly easy way to fool yourselves.
Keep the evidence behind the decision#
Being able to explain a query today isn’t the same as being able to reproduce the number presented last month.
You need the definition and query used at the time, but the underlying data may also have changed. Orders arrive late, accounts get merged, and records get corrected. Re-running the same SQL on today’s tables can give a different answer without the query having changed at all.
For important decisions, keep the result people actually saw, its reporting period and data cut-off, and a reference to the logic and assumptions behind it. Where you need to reconstruct the calculation, arrange suitable snapshots or data history as well. A saved prompt alone won’t do that job.
It can be useful to have both an updated view of the past and a record of what was known at the time. They answer different questions: how the business performed, and whether a decision was reasonable given the evidence available.
You don’t need to preserve every exploratory chart forever. But once a number supports a target, an investment or an important decision, I think it deserves more care than a chat you might struggle to find next week.
Give AI an agreed starting point#
Once you’ve agreed a definition, make it easy to find and reuse. That might mean an existing reporting model, a documented shared query, or a semantic layer which centralises metric logic. The implementation needs to fit the organisation; buying another tool isn’t a prerequisite for having the conversation.
Give the measure a clear name, document the awkward edge cases, name its owner, and provide a route for proposing changes. Ask AI-assisted analysis to identify which definition it used. If someone needs a different measure, make that difference visible rather than letting it quietly inherit the same label.
Someone still needs to lead the discussion#
Even in an organisation using AI really effectively, someone still needs to bring the relevant people together and help them reach that agreement. They need to understand why the teams disagree, draw out the consequences of each option, and get people to accept how they’ll be measured.
Metric and goal setting are part of managing people. The choices affect which work gets prioritised, whose effort is recognised, and whether a team is judged to have succeeded. People can understand a definition perfectly well and still disagree with what adopting it would mean for their targets. Getting to alignment might require changing those targets or asking a business leader to resolve competing priorities.
AI can help prepare that conversation, explain the alternatives and record the outcome. But I think we’re still a long way from handing it responsibility for effectively leading that discussion and bringing the group to alignment. That requires working relationships, judgement about the people involved, and enough organisational backing to resolve the disagreement.
It doesn’t need to be a data team doing this. A founder, a finance lead or another team leader might be the right person. But someone needs to own it. This is part of the responsibility that remains when AI takes on a data team’s tasks, wherever you choose to put that responsibility.
The documentation and shared models help preserve the agreement people reach. Writing down a technically defensible definition doesn’t establish that agreement on its own. The human work of metric setting still has to happen, even if AI can do all the SQL.
And once people agree how success will be measured, they still need to agree what to do about it. That’s the related problem in why analysis can get faster without decisions getting faster.
I’d start by taking one number that regularly causes arguments, bringing the relevant people together, and giving someone responsibility for leading that discussion. Agree what it should mean for the business and its goals, then make that agreement easy for people and their AI tools to keep using.
