Analytics, Experiments, and the Decisions You Still Own

Analytics assistants turn a question like "what share of August sign-ups activated?" into a database query, run it, and return a number or a chart. The mechanism is simple: the model is given your question plus a description of the tables and columns, writes SQL, and a tool executes it.

For a product manager this removes a real bottleneck, the queue for an analyst's time, and it makes exploratory questions cheap enough to ask.

It also introduces a failure mode spreadsheets never had.

Key idea: A query with a syntax error fails loudly. A query with a logic error, the wrong join, a missing filter, a different definition of "active", runs perfectly and returns a plausible number. Nothing in the output distinguishes it from a correct one. The risk in AI analytics is not errors; it is silent errors that look like answers.

The BIRD benchmark, a widely used test of text-to-SQL on realistic databases, measures whether a generated query returns the same result as a correct one. Its leaderboard reports a human baseline set by data engineers and database students.

Even the best system gets roughly one question in six wrong, and those entries are given hand-written hints about the data ("oracle knowledge" on the leaderboard) that a real company's analytics tool usually lacks.

Gotcha: Your own warehouse is likely harder than the benchmark: undocumented columns, legacy tables, test accounts mixed with real ones. Treat the benchmark as a ceiling on accuracy, not an estimate of it.

Asked "what share of August sign-ups activated?", where the team defines activation as creating a project within seven days of signing up, an assistant might produce:

SELECT COUNT(DISTINCT e.user_id) * 1.0 / COUNT(DISTINCT u.id)
FROM users u
LEFT JOIN events e
  ON e.user_id = u.id AND e.name = 'project_created'
WHERE u.created_at >= '2026-08-01' AND u.created_at < '2026-09-01';

It runs. The number is reasonable. It is also inflated twice: it counts a project created at any time, not within seven days, and it includes the team's own internal test accounts. The corrected query:

SELECT COUNT(DISTINCT e.user_id) * 1.0 / COUNT(DISTINCT u.id) AS activation_rate
FROM users u
LEFT JOIN events e
  ON e.user_id = u.id
 AND e.name = 'project_created'
 AND e.occurred_at < u.created_at + INTERVAL '7 days'
WHERE u.created_at >= '2026-08-01' AND u.created_at < '2026-09-01'
  AND NOT u.is_internal;

Neither error is visible in the result. Both are visible in the query, which is why the query must always be shown.

Most wrong analytics answers are not the model misunderstanding SQL. They are the model choosing a reasonable definition that differs from your company's.

Each column choice is defensible, and each gives a different number. The fix is to give the assistant your definitions rather than hoping it guesses them, usually as a documented metrics catalogue or a semantic layer that defines each metric once in code.

In practice: Before trusting any AI analytics tool, ask it five questions whose answers your team already knows from a trusted dashboard. If any disagree, find out which definition it used before asking it anything new.

An A/B test readout is a natural thing to hand to a model: a table of metrics, confidence intervals and a decision to make. Used well, it helps in specific ways (the A/B testing at scale course covers the statistics themselves):

• Explaining an interval or a p-value in plain language for a stakeholder update.
• Checking the plumbing: asking whether the split between variants matches the planned ratio, since a mismatch usually means a bug, not a result.
• Listing guardrail metrics that should have been watched, from the change's likely side effects.
• Drafting the readout in the team's standard format once the decision is made.

What it should not do is decide whether the result is real. That judgement depends on things the model usually cannot see: which metric was declared primary before the test started, how many other metrics were examined, and whether the test was stopped early when it looked good.

Key idea: The statistics in an experiment are only valid relative to a plan made in advance. A model shown only the results has no way to know what the plan was.

There is a second risk, and it comes from the person rather than the data. Sharma and colleagues at Anthropic, in Towards Understanding Sycophancy in Language Models, showed that five state-of-the-art AI assistants consistently tended to give responses matching the user's stated views over truthful ones, a behaviour called sycophancy, and traced it partly to human preference data that rewards agreement.

A product manager who wants a feature to have worked is exactly the user sycophancy serves. "Isn't this lift significant?" invites yes. Asking again with a different framing, or a different metric, until the model agrees, is the conversational version of testing many metrics and reporting the one that passed.

A protocol that resists both problems:

1. Paste the pre-registered plan first: primary metric, minimum effect, planned duration.
2. Ask neutrally: "Given this plan, what does the result support?"
3. Ask for the case against: "Argue that this result should not be shipped."
4. Record the first answer rather than the best one.

Gotcha: If you find yourself rephrasing a question because you disliked the answer, that rephrasing is the analysis going wrong.

Adopting AI tools is itself a product decision, and it deserves the same evidence standard as any other. One of the most carefully designed measurements comes from software development, a neighbouring field. METR ran a randomised study in which 16 experienced open-source developers completed 246 real issues from their own projects, each randomly assigned to allow or forbid AI tools.

The transferable lesson is about measurement, not about developers. Felt productivity and measured productivity can diverge sharply, because AI makes work feel easier and more fluent even when it is not faster.

Try it: Pick one recurring task, such as the weekly update or interview synthesis, and time it for a few weeks with and without AI, including the time spent checking the output. Decide from the log, not the feeling.

AI can inform every product decision and should own none of them. A useful way to set the boundary is by decision type:

Disclosure follows from ownership. When a stakeholder acts on your analysis, they are relying on you. Saying which parts were AI-assisted and how you checked them, for instance "query generated by the analytics assistant, reviewed against the metrics catalogue", costs one line and protects your credibility when a number turns out wrong.

Across the three lessons of this course, the same division keeps appearing. AI compresses the production of artefacts: summaries, specs, queries, prototypes, readouts. It does not compress the three things the role is actually accountable for:

1. Contact with reality. Real customers in real conversations, and the surprises that come from them. No synthesis, however fluent, replaces the input.
2. Definitions and plans. What "activated" means, which metric was primary, what counts as success. Set before the data arrives, these make every AI output checkable; left vague, they make every output unfalsifiable.
3. The call. Choosing, saying no, and owning the outcome.

Key idea: The practical test for any AI use in product work is whether you could explain and defend the result to a sceptical colleague without mentioning the tool. If you could, the tool saved you time. If you could not, it made a decision on your behalf, and you did not notice.