Discovery, Specs and Prototypes: Faster Without Hollowing Out

Customer discovery has four phases, and AI changes them very unevenly (the product discovery lesson in the becoming a product manager course covers the method itself):

Synthesis is where the time savings are largest, because every hour of interview becomes thousands of words of transcript and a study runs to a dozen or more interviews. It is also where the value of discovery lives, which is why it deserves the most care.

Key idea: AI makes the middle of discovery cheap. It does nothing for the part that makes discovery worth doing, which is talking to real people and noticing what surprised you.

Work interview by interview, keep the quotes attached, and put a person between the clustering and the conclusions.

The tempting workflow is to paste twenty transcripts into one prompt and ask for the key themes. Two things go wrong.

Convention. A language model produces the most typical structure for material of that shape. Ask for themes from customer interviews and you get the themes that customer interviews usually contain: onboarding friction, wanting integrations, price sensitivity. They will be present, because they are nearly always present. The finding that was actually new, the one participant whose workaround revealed a different job entirely, is the least typical thing in the pile and the most likely to be smoothed away.

Position. Liu and colleagues' study Lost in the Middle, published in Transactions of the ACL, found that language models use information at the beginning and end of a long input far more reliably than information in the middle, even models built for long contexts. In one giant prompt, interviews eleven to fourteen get less attention than one and twenty.

Gotcha: Both failures produce output that looks complete. The fix is structural: extract per interview, where each transcript gets the model's full attention, then cluster the extractions.

The unit of synthesis output should be a card that makes the claim checkable, not a paragraph that makes it persuasive.

The quotes are the safeguard. A model asked to produce themes with verbatim quotes and participant IDs can be checked in minutes by searching the transcripts, and a theme whose quotes do not exist, or do not say what the claim says, is caught before it reaches a roadmap.

In practice: Ask the model explicitly for counter-evidence on every theme. Left alone it reports agreement, because agreement is what a summary of themes usually contains.

Several tools offer "synthetic users": a language model answering interview questions in the persona of your target customer, instantly and without recruiting anyone. Nielsen Norman Group researchers Maria Rosala and Kate Moran compared synthetic users against three of their own real studies and concluded that synthetic-user responses for many research activities are "too shallow to be useful".

Their specific findings:

• Sycophancy: synthetic participants tended to please rather than report honestly.
• Idealised behaviour: asked about online courses, synthetic users claimed to have completed all of them, while real participants typically abandoned courses partway.
• No behaviour at all: a model can only produce imagined experiences, never observed ones.

They did list acceptable uses: preparing for real research, early desk exploration of an unfamiliar user group, generating hypotheses to test with real people, and drafting proto-personas to be refined.

Key idea: A synthetic user can help you write better questions. It cannot answer them. The synthetic consumers course on AnyLearn covers the quantitative version of this debate in depth.

A model can draft a complete-looking product spec from a paragraph. The draft will have every conventional section, and it will be generic in exactly the places where your product is specific. The stronger use inverts the roles: you write the spec, the model attacks it.

You are reviewing a product spec before engineering starts.
Do not rewrite it. Instead list, with the section each refers to:
1. Edge cases the spec does not handle (empty states, errors,
   permissions, limits, concurrency, migration of existing data).
2. Assumptions the spec relies on but never states.
3. Requirements that cannot be tested as written, with a
   suggested measurable acceptance criterion for each.
4. Questions an engineer would ask in the kickoff meeting.
Spec follows:
---

This keeps authorship, and therefore the understanding, with the person accountable for the product, while using the model for what it does well: recalling the long checklist of things specs usually forget.

Try it: Run this prompt on the last spec your team shipped, then compare its list with the bugs and change requests that actually came up. The overlap tells you how much to trust it.

Even a good model-drafted spec has a characteristic gap, and it is worth predicting before reading on.

This is why the critique workflow matters. A model reviewing your spec can ask "why was bulk editing excluded?" A model writing your spec simply leaves the question unasked, and the team rediscovers the answer in a planning meeting three months later.

AI coding tools let a product manager turn a description into a working, clickable prototype in an afternoon, which changes what discovery can test. The question is always what the prototype is for.

Gotcha: A working prototype looks finished. Stakeholders who see one reasonably ask why shipping it will take a quarter. Label prototypes as throwaway, keep them out of production repositories, and state up front which of the questions above each one exists to answer.

Pulling the lesson together into rules that survive a busy week:

1. Talk to real people. No generated participant replaces a single interview; use synthetic personas only to sharpen your questions.
2. Remove identifiers first, and use only tools approved for personal data.
3. Extract per interview, then cluster. Never one giant prompt for the whole study.
4. Every theme carries quotes, a count and counter-evidence, and you spot-check the quotes against transcripts.
5. Write down what surprised you after each call, before any AI summary exists, and check the final themes against that list. If your surprises are missing from the themes, the synthesis regressed to convention.
6. You write the spec; the model critiques it.
7. Every prototype states its question and is labelled as throwaway.

In practice: Rule 5 is the cheapest and the most protective. Your own notes on what surprised you are the one input the model cannot generate, and they are exactly what convention erases.