Asking a Question the Data Can Actually Answer

Analysis usually starts with a question from someone who does not know exactly what they want, and the most common failure is answering it as asked.

Are our customers happy? Is marketing working? Are we getting more efficient? Should we hire more people? Each of these sounds like a question and none of them is answerable, because none specifies what would count as an answer.

What turns them into answerable questions is a specific set of decisions you must make and should make explicitly.

Which population? All customers, or the ones who bought this year, or the ones still active? Each gives a different answer and the difference is often larger than the effect you are looking for.

What measure? Happy could be a survey score, repeat purchase rate, complaint volume, or retention. These frequently disagree with each other, and choosing after you see the results is how analysis becomes advocacy.

Over what period? And compared to what, since lesson one of the storytelling cursus established that a number without a comparison means nothing.

And what would count as a meaningful difference? Deciding in advance that a change under two percent is noise prevents you from discovering significance in a fluctuation afterwards.

So the first output of any analysis is not a number. It is a restatement: you asked whether customers are happy; I am going to measure repeat purchase rate among customers who bought in the last twelve months, compared with the same cohort a year earlier.

That sentence takes five minutes and it prevents the most common failure in this discipline, which is confidently answering a question nobody asked.

The single most important fact about business data, and the one most often forgotten: it was not collected to answer your question.

Sales records exist to invoice customers. The support ticket system exists to track work. The website analytics exist because someone installed a tag. The customer database exists so that orders can be despatched. Each was designed for an operational purpose, and the data reflects that purpose rather than any analytical one.

What that causes, specifically.

Fields mean what the operational process needed them to mean. Order date might be when the customer ordered, when the system processed it, or when finance recognised it, and the difference matters enormously for a question about timing.

Missing data is not random. A field is blank because it was optional and busy people skipped it, which means the records with it filled in are a biased subset. Analysing only complete records quietly analyses a different population.

History is not consistent. Someone changed a category list in March. A new system was introduced last year and old records were migrated with assumptions. A definition changed and nobody documented it. Long time series are full of these, and they look exactly like real changes in the business.

And deleted or updated records may have overwritten what was true at the time. If the customer record holds their current address, you cannot answer where they lived when they ordered.

Which yields the habit that separates useful analysis from confident error. Before analysing a field, find out who enters it, when, why, and what happens if they leave it blank. That is a conversation with an operational colleague, not a data task, and it routinely changes the analysis.

There is a step almost everyone skips, and skipping it is responsible for a large share of wrong answers: actually looking at the data.

What to do before any calculation.

Count the rows. Does the number match what you expect? A sales table with far fewer rows than you have orders means a join dropped records, which is the most common silent error in spreadsheet analysis.

Look at the extremes. Sort by each important column and read the top and bottom twenty rows. This finds the negative quantities, the order dated 1900, the test record with a value of nine million, and the customer called Test Test. These are present in almost every real dataset and they wreck averages.

Count the blanks in each column, and ask why they are blank rather than filling them.

Check the categories. How many distinct values does this column have? A country field with two hundred and forty entries including UK, U.K., United Kingdom and england is going to produce four separate groups that should be one.

Plot the distribution of anything you intend to average, which the storytelling cursus argued for and which bears repeating because it changes findings so often.

And check for duplicates, which arrive from re-imports, from joins, and from people entering something twice.

None of this is sophisticated. It is twenty minutes with sorting and counting, and it catches the errors that no amount of downstream sophistication will fix. An elegant model built on a table that silently lost a third of its rows produces an elegant wrong answer.

How an analysis should actually proceed, with the steps people skip marked.

Start with the business question, and restate it as a measurable one: population, measure, period, comparison, and what counts as meaningful. If you cannot write that sentence, stop and go back to whoever asked.

Then find out what the data means. Who enters these fields, when, and why. This is a conversation, and it is the step most often skipped entirely.

Then inspect. Row counts, extremes, blanks, distinct values, duplicates, distributions. Twenty minutes, no calculations.

Then clean, deliberately, recording what you did. Every cleaning decision is an analytical choice: excluding outliers, standardising categories, handling blanks. These change results and they need to be visible.

Only then calculate.

Then sense-check the answer against something you know independently. Does the total match the finance figure? Does the customer count match what the sales team believes? Divergence means one of you is wrong and finding out which is the point.

And then interpret, with the caveats that came from the earlier steps attached.

The shape worth noticing. Calculation sits in the middle and is the smallest part. Most analytical errors happen before it, and most credibility is won after it, in the sense-check that catches the answer that cannot be right.

Data cleaning is treated as a technical chore preceding the real work. It is not: every cleaning choice changes the answer, and made carelessly they change it in whatever direction the analyst was already leaning.

The choices that matter most.

Outliers. Removing them is sometimes correct, when a value is genuinely an error, and sometimes it removes the finding. A handful of enormous orders may be data entry mistakes, or may be your most important customers. The rule is to investigate rather than to exclude by threshold: find out what those records actually are.

Missing values. The options are excluding those rows, filling with an average, filling with zero, or treating missing as its own category. These give different answers, and the choice should follow from why the data is missing. Filling a missing price with the average price invents revenue that did not exist.

Category standardisation. Merging UK and United Kingdom is obviously right. Merging two product categories because they seem similar is a judgement that could hide the finding.

Date handling. Which date field, what happens to records with none, and what timezone. Timezone errors quietly shift a percentage of records into adjacent days, which matters for any daily pattern.

Deduplication. What counts as a duplicate is a definition, not a fact. Two orders from the same customer for the same amount on the same day might be an error or might be two orders.

The practical protection is the one from the storytelling cursus. Decide these before you see which way they cut, and where you cannot, record the decision and check whether the conclusion survives the alternative. If your finding depends on excluding six records, that is something the audience needs to know.

A small number of arithmetic errors account for a large share of wrong business analysis, and they are worth knowing by name.

Percentage points against percent. If a conversion rate goes from two percent to three percent, that is a rise of one percentage point and a rise of fifty percent. Both are true and they sound completely different. Reporting the larger one without saying which is the most common way business numbers mislead without anyone lying.

Percentage of what. A ten percent discount followed by a ten percent increase does not return to the original price, because the second percentage applies to a smaller base. Sequential percentage changes do not add.

Averaging percentages. The average of two conversion rates is not the overall conversion rate unless the denominators are equal. To get the overall rate you sum the numerators and sum the denominators. Averaging rates across regions or periods of different sizes is a routine error and it produces numbers that are simply wrong.

Ratios with unstable denominators. Revenue per customer rising can mean revenue rose or customers fell, and those imply opposite actions. Always look at the numerator and denominator separately before interpreting a ratio.

Small denominators. A conversion rate of fifty percent from four visitors is not a fifty percent conversion rate. Rates from small counts swing wildly and get reported as findings.

And growth from a small base. Three hundred percent growth from two units to eight is technically accurate and communicates something false.

The defence for all of these is stating the raw counts alongside every rate. It costs nothing, and most of these errors become visible the moment the denominators are on the page.

The question that should follow every observed difference, and the one most often skipped in favour of explaining the difference.

Business numbers vary week to week for reasons that have nothing to do with anything. Different numbers of working days. Weather. A large customer ordering on Monday rather than Friday. Random variation in who happened to buy.

What happens in most organisations is that a number moves, and someone constructs an explanation. The explanation is usually plausible, occasionally correct, and generated regardless of whether the movement meant anything. Once stated, it gets acted on.

Some cheap protections, none requiring statistical training.

Look at the history of variation before interpreting a single move. If this metric has bounced between forty and sixty every month for two years, a reading of fifty-eight is not news. Plotting twenty-four months takes a minute and prevents most of this.

Count how many things you are watching. If you monitor thirty metrics weekly, some will hit a record every week by chance alone. Reacting to whichever moved most is guaranteed to react to noise.

Be suspicious of explanations that arrive instantly. If someone can explain a move within seconds, they could probably have explained the opposite move just as fast.

Ask whether the change is large enough to act on, separately from whether it is real. Many statistically detectable changes are too small to justify doing anything.

And for anything important, the machinery in this catalogue's cursus on knowing what actually works applies: comparison groups, and why before-and-after misleads. This lesson is about not fooling yourself with basic arithmetic. That one is about establishing causes, which is a harder problem.

The habits that catch most errors, assembled into something you can run.

Can I state the question as population, measure, period and comparison? If not, I do not yet know what I am answering.

Do I know what each field I used actually means, from the person who enters it?

Did I count the rows and confirm the number is what I expect? Did I read the extremes, count the blanks, check the distinct values, and look for duplicates?

Is every cleaning decision recorded, and does the conclusion survive the reasonable alternative? Specifically, does it survive not excluding those outliers?

Are the raw counts stated alongside every rate and percentage? Have I said percentage points where I mean percentage points?

Did I look at the numerator and denominator separately for every ratio?

Does the answer match something I know independently? A total that disagrees with finance by ten percent means something is wrong, and it is worth more than any refinement of the analysis to find out what.

Have I plotted the history to see whether this movement is unusual, and counted how many metrics I am watching?

And could I hand this to a colleague and have them reproduce it? If the analysis exists only as a sequence of manual steps in a spreadsheet nobody else can follow, it is not checkable, and unchecked analysis is how confident errors reach decisions.

That last point is the one that compounds. An analysis someone else can rerun is one that can be corrected. One that cannot be reproduced has to be believed or discarded, and organisations usually choose to believe it.