Data Scientist Interview Prep: Questions and a Mock Test

Nearly every data science loop contains an experimentation round, and the good answers spend most of their time before the test runs rather than after.

The first thing to establish is the metric. Being able to distinguish the primary metric the decision rests on, guardrail metrics that must not degrade, and secondary metrics that provide explanation, is the structure interviewers listen for. A single overall evaluation criterion, agreed in advance, is what stops the analysis becoming a search for something that moved.

The second is the power calculation. You need the baseline rate, the smallest effect that would actually change the decision, and the significance and power you are willing to accept, and from those the sample size and therefore the runtime. Candidates who cannot say what effect size they would care about have skipped the only part of this that requires judgement, since the minimum detectable effect is a business question rather than a statistical one.

The third is randomisation unit. Randomising by request when the user sees several requests breaks independence and understates variance. Randomising by user when the effect spills between users, as in a marketplace or a social graph, produces interference that no amount of statistical care fixes; cluster or geographic randomisation is the usual response.

Running a full week to cover the weekday cycle, and refusing to stop the moment the result crosses significance, are both expected.

This is the single most reliable place to distinguish candidates, because the wrong interpretations are so widespread that they sound normal.

A p-value is the probability of observing data at least as extreme as yours, assuming the null hypothesis is true. It is not the probability that the null hypothesis is true. It is not the probability that your result is a fluke, and one minus the p-value is not the probability that the effect is real. A p-value of 0.04 does not mean there is a 96 percent chance the treatment works.

The practical consequences come up as scenarios. Peeking, that is, checking significance repeatedly and stopping when it crosses the threshold, inflates the false positive rate substantially, because you have given yourself many chances to cross a line that random walks cross. The answers are fixing the sample size in advance, or using a method designed for continuous monitoring such as sequential testing with alpha spending.

Multiple comparisons work the same way across metrics rather than across time: at twenty metrics and a threshold of 0.05, one significant result is the expected outcome under the null. Corrections exist, but the more useful interview answer is to nominate the primary metric before looking.

And a non-significant result is not evidence of no effect. Absence of evidence at your sample size usually means the confidence interval is wide, which is why reporting the interval rather than the verdict is the better habit.

Not everything can be randomised, and interviewers know it. Expect at least one question where an experiment is impossible and the data is what it is.

The first move is naming the confounder: something that affects both the treatment and the outcome and therefore manufactures an association between them. Users who opt into a feature differ from those who do not in ways that also drive retention, so comparing them measures the difference between those kinds of people rather than the effect of the feature.

The standard toolkit is worth being able to sketch. Difference-in-differences compares the change over time in a treated group with the change in an untreated one, and its credibility rests entirely on parallel trends before the intervention, which you should say you would check. Regression discontinuity exploits a threshold rule so that units just either side are comparable. Instrumental variables need something that moves treatment without affecting the outcome any other way, which is a strong assumption and rarely available. Propensity matching balances observed covariates and, crucially, does nothing about unobserved ones.

The answer that scores best is usually the one that ends with a caveat: what the estimate assumes, and what would have to be true for it to be wrong. Simpson's paradox is a good thing to be able to describe concretely, because it shows in one example how an aggregate can reverse within every subgroup.

Modelling rounds test whether your evaluation would survive contact with reality. scikit-learn's documentation states the core problem plainly: "Learning the parameters of a prediction function and testing it on the same data is a methodological mistake: a model that would just repeat the labels of the samples that it has just seen would have a perfect score but would fail to predict anything useful on yet-unseen data."

The subtler failure is tuning on the test set. As the same documentation puts it, when evaluating hyperparameters "there is still a risk of overfitting on the test set because the parameters can be tweaked until the estimator performs optimally. This way, knowledge about the test set can leak into the model and evaluation metrics no longer report on generalization performance." The structural answer is three splits, or cross-validation for selection with a final untouched holdout for reporting.

Which splitter you use is itself an examinable judgement. Stratified variants preserve class frequencies across folds, which matters when a class is rare. Group-based splitters exist because the independence assumption breaks when samples come in clusters: the requirement is to "ensure that all the samples in the validation fold come from groups that are not represented at all in the paired training fold", so one user's rows cannot appear on both sides.

Time series is the case people get wrong most often. Because of autocorrelation, standard k-fold trains on the future to predict the past. A forward-chaining split, where "successive training sets are supersets of those that come before them", is the honest alternative.

Most loops include a round that is nominally about analysis and is really about judgement: a metric moved, what do you do.

The expected structure is to clarify before diagnosing. What exactly is the metric, over what population, and is the change larger than its normal variation. A surprising number of alarming movements are logging changes, a bot, a release that altered how an event fires, or a definitional change nobody announced. Checking whether the change is real precedes explaining it.

Then decompose. A rate is a ratio, so it moves when the numerator moves, when the denominator moves, or when the mix underneath changes while every segment stays flat. That last case is Simpson's paradox in operational clothing and is worth naming explicitly.

The metric definition question is the other staple: define success for a described feature. Strong answers distinguish leading from lagging indicators, propose a guardrail so the metric cannot be gamed by harming something else, and say how the metric could be moved in a way nobody wants. Google's Rules of Machine Learning makes the general version of the point in its opening advice, that infrastructure and measurement come before sophistication, and that a first version does not have to be clever to be useful.

The distinguishing habit is stating what decision the number will inform. A metric that no decision depends on is a report, not a measurement.