An analytics interview is mostly about judgement. The tools can be learned in weeks; what takes longer is knowing which comparison is fair, which definition is being assumed, and which confident number is wrong.
Expect questions that look like statistics and are really about scepticism - what is missing from this data, what else could explain this, and what would have to be true for the answer to be wrong.
Answer each one out loud before you open it. An answer you have read is not an answer you can give under pressure, which is the only kind that counts.
Q1 What is the first thing you do when given an analytical question?
Restate it, establish the comparison and find out what decision it feeds.
Nobody asks a well-formed question. The most expensive analysis is a correct answer to the wrong one, because it looks finished.
What it is really testing. Whether you restate before analysing, or start pulling data immediately.
Q2 Why do two teams report different numbers for the same metric?
Because they are using different definitions and neither knows it.
Active user, revenue and customer all have several defensible definitions. Writing the definition next to the number prevents most of this.
What it is really testing. Whether you have seen it happen. It destroys trust faster than a wrong number.
Q3 What should a metric definition specify?
The population, the time window, the inclusions and exclusions, and the event that counts.
Without those four, a metric is a label rather than a measurement, and it will quietly change meaning as systems change.
What it is really testing. Whether you can list the components rather than gesturing at clarity.
Q4 Why is the mean usually the wrong summary?
Because business distributions are skewed, so a few large values drag it away from typical.
Revenue per customer, session length and response time are all skewed. The average customer frequently does not exist.
What it is really testing. Whether you know skew is the normal case rather than the exception.
Q5 When should you use a median or percentiles?
A median for a typical value, percentiles when the tail is what people experience.
In performance work the 95th and 99th percentiles describe the experience people complain about; the average describes nobody's.
What it is really testing. Whether you reach for percentiles where the tail is the experience.
Q6 What does a flat total hide?
Segments moving in opposite directions and cancelling out.
The first response to any aggregate movement should be to break it down, because the breakdown is frequently the entire story.
What it is really testing. Whether you break totals down by reflex rather than on request.
Q7 What is Simpson's paradox and why does it matter?
A rate can improve in every segment while the overall rate worsens, because the mix shifted.
It is a genuine statistical phenomenon rather than an arithmetic error, and it reverses conclusions drawn from totals alone.
What it is really testing. Whether you know it is a real phenomenon rather than a calculation error.
Q8 What makes a comparison baseline fair?
It controls for seasonality and growth and compares completed periods.
Previous period ignores seasonality, same period last year ignores growth, and an unfinished week always looks low.
What it is really testing. Whether you can name what each baseline ignores.
Q9 What is a cohort analysis and when is it the right tool?
Grouping by join date and following each group over its own lifetime.
It removes the contamination from a changing population, which is why a successful campaign can make average revenue per user fall.
What it is really testing. Whether you can explain why the aggregate misleads when the population changes.
Q10 How does sample size affect precision?
Precision improves with the square root of the size, so four times the data halves the uncertainty.
That is why a small increase in a sample rarely settles an argument, and why comparing two small segments is usually noise.
What it is really testing. Whether you know the square-root relationship rather than assuming linearity.
Q11 Does a larger sample fix bias?
No. Size reduces variance and does nothing for bias.
A survey answered only by enthusiasts gives a precise measurement of enthusiasts, and more of them makes it more precisely wrong.
What it is really testing. Whether you separate bias from variance clearly.
Q12 What is survivorship bias in an analytics context?
Drawing conclusions from the population that remains, excluding everyone who left.
Studying current customers to learn why customers stay excludes churned users, who are the group that would actually tell you.
What it is really testing. Whether you ask who is missing from the data.
Q13 What does correlation not establish?
Direction, or whether a third variable caused both.
Users who engage retain better has at least two readings, and distinguishing them requires an experiment or a natural one.
What it is really testing. Whether you can name an alternative explanation on the spot.
Q14 What are the honest options when you cannot run an experiment?
A natural experiment, controlling for named confounders, or stating the association without claiming cause.
Declining to claim causation is a respectable finding. Implying it with causal language while not defending it is not.
What it is really testing. Whether you treat declining to claim causation as a legitimate answer.
Q15 What is the most common mistake in running an A/B test?
Stopping when the result looks good.
Checking repeatedly inflates false positives, because a test examined often enough will cross a threshold by chance. Fix the sample size in advance.
What it is really testing. Whether you know peeking inflates false positives.
Q16 What is the multiple comparisons problem?
Measuring many metrics and reporting the one that moved.
With twenty metrics, one reaching conventional significance by chance is expected rather than surprising.
What it is really testing. Whether you know measuring many things changes the interpretation.
Q17 Is a statistically significant result always meaningful?
No. With enough traffic a trivial difference becomes significant.
Whether a 0.1% improvement is worth the engineering is a business question, not a statistical one.
What it is really testing. Whether you separate statistical and practical significance.
Q18 What is statistical power and why should you care before the test?
The probability of detecting an effect that is really there.
An underpowered test that shows nothing has told you nothing, which is a worse outcome than not running it.
What it is really testing. Whether you think about power before running rather than after.
Q19 What makes a dashboard trustworthy?
A visible refresh time, definitions next to the numbers, few charts, and a named owner.
A pipeline failure shows as a flat line that looks like a business result, so staleness must be visible.
What it is really testing. Whether you name ownership as well as design.
Q20 Why do dashboards decay?
Definitions drift, systems change, and nobody is responsible for noticing.
A dashboard known to be wrong but still present is worse than none, because it occupies the space a trusted one could have.
What it is really testing. Whether you treat a dashboard as a commitment rather than a deliverable.
Q21 How should you present a finding?
Lead with the conclusion, one message per chart, and a title that states the finding.
Starting with methodology loses the audience before the point arrives. The journey is rarely what they need.
What it is really testing. Whether you lead with the conclusion rather than the method.
Q22 How should uncertainty be communicated?
As a range with a most likely value, stated plainly.
Between 8 and 12 per cent is more useful and more credible than a single number with implied precision nobody believes.
What it is really testing. Whether you can state uncertainty without hedging it into uselessness.
Q23 Is an inconclusive result a failure?
No - it is a finding, provided you say what would be needed to answer it.
An analyst who reports inconclusiveness when it is true will be believed when they report something else.
What it is really testing. Whether you are willing to deliver a negative result.
Q24 What is the difference between a metric and a KPI?
A metric measures something; a KPI is a metric a decision is tied to.
Treating every metric as a KPI produces dashboards nobody acts on and dilutes attention from the few that matter.
What it is really testing. Whether you tie a KPI to a decision rather than to importance.
Q25 What is a vanity metric?
One that rises reliably and informs no decision - cumulative totals, raw page views.
Cumulative counts can only go up, so they always look like progress regardless of what is happening.
What it is really testing. Whether you can name one and explain why it is useless.
Q26 What is the difference between a leading and a lagging indicator?
A leading indicator moves before the outcome; a lagging one confirms it afterwards.
Revenue is lagging; trial activation is leading. Managing only by lagging indicators means learning too late to act.
What it is really testing. Whether you can give an example pair from a real product.
Q27 How do you investigate a sudden metric change?
Check for an instrumentation or pipeline change first, then segment, then look for an external cause.
A surprising share of dramatic movements are tracking changes rather than behaviour changes.
What it is really testing. Whether you check instrumentation before behaviour.
Q28 What is instrumentation bias?
An event exists only if somebody added the tracking, so absence may mean nothing happened or nothing was recorded.
Comparing periods across a tracking change is comparing two different measuring instruments.
What it is really testing. Whether you know a tracking change makes periods incomparable.
Q29 What is a funnel analysis and what does it miss?
Measuring progression through steps - and it misses people who re-enter, skip or take another path.
A linear funnel imposed on non-linear behaviour produces drop-off that is an artefact of the model.
What it is really testing. Whether you know where the funnel model breaks down.
Q30 What is retention and why is the curve shape what matters?
The proportion still active after a period - and a flattening curve means a sustainable product.
A curve that keeps declining to zero means no cohort ever sticks, which no amount of acquisition fixes.
What it is really testing. Whether you read the shape rather than a single retention number.
Q31 What is churn and why is it harder to define than it looks?
Customers leaving - hard because for non-subscription products there is no moment of leaving.
You must choose an inactivity window, and the choice changes the number substantially without changing reality.
What it is really testing. Whether you know the definition is a choice that moves the number.
Q32 What does segmentation buy you?
It separates populations behaving differently, which an aggregate blends into something describing neither.
New and returning users, regions and channels usually behave differently enough that a combined figure is meaningless.
What it is really testing. Whether you segment by default rather than when asked.
Q33 What is a weighted average and when do you need one?
An average respecting group sizes - needed whenever you average rates across unequal groups.
Averaging conversion rates across regions of different sizes gives a number that matches no actual population.
What it is really testing. Whether you know averaging rates across unequal groups is wrong.
Q34 How should you treat outliers in analysis?
Investigate before removing - decide whether they are errors, rare events or the point.
A fixed removal rule discards fraud, failures and your largest customers along with the typos.
What it is really testing. Whether you investigate before applying a removal rule.
Q35 What is the difference between a count of events and a count of users?
One counts actions, the other people, and a heavy user can dominate an event count.
Reporting events while describing users is a common and misleading slip that inflates apparent engagement.
What it is really testing. Whether you are precise about the unit being counted.
Q36 How do you sanity check a number before presenting it?
Compare it with a known total, a previous period, and an independent source.
If it does not reconcile with something already trusted, the discrepancy is the finding until explained.
What it is really testing. Whether you reconcile against something already trusted.
Q37 Why should caveats be attached to the number itself?
Because the number travels further than the analysis behind it.
Revenue grew 12 per cent, excluding refunds, UK only, survives being copied into a message. A footnote does not.
What it is really testing. Whether you have watched a number lose its caveats in transit.
Q38 What is data lineage and why does it matter?
Knowing where a figure came from and what transformed it on the way.
Without it, nobody can answer whether a number changed because the business changed or because a pipeline did.
What it is really testing. Whether you can explain why lineage answers the business-or-pipeline question.
Q39 What is the analyst's most valuable habit?
Scepticism applied first to their own results.
A result that seems too good usually is, and checking before presenting is cheaper than being corrected afterwards.
What it is really testing. Whether you apply scepticism inward rather than only to others' work.
Q40 What makes an analyst trusted?
Having already asked the question the audience is about to ask, and answering it honestly.
Trust is built by the inconclusive results and the caveats, not by the confident charts.
What it is really testing. Whether you understand trust is built on the uncomfortable answers.