The Hard Part of Analytics Was Never the Chart

Ask most people what a data analyst does and they will describe the output: dashboards, reports, charts in a slide deck. That is the visible part, and it is perhaps a fifth of the job. The other four-fifths is a set of less photogenic activities — working out what question is actually being asked, finding out whether the data can answer it, and resisting the many available ways to produce a confident number that is wrong. The tools are learnable in weeks. The judgement takes considerably longer, and it is the thing that separates an analyst whose numbers get trusted from one whose numbers get checked. The question behind the question Nobody asks an analyst a well-formed question. They ask something like "why did sales drop last month", and the analyst's first job is to find out what would actually be useful. Sales dropped compared with what — the previous month, the same month last year, the forecast? Which sales: orders, revenue, revenue after refunds? All regions or the one the person asking is responsible for? And underneath: what decision does the answer feed? Someone deciding whether to extend a promotion needs something quite different from someone preparing a board update, even though both asked the same sentence. The habit worth building is to restate the question in your own words and get agreement before touching any data. It feels slow and it routinely saves days, because the most expensive analysis is the correct answer to the wrong question — it looks finished, it gets presented, and the error is only discovered when a decision based on it goes badly. Definitions are where the real errors live Every organisation has terms everybody uses and nobody has defined identically. What is an active user? Someone who logged in this month, or opened the app, or performed a meaningful action? Over what window? Does a user who cancelled yesterday count for today? What is revenue? Booked or recognised? Before or after refunds, discounts, tax, currency conversion? What is a customer? An account, a person, a company? If one company has four accounts, is that one customer or four? What happens when two accounts merge? These are not pedantic. Two teams reporting different numbers for the same metric is one of the most common and most trust-destroying situations in an organisation, and the cause is almost always a definitional difference neither side knew about. Writing the definition down alongside the number — in the dashboard, not in a separate document — prevents an enormous amount of later confusion. Where the data came from decides what it can tell you Data is not a neutral record of the world. It is a record of what some system happened to capture, shaped by how and why it was built. An event only exists if somebody instrumented it, so the absence of an event may mean it did not happen or may mean nobody added the tracking. A field that is optional in a form will be systematically missing for the people least motivated to fill it in. A system that was replaced in March will have different semantics before and after, and the join between the two eras is somebody's best guess. Survivorship bias deserves specific mention because it is so easy to walk into. Analysing the behaviour of current customers to understand what makes customers stay excludes everyone who left — which is precisely the group that would have told you. The data you have is the data that survived whatever process produced it, and that process is rarely neutral. The question to ask of every dataset: what is systematically absent from this, and who would have been in it? Averages hide almost everything The mean is the most reported and most misleading statistic in common use, because it describes a distribution accurately only when the distribution is symmetric and has no long tail — and almost nothing in business is. Revenue per customer, session duration, response time, items per order: all are skewed, often severely. A handful of very large values pull the mean somewhere that describes nobody. The average customer, in a typical business, does not exist. The practical replacements are undramatic. Report the median for a typical value. Report percentiles when the tail is what matters — in performance work, the 95th and 99th percentiles describe the experience people complain about, while the average describes an experience nobody has. And plot the distribution before summarising it, because a bimodal distribution has a mean sitting in the empty space between its two humps. Correlation, causation, and the third variable Everybody knows correlation does not imply causation, and everybody slips anyway, because the slip usually arrives disguised as a plausible story. Users who engage with a feature retain better. Customers who contact support spend more. Students who attend review sessions score higher. Each of these has an obvious causal reading and each has at least one equally plausible alternative: people who were already going to retain are the ones who explore features; high-spending customers have more to ask about; conscientious students both attend sessions and study. The honest options are limited and worth knowing. Run an experiment, which is the only reliable answer. Find a natural experiment where something changed for reasons unrelated to the outcome. Control for the confounders you can name, while being clear that you cannot control for the ones you cannot. Or state the association and explicitly decline to claim the cause, which is a perfectly respectable finding. What is not acceptable is presenting an association with causal language and letting the audience draw the conclusion you implied but did not defend. Segments move independently, and the total can lie A total can stay flat while everything underneath it changes dramatically. Two segments moving in opposite directions cancel out, and the summary number reports stability that does not exist. Worse, a metric can improve in every segment while the overall figure gets worse, purely because the mix shifted toward a segment with a lower rate. This is a genuine statistical phenomenon rather than an arithmetic error, and it is the reason the first response to any aggregate movement should be to break it down. The practical habit: never report a change in a total without looking at it by the two or three dimensions that matter most for that metric — region, segment, channel, new versus returning. The breakdown is frequently the entire story, and the total was hiding it. Comparisons need a baseline that is fair Most analytical claims are comparisons, and most bad analytical claims are comparisons against the wrong baseline. Comparing to the previous period ignores seasonality — December is not a fair comparison for January in most businesses. Comparing to the same period last year handles seasonality but ignores growth. Comparing against a forecast only works if the forecast was any good. Then there is the partial period problem, which is endemic: this week against last week, when this week is on Wednesday. The number is lower because the week is not finished, and a surprising number of dashboards present exactly this without flagging it. And when a comparison involves a change that was rolled out, the before-and-after is contaminated by everything else that happened at the same time. A control group is the only clean answer, which is why experiments matter. Experiments, briefly, and the mistakes that are common A randomised experiment is the cleanest tool analytics has, and the ways it goes wrong are predictable. Stopping when the result looks good inflates false positives substantially, because a test checked repeatedly will eventually cross a threshold by chance. The sample size should be decided before the test starts, from the effect size worth detecting. Measuring many metrics and reporting the one that moved is the same error wearing a different hat. With twenty metrics, one will reach conventional significance by chance alone. A statistically significant result is not necessarily a meaningful one. With enough traffic, a 0.1% improvement becomes significant — and the question of whether it is worth the engineering is separate, and not a statistical question at all. Dashboards decay unless someone owns them A dashboard is not a deliverable; it is a commitment. Metrics drift as the underlying systems change, definitions shift, a pipeline silently fails and the chart shows a flat line that looks like a business result. The dashboards that stay trustworthy have a few things in common. They show when the data was last refreshed, so a stale chart is visibly stale. They carry the definition next to the number. They are small — a dashboard with forty charts is one nobody reads and nobody validates. And somebody is responsible for them, which is the part that is organisational rather than technical. A dashboard nobody owns becomes wrong, then becomes known to be wrong, then gets quietly ignored while continuing to exist. That is worse than never having built it, because it occupied the space where a trusted one could have been. Presenting a finding The analysis is not finished when the number is correct. It is finished when a decision-maker understands it well enough to act. Lead with the finding rather than the method. The audience does not need the journey, and starting with methodology loses them before the point arrives. One chart per message. The title should state the conclusion — "Churn is concentrated in accounts under three months old" — rather than naming the axes. State the uncertainty, honestly and without hedging everything into mush. "Between 8% and 12%, most likely around 10%" is more useful and more credible than a single number with implied precision nobody believes. And be willing to deliver the inconclusive result. "The data cannot answer this, and here is what we would need to collect" is a genuine finding, and an analyst who says it when it is true will be believed when they say something else. Sampling, and the number behind the number A great deal of analytics is quietly built on samples — a subset of logs, a survey, an experiment arm, a week chosen to stand in for a quarter — and the sample size determines how much of the variation you see is real. The specific intuition worth having is that precision improves with the square root of the sample size. Four times the data halves the uncertainty, not quarters it. This is why a small increase in a sample rarely settles an argument, and why a comparison between two small segments is usually noise wearing a pattern. The other half is that a biased sample does not improve with size. A survey answered only by enthusiasts gives a precise measurement of enthusiasts, and collecting ten times more of them makes it more precisely wrong. Size fixes variance; it does nothing for bias, and distinguishing which of the two is limiting a conclusion is a core analytical judgement. Cohorts, and the comparison that controls for time Many metrics are contaminated by the fact that the population changes. Average revenue per customer falls when a successful marketing campaign brings in many new customers who have not bought much yet — which looks like a decline and is the opposite. A cohort analysis fixes this by grouping people by when they joined and following each group over its own lifetime. The question stops being "how are customers doing this month" and becomes "how does a customer in month three compare with earlier customers in their month three", which is a comparison between like and like. This is the single most useful technique for anything involving retention, lifetime value, or behaviour that changes with tenure, and it routinely reverses conclusions drawn from the aggregate. It also makes improvement visible early: a newer cohort doing better at month one is evidence a change worked, long before it shows up in a total dominated by older cohorts. Writing the number down with its caveats attached A figure travels further than the analysis behind it. The number appears in a message, gets copied into a document, reaches a meeting, and by then every qualification that made it honest has been stripped away. The only reliable defence is to attach the caveats to the number itself rather than to the surrounding discussion. Not "revenue grew 12%" in the chart with the exclusions explained in a footnote nobody carries forward, but "revenue grew 12% (excluding refunds, UK only)" as the label. It is less elegant and it survives being copied, which is the property that actually matters. What to practise

1. Take a metric your organisation reports and write down its exact definition. Then ask two other people for theirs.
2. Break a flat total into segments and find the offsetting movements underneath it.
3. Replace an average with a median and a distribution plot, and see what you had been hiding.
4. Write down the alternative explanation for a correlation you believe is causal, and work out what evidence would distinguish them.
5. Audit a dashboard you rely on — check whether each number still means what its label says. Analytics rewards scepticism applied first to your own results. The analyst people trust is not the one who produces the most charts. It is the one who has already asked the question the audience was about to ask, and has an honest answer.