In Data Science, the Language Is the Easy Part. The Data Is Not

There is a predictable arc to learning data science with Python. You install the libraries, follow a tutorial, load a clean dataset, call a few methods, and produce a chart that looks professional. It takes an afternoon and it feels like you have learned the subject.

Then you are given real data, and almost nothing transfers. The file has columns nobody can explain. Dates are stored four different ways. A quarter of one column is missing and the missingness is not random. Two systems disagree about what a customer is. The tutorial never mentioned any of this because the tutorial's dataset had been cleaned by somebody else, and that cleaning was most of the work.

This is the thing worth knowing before you start: in practical data work, the Python is rarely the hard part. The hard part is the data, and the skills that matter are the ones for interrogating it.

Why Python won, and what that means for you

Python was not an obvious choice for numerical computing. It is slow, it is dynamically typed, and its concurrency story is weak. It won anyway, and the reason is worth understanding because it shapes how the ecosystem behaves.

The foundation is that Python is good at being a wrapper. The heavy numerical work happens in compiled code — C, C++, Fortran — and Python drives it. When you call a function on a million-row array, the loop runs in optimised machine code, not in the interpreter. Python supplies the readable surface; something else supplies the speed.

The practical consequence is a rule that governs performance in almost all data work: if you are looping over rows in Python, you are doing it wrong. The same operation expressed as a whole-array operation can be a hundred times faster, not because the algorithm changed but because the loop moved into compiled code. Learning to think in whole arrays rather than in iterations is the single biggest performance skill in the subject.

NumPy is the layer everything else sits on

The array is the foundational object. It differs from a Python list in three ways that matter: every element has the same type, the data sits in one contiguous block of memory, and operations apply to the whole array at once.

That last property — vectorisation — is what makes the ecosystem viable. Adding two million-element arrays is one function call, and the loop happens in C.

Broadcasting is the rule that lets arrays of different shapes combine without writing loops or copying data. It feels like magic until you understand it and then it feels inevitable, and it is worth learning properly because almost every confusing shape error in the ecosystem traces back to a misunderstanding of it.

Two things catch people. Slicing an array gives a view, not a copy, so writing into the slice modifies the original — which is efficient and occasionally surprising. And arrays have a fixed type, so putting a very large integer or a missing value into an integer array does something you did not intend rather than failing.

Pandas, and the thing it is actually for

Pandas puts labels on data. A DataFrame is a table where rows and columns have names, columns can have different types, and the index means something.

The label is the feature. Joining two tables on a key, grouping by a category, resampling a time series, aligning two series that cover different dates — all of these are about identity rather than position, and all of them are miserable to do with bare arrays.

The method worth learning first is groupby, because the split-apply-combine pattern it implements covers a startling proportion of real analysis: split the data into groups, compute something per group, assemble the results. Most questions a business asks have that shape.

The second is merging. Joining tables is where silent errors concentrate, and the specific failure to watch for is a join that multiplies rows. If the key is not unique on the side you think it is, a join produces more rows than you started with, every downstream number is inflated, and nothing errors. Checking the row count before and after every join is a two-second habit that catches a bug class that can otherwise survive to a published report.

Missing data is information, not an inconvenience

The instinct is to remove missing values and move on. That instinct is frequently wrong, because why the value is missing usually matters.

If income is missing at random, dropping those rows loses precision but not correctness. If income is missing because high earners decline to answer, dropping them biases every conclusion about income downward, and no amount of statistical sophistication later will recover what was thrown away.

Before handling missing values, the question to ask is what their absence means. Sometimes a missing value is genuinely zero — no purchases rather than unknown purchases. Sometimes it marks a different population — a field that only exists for customers since a certain date. Sometimes it is a data pipeline failure on particular days, which you can see by plotting missingness over time.

Filling with the mean is the default advice and it is usually the worst option, because it preserves the average while destroying the variance and inventing certainty where there was none. Adding an explicit indicator column recording that a value was missing often carries more signal than the imputed value itself.

Dates, and why they are their own discipline

Dates and times cause more real bugs in data work than any other type.

Time zones are the obvious trap: data recorded in local time across several regions, aggregated without conversion, produces daily totals that are wrong at the boundaries and seasonal patterns that are artefacts. Daylight saving means some days have 23 hours and some have 25, which breaks anything assuming a day is a fixed interval.

Less obvious is ambiguity in parsing. A date written as 03/04/2026 is two different days depending on where it was written, and a parser guessing per-row can silently produce a dataset where some dates are day-first and others month-first. Specifying the format explicitly rather than letting it be inferred is one of the highest-value defensive habits available.

And the subtlest: comparing a period that has not finished against periods that have. This month's total is lower than last month's because this month is not over, and that comparison appears in an enormous number of dashboards without a caveat.

Visualisation is for finding things, not just for showing them

There are two distinct activities that both get called plotting.

Exploratory plotting is for you. It should be fast and ugly. Its purpose is to let you see the distribution, spot the impossible values, notice the cluster, and find the day the sensor broke. Summary statistics hide all of that — several famously different datasets share identical means, variances and correlations, and look nothing alike when plotted. Plotting the data before computing anything about it is not a nicety.

Explanatory plotting is for someone else, and has different rules: one message per chart, labelled axes, a title that states the finding rather than the variables, and no decoration that does not carry information.

Conflating the two wastes time in both directions — polishing charts you will throw away, and publishing charts nobody but you can read.

The reproducibility problem, and the notebook

Notebooks are excellent for exploration. The ability to run a cell, look, adjust and run again matches how analysis actually proceeds.

They also have a structural problem: cells can be run in any order, and the notebook's saved state reflects the order you happened to use rather than the order shown. A notebook that runs top to bottom on your machine may not run at all on someone else's, and a result that depends on a variable defined in a cell you later edited is not reproducible by anyone, including you next month.

The practices that address this are undramatic. Restart the kernel and run everything before trusting a result. Move stable code out of the notebook into a module and import it. Keep the raw data immutable and write every transformation as code, so the path from source to conclusion can be re-run. Pin library versions, because the numbers can change between releases.

The test of an analysis is not whether the conclusion is interesting. It is whether somebody else can reproduce it from the raw data, and if the answer is no, the conclusion is an opinion.

Performance, and when to care

The ordering is worth knowing because people optimise in the wrong order constantly.

First, use vectorised operations instead of loops. This is usually a hundredfold difference and it is the only step most work needs.

Second, use appropriate types. Converting a repeated string column to a categorical type can cut memory by an order of magnitude, and memory is usually the constraint that bites before speed does.

Third, only read what you need. Reading selected columns, filtering at load time, and using a columnar file format rather than CSV all reduce the problem before it reaches memory.

Only after those three does it make sense to reach for parallelism or an out-of-core framework. A great deal of distributed computing has been deployed on datasets that would have fitted comfortably in memory with better types.

The skill that is not technical

The most consequential mistakes in data work are not coding errors. They are questions answered confidently that were never properly asked.

A column named revenue that turns out to include refunds. A customer identifier that was reused after an account migration. A filter inherited from an earlier analysis that silently excludes a region. A correlation presented as a cause.

None of these are caught by better Python. They are caught by asking where the data came from, who generated it, what each column means to the person who defined it, and what would have to be true for the answer to be wrong. Analysts who do this consistently are more valuable than analysts who know more methods, and the gap widens with seniority.

Joins, keys, and the arithmetic that should always be checked

Joining tables is where quiet errors concentrate, and the reason is that the

most common failures produce more data rather than an error.

A one-to-many join is usually intended. A many-to-many join usually is not: if

the key is duplicated on both sides, the result contains every combination, so

three matching rows on each side become nine. Every total computed afterwards

is inflated, every average is distorted, and nothing anywhere raises a problem.

The habit worth making automatic is to record the row count before a join and

compare after, and to check key uniqueness on whichever side you believe is

unique. Modern tooling will validate this for you if you ask, and asking costs

one argument.

The second common failure is the opposite: rows silently disappearing because

the keys do not match. Trailing whitespace, differing case, one side storing

the identifier as text and the other as a number, a date with a time component

on one side only. An inner join drops the unmatched rows without comment. The

defence is to look at how many keys matched, not just at whether the join ran.

The pipeline, and why order of operations matters

An analysis is a sequence of transformations, and several of them are not

commutative in ways that are easy to miss.

Removing outliers before or after aggregating gives different answers.

Filtering before or after a join gives different answers when the join is not

inner. Deduplicating before or after a merge gives different answers when the

merge introduces duplicates. Computing a rate before or after grouping gives

genuinely different numbers — the average of ratios is not the ratio of

averages, and reporting the wrong one is a common and consequential error.

Writing the pipeline as an explicit sequence of named steps rather than a long

chain makes these decisions visible, which is the point. Each step should be

something you could describe in a sentence and check in isolation.

Types and memory, the cheapest optimisation available

The default types a loader chooses are rarely the ones you want. A column of

repeated category names stored as free text costs many times what the same

column costs as a categorical type, and an integer column stored at full width

when its values fit in a byte is wasting most of what it occupies.

Downcasting numeric columns and converting low-cardinality text to categories

routinely reduces a dataset's footprint by a large factor, and memory is far

more often the binding constraint than processing speed. It is also the

cheapest change available: a few lines after loading, with no change to any

logic downstream.

What to practise

1. Take a messy public dataset — real, not a tutorial one — and spend an hour only describing it: shapes, types, missingness, ranges, duplicates. Do not model anything.
2. Rewrite a row-wise loop as a vectorised operation and time both.
3. Deliberately produce a bad join that multiplies rows, then write the row-count check that catches it.
4. Plot a dataset before computing its summary statistics, and find something the statistics hid.
5. Restart your kernel and re-run a notebook you believe is finished. The first few times, it will not work.

The best data people are not the ones with the most methods. They are the ones who are hardest to fool, including by themselves — and most of that skill is a habit of checking the data rather than trusting it.