Python for Data Science

Because the heavy numerical work runs in compiled code and Python only drives it.

An operation on a million-row array loops in C, not in the interpreter. Python supplies the readable surface and something else supplies the speed.

What it is really testing. Whether you know the heavy work is not happening in Python.

Applying an operation to a whole array at once rather than looping over elements.

The loop moves into compiled code, which is routinely a hundredfold difference. If you are iterating rows in Python you are almost always doing it wrong.

What it is really testing. Whether you treat row iteration as a mistake rather than a style choice.

Fixed type, contiguous memory, and operations that apply to the whole array.

The contiguity is what lets the CPU process it efficiently; a list is a collection of pointers to objects scattered in memory.

What it is really testing. Whether you mention memory layout rather than just typing.

Rules that let arrays of different shapes combine without copying or explicit loops.

Dimensions are compared from the right and must match or be one. Almost every confusing shape error traces back to misunderstanding this.

What it is really testing. Whether you can state the rule rather than treating it as magic.

Slicing gives a view sharing memory; fancy indexing gives a copy.

Writing into a view modifies the original, which is efficient and occasionally surprising when you expected independence.

What it is really testing. Whether you know slicing shares memory, which causes surprising mutations.

A labelled table where columns can have different types and the index means something.

Joining, grouping, aligning and resampling are all about identity rather than position, and all are painful with bare arrays.

What it is really testing. Whether you can say what labels buy beyond convenience.

Splits rows into groups, applies a function per group, and combines the results.

Split-apply-combine covers a startling proportion of real analysis, because most questions a business asks have that shape.

What it is really testing. Whether you recognise split-apply-combine as the shape of most analysis.

A join that multiplies rows, because the key is not unique on the side you assumed.

Three matching rows on each side become nine, every downstream total is inflated, and nothing raises an error.

What it is really testing. Whether you have been caught by it. Nothing errors, which is the problem.

Check row counts before and after, and validate key uniqueness on the side you believe is unique.

Modern tooling will validate the relationship if you ask, and asking costs one argument.

What it is really testing. Whether you have a habit rather than an intention.

Keys that do not match - whitespace, case differences, or a type mismatch between text and number.

An inner join drops the unmatched rows silently, so you must check how many keys matched rather than whether the join ran.

What it is really testing. Whether you check match rates rather than only whether the code ran.

Because why a value is missing usually matters, and missingness is rarely random.

If high earners decline to report income, dropping those rows biases every conclusion about income downward and nothing later recovers it.

What it is really testing. Whether you ask why a value is missing before deciding what to do.

It preserves the average while destroying the variance and inventing certainty.

An indicator column recording that the value was missing often carries more signal than any imputed number.

What it is really testing. Whether you know it destroys variance and fabricates confidence.

That you may be writing to a copy rather than the original, so the write may be lost.

It arises from chained indexing. Using a single .loc assignment makes the intent unambiguous and the warning disappears.

What it is really testing. Whether you understand chained indexing rather than silencing the warning.

loc selects by label, iloc by integer position.

With a default integer index they look identical, which is exactly why code written against one breaks when the index changes.

What it is really testing. Whether you know they diverge once the index is not the default.

Time zones, ambiguous formats and daylight saving all silently produce wrong aggregates.

A date written 03/04/2026 is two different days depending on convention, and a parser guessing per row can mix both in one column.

What it is really testing. Whether you can name more than one failure mode.

Specify the format explicitly rather than letting it be inferred.

Inference works until one row is ambiguous, and then it produces a dataset that is quietly wrong rather than an error.

What it is really testing. Whether you specify formats rather than relying on inference.

Comparing an unfinished period with completed ones, so the latest value always looks low.

It appears in an enormous number of dashboards without a caveat, and the apparent decline is an artefact of the calendar.

What it is really testing. Whether you have spotted it in a real dashboard.

Downcast numeric columns and convert low-cardinality text to a categorical type.

Memory is usually the binding constraint before speed, and this is a few lines after loading with no change to any logic.

What it is really testing. Whether you know memory usually binds before speed does.

It is row-oriented text with no types, so everything is parsed and nothing can be read selectively.

A columnar format stores types, compresses well, and lets you read only the columns you need.

What it is really testing. Whether you can name what a columnar format gives you.

After vectorising, fixing types and reading only what you need - and usually not before.

A great deal of distributed computing has been deployed on datasets that would have fitted in memory with better types.

What it is really testing. Whether you exhaust the cheap options before reaching for a cluster.

apply runs a Python function per row or element; a vectorised operation runs in compiled code.

apply is a loop wearing a method name. It is sometimes necessary and almost never the fast option.

What it is really testing. Whether you recognise apply as a loop in disguise.

Expressing a pipeline as a sequence of operations without intermediate variables.

It makes the order of transformations explicit, which matters because several of them are not commutative.

What it is really testing. Whether you connect it to order-dependence rather than to style.

Filtering, joining, deduplicating and aggregating all give different answers in different orders.

The average of ratios is not the ratio of averages, which is a common and consequential error when grouping.

What it is really testing. Whether you can name a case where order changes the answer.

Wide has one column per variable; long has one row per observation with a variable column.

Most plotting and statistical tools expect long format, which is why melt and pivot are so frequently used.

What it is really testing. Whether you know which tools expect which shape.

Reshaping with an aggregation - rows, columns and a function applied to each cell.

It is groupby with the result laid out as a grid, which is usually easier to read and harder to compute with.

What it is really testing. Whether you can describe it as groupby plus layout.

Because statistics hide shape - several famously different datasets share identical means and correlations.

Plotting reveals the outlier, the cluster, the bimodality and the day the sensor broke, none of which appear in a summary.

What it is really testing. Whether you plot first as a habit rather than at the end.

Exploratory is fast and ugly and for you; explanatory is polished and for someone else.

Conflating them wastes time in both directions - polishing throwaway charts and publishing ones nobody can read.

What it is really testing. Whether you have different standards for each.

Cells can run in any order, so the saved state may not match a top-to-bottom execution.

A result depending on a cell you later edited is not reproducible by anyone, including you next month.

What it is really testing. Whether you have been bitten by out-of-order execution.

Restart and run everything before trusting a result, and move stable code into imported modules.

Pinning library versions matters too, because numeric results can change between releases.

What it is really testing. Whether you restart and rerun before trusting a number.

So every transformation is code and the path from source to conclusion can be re-run.

Editing the source file by hand makes the analysis unreproducible and the edit undiscoverable.

What it is really testing. Whether you treat the source as read-only on principle.

Information from outside the training data influencing it - including scaling fitted on everything.

Splitting first and fitting every transformation on the training data only is the discipline that prevents most of it.

What it is really testing. Whether you know scaling before splitting is leakage.

Because the data you have is what survived some process, and that process is rarely neutral.

Analysing current customers to learn why customers stay excludes everyone who left, who are the informative group.

What it is really testing. Whether you ask who is missing from the data.

Where it came from, who generated it, what each column means, and what is systematically absent.

An event only exists if somebody instrumented it, so an absence may mean it did not happen or that nobody tracked it.

What it is really testing. Whether you interrogate provenance rather than accepting columns at face value.

Because most business distributions are skewed, so a few large values pull it away from typical.

The median describes a typical value and percentiles describe the tail. The average customer frequently does not exist.

What it is really testing. Whether you know skew is the normal case in business data.

Whenever the tail is what people experience - latency, response times, spend.

The 95th and 99th percentiles describe the experience people complain about; the average describes one nobody has.

What it is really testing. Whether you reach for percentiles where the tail is the experience.

Which variable caused which, or whether a third variable caused both.

The honest options are an experiment, a natural experiment, controlling for named confounders, or declining to claim causation.

What it is really testing. Whether you can name what evidence would distinguish the explanations.

A trend appearing in every subgroup can reverse when the groups are combined.

It is why the first response to any aggregate movement should be to break it down by the dimensions that matter.

What it is really testing. Whether you break totals down before interpreting a movement.

Investigate before removing - decide whether they are errors, rare events or the thing you care about.

Removing them by a fixed rule discards fraud, failures and your most valuable customers along with the typos.

What it is really testing. Whether you investigate before removing rather than applying a rule.

Largely emphasis - analysts answer questions about the past, scientists build things that predict.

Both spend most of their time on data quality and definitions rather than on methods, which surprises people entering the field.

What it is really testing. Whether you can describe the overlap honestly rather than drawing a hard line.

Asking what the question actually is before answering it.

The most expensive analysis is a correct answer to the wrong question - it looks finished, gets presented, and is discovered later.

What it is really testing. Whether you restate the question before answering it.