The scientific Python stack is four libraries and a decision about where to run them. Nearly everything else you will meet - scikit-learn, statsmodels, PyTorch, Polars - sits on top of the same two ideas: a typed array in contiguous memory, and a labelled table built on top of it.
The four you will actually use
NumPy is the foundation. One ndarray type, fixed dtype, contiguous memory, and operations that run in compiled C over the whole array at once. Every other library in this list either uses it or imitates it.
pandas adds labels. A Series is a NumPy array with an index; a DataFrame is a dictionary of Series sharing one. That gives you named columns, alignment on the index, grouping, joining and a great deal of convenience, at the cost of some memory and some speed.
Matplotlib draws. It is old, its API has two layers, and it is the thing almost every other plotting library wraps. Learn enough of it to adjust a figure someone else's library produced.
scikit-learn is the modelling layer: one interface - fit, predict, transform - over a hundred algorithms, plus the pipeline and cross-validation machinery that stops you fooling yourself.
import numpy as np
import pandas as pd
# NumPy: one dtype, one block of memory, no labels.
units = np.array([120, 95, 143, 60])
price = np.array([12.5, 12.5, 11.0, 14.0])
print(units.dtype, units.shape, units.nbytes)
print(units * price)
# pandas: the same numbers, with labels and mixed types.
sales = pd.DataFrame({
'region': ['North', 'South', 'North', 'East'],
'units': units,
'price': price,
})
sales['revenue'] = sales['units'] * sales['price']
print(sales)
print(sales.dtypes)
Open this example in the editor and change it. Nothing you type is saved.
Notice the division. NumPy knows units * price is four multiplications of float64. pandas knows that region is text and units is a number, and that the fifth row would be the fifth of both.
The rule for choosing: NumPy when the data is a rectangle of one type, pandas when the columns mean different things and you want to refer to them by name.
Environments, before anything else
The single most common way a data project becomes unreproducible is that nobody wrote down which versions it was built with.
# Create and activate - these are shell commands, shown here as text.
#
# python -m venv .venv
# source .venv/bin/activate # on Windows: .venv/Scripts/activate
# pip install numpy pandas matplotlib scikit-learn
# pip freeze > requirements.txt
#
# Then anyone can reproduce it:
#
# python -m venv .venv && source .venv/bin/activate
# pip install -r requirements.txt
import sys
import platform
print(platform.python_version())
print(sys.executable.endswith('python') or sys.executable.endswith('python3'))
Open this example in the editor and change it. Nothing you type is saved.
pip freeze pins everything, including transitive dependencies, which is what you want for an analysis you may need to re-run in a year. uv and conda solve the same problem with better ergonomics; the principle does not change.
Record the versions in the output, not only in a file:
import numpy as np
import pandas as pd
import sys
versions = {
'python': sys.version.split()[0],
'numpy': np.__version__,
'pandas': pd.__version__,
}
for name, version in versions.items():
print(f'{name:8} {version}')
Open this example in the editor and change it. Nothing you type is saved.
Three lines at the top of every notebook, and a result that disagrees with yours six months later has an immediately visible first suspect.
Notebooks, honestly
A notebook is excellent for exploring and bad for anything that must run the same way twice. The reason is one feature: cells can be run in any order, and the output you are looking at may not come from the code you are looking at.
# In a notebook, this sequence is possible and leaves no trace:
#
# In [1]: df = load() # 10,000 rows
# In [2]: df = df[df.region == 'North'] # now 2,000 rows
# In [1]: df = load() # re-run cell 1: back to 10,000
# In [3]: print(len(df)) # 10,000 - but cell 2 is still
# # on screen, showing 2,000
#
# The numbers in the brackets are the only clue, and nobody reads them.
counts = {'after load': 10000, 'after filter': 2000, 'what you see': 2000}
print(counts)
Open this example in the editor and change it. Nothing you type is saved.
Three habits make notebooks safe:
- Restart and run all before you believe any result, and always before you share one.
- Keep the cells in execution order, so restart-and-run-all is the same as what you did.
- Move anything reusable into a module and import it, so the notebook is a thin script over tested functions.
The division that works: notebooks for looking at things, .py files for anything that runs more than once.
A worked example
import sys
import numpy as np
import pandas as pd
print(f'python {sys.version.split()[0]} numpy {np.__version__} pandas {pd.__version__}')
print()
# A deterministic generator, so this file produces the same numbers
# every time anyone runs it. Without the seed it would not.
rng = np.random.default_rng(seed=42)
n = 500
regions = np.array(['North', 'South', 'East', 'West'])
sales = pd.DataFrame({
'region': rng.choice(regions, size=n, p=[0.4, 0.3, 0.2, 0.1]),
'units': rng.integers(10, 200, size=n),
'price': np.round(rng.normal(12.5, 2.0, size=n), 2),
})
sales['revenue'] = (sales['units'] * sales['price']).round(2)
# The three lines to run on any new table, before anything else.
print(sales.shape)
print(sales.dtypes.to_dict())
print(sales.head(3))
print()
# NumPy underneath: the column IS an array, and you can drop down to it.
revenue = sales['revenue'].to_numpy()
print(type(revenue).__name__, revenue.dtype, revenue.shape)
print(f'total {revenue.sum():,.2f} mean {revenue.mean():,.2f} '
f'median {np.median(revenue):,.2f}')
print()
# pandas on top: the same work, by label.
summary = (
sales.groupby('region', as_index=False)
.agg(orders=('units', 'size'),
units=('units', 'sum'),
revenue=('revenue', 'sum'))
.sort_values('revenue', ascending=False)
.reset_index(drop=True)
)
print(summary)
print()
# The same aggregate in NumPy alone, to show what pandas is saving you.
mask = sales['region'].to_numpy() == 'North'
print(f'north revenue, numpy: {revenue[mask].sum():,.2f}')
print(f'north revenue, pandas: {summary.loc[summary.region == "North", "revenue"].iloc[0]:,.2f}')
# And the reproducibility claim, tested rather than asserted.
again = np.random.default_rng(seed=42).integers(10, 200, size=5)
first = np.random.default_rng(seed=42).integers(10, 200, size=5)
print('same seed, same numbers:', bool(np.array_equal(again, first)))
Open this example in the editor and change it. Nothing you type is saved.
Four habits are visible there and they are the ones to copy. The versions are printed, so the output is self-describing. The randomness is seeded through a Generator rather than the legacy np.random.seed, so it is local rather than global. The shape and dtypes are printed before any analysis. And the final two lines check a claim the code makes about itself.
The versions this course uses
import numpy as np
import pandas as pd
# The APIs in this course are those of:
# numpy 2.x
# pandas 2.2 or later
#
# Two things changed recently enough to catch you out in older code:
# - np.random.default_rng() replaced np.random.seed()
# - pandas copy-on-write became the default, which removes most
# SettingWithCopyWarning cases - and changes the behaviour of
# code that relied on a view.
print('numpy', np.__version__.split('.')[0], 'x')
print('pandas', '.'.join(pd.__version__.split('.')[:2]))
Open this example in the editor and change it. Nothing you type is saved.
When you read a Stack Overflow answer from 2016, check which of those two it predates. Most "why does pandas warn about this" questions were answered before copy-on-write existed.
When it goes wrong
| Symptom | Cause |
|---|---|
| The same notebook gives different numbers | No seed, or cells run out of order |
ImportError for a library you installed |
Installed into a different environment |
| A colleague cannot reproduce your result | No pinned requirements |
A column of numbers has dtype object |
pandas read them as strings; see the cleaning lesson |
| A loop over rows takes minutes | Use a column operation, not iterrows |
| Memory exhausted on a file that fits on disk | Default dtypes; see the performance lesson |
| An answer from the internet does not work | It predates a major-version change |
np.random.seed had no effect |
A Generator was used; seed that instead |
A check you can run
Open any notebook you have, choose "Restart kernel and run all cells", and compare the output to what was on screen before.
If anything differs - a number, a plot, an error - you have found hidden state, and every conclusion below that cell is suspect. It takes thirty seconds and it is the single most valuable habit in this course.