A Series is a labelled array. A DataFrame is a dictionary of Series that share an index. Everything in pandas follows from those two sentences, including the parts that surprise people.
The two structures
import pandas as pd
# A Series: values plus an index.
units = pd.Series([120, 95, 143, 60], name='units')
print(units)
print(units.index, units.dtype, units.shape)
# An index you chose, rather than 0..n.
by_region = pd.Series([120, 95, 143], index=['North', 'South', 'East'], name='units')
print(by_region['South'])
print(by_region[['North', 'East']])
Open this example in the editor and change it. Nothing you type is saved.
import pandas as pd
sales = pd.DataFrame({
'region': ['North', 'South', 'North', 'East'],
'units': [120, 95, 143, 60],
'price': [12.5, 12.5, 11.0, 14.0],
})
print(sales)
print(sales.dtypes)
print(sales.shape, len(sales), sales.columns.tolist())
Open this example in the editor and change it. Nothing you type is saved.
Each column is one dtype. The index - here 0 to 3, because nobody chose one - is how rows are addressed, and it is worth setting deliberately when you have a natural key.
The index is not a column. It is the row labels, it participates in alignment, and a great deal of pandas behaviour that looks magical is the index quietly doing its job:
import pandas as pd
a = pd.Series([1, 2, 3], index=['x', 'y', 'z'])
b = pd.Series([10, 20, 30], index=['z', 'y', 'x'])
print(a + b) # aligned by LABEL, not by position
print((a + b)['x']) # 31 - a['x'] + b['x']
Open this example in the editor and change it. Nothing you type is saved.
That alignment is the single most useful property of pandas and the single most confusing when you did not expect it. Two Series with different indexes add to a union of both indexes, with NaN where one is missing.
Selecting: the part to get right early
import pandas as pd
sales = pd.DataFrame({
'region': ['North', 'South', 'North', 'East'],
'units': [120, 95, 143, 60],
'price': [12.5, 12.5, 11.0, 14.0],
})
print(sales['units'].tolist()) # one column, as a Series
print(sales[['region', 'units']].shape) # two columns, as a DataFrame
print(sales[sales['units'] > 100]) # rows where the mask is True
Open this example in the editor and change it. Nothing you type is saved.
Then the pair that confuses everybody:
import pandas as pd
sales = pd.DataFrame(
{'units': [120, 95, 143, 60]},
index=['a', 'b', 'c', 'd'],
)
print(sales.loc['a', 'units']) # by LABEL
print(sales.iloc[0]['units']) # by POSITION
print(sales.loc['b':'c']) # label slicing INCLUDES the end
print(sales.iloc[1:3]) # position slicing excludes it
Open this example in the editor and change it. Nothing you type is saved.
With a default index loc[0] and iloc[0] agree, which is why the difference goes unnoticed until the day you sort or filter and the labels stop matching the positions. Use loc when you mean a label and iloc when you mean a position, and say which you meant.
Note the slicing asymmetry: loc['b':'c'] includes 'c'; iloc[1:3] excludes index 3. That is deliberate - a label slice has no "one past the end" to name.
Combining masks needs parentheses and the bitwise operators:
import pandas as pd
sales = pd.DataFrame({
'region': ['North', 'South', 'North', 'East'],
'units': [120, 95, 143, 60],
})
print(sales[(sales['units'] > 100) & (sales['region'] == 'North')])
print(sales[sales['region'].isin(['South', 'East'])])
print(sales[~sales['region'].str.startswith('N')])
print(sales.query('units > 100 and region == "North"'))
Open this example in the editor and change it. Nothing you type is saved.
and, or and not do not work on Series - they ask for a single truth value and a Series of four booleans has none. Use &, | and ~, and wrap each comparison in parentheses because & binds tighter than >.
Adding and transforming columns
import pandas as pd
import numpy as np
sales = pd.DataFrame({
'region': ['North', 'South', 'North', 'East'],
'units': [120, 95, 143, 60],
'price': [12.5, 12.5, 11.0, 14.0],
})
sales['revenue'] = sales['units'] * sales['price']
sales['big'] = sales['units'] > 100
sales['band'] = np.where(sales['units'] > 100, 'high', 'low')
# assign returns a NEW frame, which makes it chainable.
enriched = sales.assign(
per_unit=lambda d: d['revenue'] / d['units'],
rank=lambda d: d['revenue'].rank(ascending=False).astype(int),
)
print(enriched[['region', 'revenue', 'per_unit', 'rank']].round(2))
Open this example in the editor and change it. Nothing you type is saved.
Vectorised, exactly as in NumPy. If you find yourself writing for i in range(len(df)), stop: there is nearly always a column operation that does it, and it will be both faster and clearer.
For a mapping with several branches, np.select beats a chain of np.where:
import pandas as pd
import numpy as np
sales = pd.DataFrame({'units': [120, 95, 143, 60, 30]})
conditions = [sales['units'] > 130, sales['units'] > 90, sales['units'] > 50]
choices = ['high', 'medium', 'low']
sales['band'] = np.select(conditions, choices, default='tiny')
print(sales)
Open this example in the editor and change it. Nothing you type is saved.
Looking at it
import pandas as pd
import numpy as np
rng = np.random.default_rng(0)
df = pd.DataFrame({
'region': rng.choice(['North', 'South'], 100),
'units': rng.integers(1, 200, 100),
'price': rng.normal(12, 2, 100).round(2),
})
df.loc[rng.choice(100, 7, replace=False), 'price'] = np.nan
print(df.head(3))
print()
df.info()
print()
print(df.describe().round(2))
print()
print(df['region'].value_counts())
print()
print(df.isna().sum())
Open this example in the editor and change it. Nothing you type is saved.
info() is the first thing to run on any new data. It tells you in one screen how many rows you have, which columns have missing values and - crucially - whether a column you expect to be a number has arrived as an object, which is pandas saying "these are strings".
describe() on an object column gives count, unique, top and frequency instead, which is often what you want for a categorical.
The copy warning
import pandas as pd
sales = pd.DataFrame({'units': [120, 95, 143, 60]})
subset = sales[sales['units'] > 100]
# In older pandas: SettingWithCopyWarning, and it was ambiguous
# whether the write reached `sales`.
# Under copy-on-write (the default from pandas 3.0), `subset` is
# always a copy and the original is never touched.
subset['units'] = 0
print(sales['units'].tolist()) # unchanged
print(subset['units'].tolist())
Open this example in the editor and change it. Nothing you type is saved.
Copy-on-write removed the ambiguity: a slice behaves as a copy, and if you want to change the original you say so.
import pandas as pd
sales = pd.DataFrame({'units': [120, 95, 143, 60]})
# Change the original, on purpose, in one statement.
sales.loc[sales['units'] > 100, 'units'] = 0
print(sales['units'].tolist()) # [0, 95, 0, 60]
Open this example in the editor and change it. Nothing you type is saved.
Write it as one loc assignment rather than a filter followed by a write. It is unambiguous under every version of pandas, and it says what you meant.
A worked example
import numpy as np
import pandas as pd
rng = np.random.default_rng(seed=11)
n = 400
orders = pd.DataFrame({
'order_id': np.arange(1000, 1000 + n),
'region': rng.choice(['North', 'South', 'East', 'West'], n, p=[.4, .3, .2, .1]),
'channel': rng.choice(['web', 'phone', 'store'], n, p=[.6, .2, .2]),
'units': rng.integers(1, 40, n),
'unit_price': rng.normal(12.5, 3.0, n).round(2),
'placed_at': pd.to_datetime('2026-01-01') + pd.to_timedelta(rng.integers(0, 120, n), 'D'),
})
# A natural key beats a positional index.
orders = orders.set_index('order_id')
print(orders.index.name, orders.shape)
print()
# Derived columns, chained so each step is visible.
orders = orders.assign(
revenue=lambda d: (d['units'] * d['unit_price']).round(2),
month=lambda d: d['placed_at'].dt.to_period('M').astype(str),
size=lambda d: pd.cut(d['units'], [0, 5, 20, 100], labels=['small', 'medium', 'large']),
)
print(orders.head(3)[['region', 'channel', 'units', 'revenue', 'month', 'size']])
print()
# Selection by label and by position, side by side.
first_id = orders.index[0]
print('by label ', orders.loc[first_id, 'revenue'])
print('by position', orders.iloc[0]['revenue'])
print()
# A compound mask - note the parentheses and the bitwise operators.
big_web = orders[(orders['channel'] == 'web') & (orders['units'] >= 20)]
print(f'{len(big_web)} large web orders, {big_web["revenue"].sum():,.2f} revenue')
print()
# The same question three ways, all giving the same answer.
by_region = orders.groupby('region')['revenue'].sum().sort_values(ascending=False)
print(by_region.round(2))
print()
print('North, via groupby:', round(float(by_region['North']), 2))
print('North, via mask: ', round(float(orders.loc[orders.region == 'North', 'revenue'].sum()), 2))
print('North, via query: ', round(float(orders.query('region == "North"')['revenue'].sum()), 2))
print()
# Alignment, which is the thing to understand about the index.
units_by_region = orders.groupby('region')['units'].sum()
revenue_by_region = orders.groupby('region')['revenue'].sum()
per_unit = (revenue_by_region / units_by_region).round(2)
print('revenue per unit, aligned by region label:')
print(per_unit)
print()
# What alignment does when the indexes differ.
partial = revenue_by_region.drop('West')
print((partial / units_by_region).round(2)) # West becomes NaN
print()
# Changing the original on purpose, in one unambiguous statement.
before = float(orders.loc[orders['region'] == 'West', 'revenue'].sum())
orders.loc[orders['region'] == 'West', 'revenue'] = 0.0
after = float(orders.loc[orders['region'] == 'West', 'revenue'].sum())
print(f'West revenue {before:,.2f} -> {after:,.2f}')
Open this example in the editor and change it. Nothing you type is saved.
Three habits are on display. The index is a real key, so loc means something. Derived columns are built with assign and lambdas, so the frame is never half-built. And the one place the original is modified uses a single loc assignment rather than a filter-then-write, which is unambiguous under any version.
When it goes wrong
| Symptom | Cause |
|---|---|
truth value of a Series is ambiguous |
and or or on a Series; use the bitwise operators with parentheses |
| A write to a filtered frame did nothing | It is a copy; use a single .loc assignment |
Two Series added to mostly NaN |
Indexes do not match; align or reset_index |
loc[0] and iloc[0] disagreed |
The frame was sorted or filtered |
KeyError on a column that exists |
Whitespace in the name; check df.columns.tolist() |
A numeric column has dtype object |
A stray string; see the cleaning lesson |
| A loop over rows is slow | iterrows is the slowest option; vectorise |
describe() ignored a column |
It is not numeric; pass include='all' |
A check you can run
import pandas as pd
a = pd.Series([1, 2, 3], index=[0, 1, 2])
b = pd.Series([10, 20, 30], index=[1, 2, 3])
print(a + b)
Open this example in the editor and change it. Nothing you type is saved.
The result has four rows and three of them are NaN. Nothing is broken: pandas aligned on the index, found one label in common, and told you about the rest.
Once you expect that, half of pandas stops being surprising - and the fix, when you wanted positional addition, is a.to_numpy() + b.to_numpy() or a.reset_index(drop=True) + b.reset_index(drop=True), chosen deliberately.