PANDAS:Missing values, dtypes, and cleaning

Missing is a value

Pandas uses NaN for a missing number and None or NaN for missing text, depending on the dtype. Arithmetic skips NaN in sum and mean by default. That is helpful until you forget two rows were never measured and publish a mean as if every bird was weighed.

import pandas as pd

penguins = pd.read_csv("penguins.csv")
print(penguins["sex"].isna().sum())
print(penguins["body_mass_g"].mean())

Output:

11
4201.754385964912

The mean used the rows that have a mass. The two missing masses were ignored, not treated as zero.

Drop or fill, on purpose

dropna removes rows. Pass subset so you only drop rows that are missing the columns you actually need.

import pandas as pd

penguins = pd.read_csv("penguins.csv")
measured = penguins.dropna(subset=["body_mass_g", "bill_length_mm"])
print(penguins.shape, measured.shape)

Output:

(344, 7) (342, 7)

Two rows had no measurements. The rows that are missing only sex stay, because sex was not in subset.

Fill when "unknown" is a category you want to keep:

import pandas as pd

penguins = pd.read_csv("penguins.csv")
penguins["sex"] = penguins["sex"].fillna("UNKNOWN")
print(penguins["sex"].value_counts())

Output:

sex
MALE       168
FEMALE     165
UNKNOWN     11
Name: count, dtype: int64

Dtypes drift

A column of integers becomes float64 as soon as it contains a missing value, because NaN is a float. Strings that should be categories can stay object. Force a dtype when you know better:

import pandas as pd

penguins = pd.read_csv("penguins.csv")
penguins["species"] = penguins["species"].astype("category")
print(penguins["species"].dtype)

Output:

category

pd.to_numeric(series, errors="coerce") turns bad tokens into NaN instead of raising. Use that when a CSV mixed numbers and the word "N/A".

What to notice

• Decide drop versus fill per column. Body mass has no honest fill. Sex can become "UNKNOWN".
• dropna() with no arguments drops a row if any column is missing. On this file that also drops the 11 birds missing sex, which you may still want for a mass average.
• After cleaning, print isna().sum() again. The point of the print is to show the problem moved, or that it is gone.

Try this

Load the penguins file, drop rows missing bill_length_mm, and print the new missing-value counts. Then fill the remaining missing sex values with "UNKNOWN" and confirm the count is 0.

Next: grouping rows and joining two tables.