EDA - scouting the terrain before the hunt

You have just been handed a fresh set of observations from the reserve, @name. Before you train any model on it, you have to do what every good tracker does before the hunt: walk the terrain first. EDA (Exploratory Data Analysis) is that scouting trip - you answer a series of simple questions about the data until you know exactly what you are dealing with. The whole point of it is understanding data structure, discovering patterns and anomalies - not creating user interfaces, not deploying applications to production, and not yet training machine learning models. Training comes later, once the scouting is done. We will walk through the questions one at a time.

Question 1: how big is the terrain?

The first look is always about the size and shape of the data - how many observations, how many columns, and what the first rows actually contain.

1import pandas as pd
2import numpy as np
3import matplotlib.pyplot as plt
4import seaborn as sns
5
6df = pd.read_csv('safari_observations.csv')
7
8print("Shape:", df.shape)   # (number of rows, number of columns)
9print(df.head())            # the first few observations
10print(df.info())            # column names, types and non-null counts
11print(df.describe())        # statistics of the numerical columns

These few lines are your first report from the field.

shape

tells you how big the set is,

head

shows what is actually recorded in each column,

info

lists the columns with their types and how many values are filled in, and

describe

hands you means, minimums and maximums straight away. Before you go a step further you already know whether you are dealing with a hundred rows or a million. This is exactly the opening move of the workflow: first load the data, then check shape and info.

Question 2: what lives here? (column types)

A model treats numbers and categories differently, so you have to separate the numerical columns (population, weight) from the categorical ones (species, habitat).

1categorical = df.select_dtypes(include=['object', 'category']).columns
2numerical = df.select_dtypes(include=[np.number]).columns
3
4for col in categorical:
5    print(f"{col}: {df[col].nunique()} unique values")
6    print(df[col].value_counts().head())

select_dtypes

splits the columns by type, and

value_counts

shows how many rows fall into each category. This part of the scouting matters more than it looks: if a species column holds three distinct values it is a genuine category, but if it holds 10,000 it is really an identifier, and a model should never treat it as a feature.

Question 3: where are the holes? (missing data)

Real field data is full of holes - some observations come back incomplete. You need to know where and how many before those gaps quietly ruin your model.

1missing = df.isnull().sum()
2missing_pct = (missing / len(df)) * 100
3missing_df = pd.DataFrame({'missing': missing, 'percent': missing_pct})
4print(missing_df[missing_df['missing'] > 0].sort_values('missing', ascending=False))
5
6sns.heatmap(df.isnull(), cbar=True, yticklabels=False)
7plt.title('Missing Values Map')
8plt.show()

df.isnull().sum()

is the one command to memorize here: it counts the missing values column by column. Pandas has no

df.missing()

and no

df.count_nulls()

- those simply do not exist - and

df.empty

answers a completely different question, namely whether the frame has no rows at all. The table tells you which columns are leaky and how badly (the percentage matters more than the raw count), and the heat map adds the picture: if the gaps line up in stripes, whole batches of observations may be missing - a trail suggesting something went wrong during collection rather than by chance.

Question 4: what is normal and what stands out? (distributions)

Now you look at the numerical columns one by one: how the values are spread out, and whether there are outliers - observations that do not fit with the rest.

1df[numerical].hist(figsize=(15, 10), bins=30, edgecolor='black')
2plt.tight_layout()
3plt.show()
4
5def find_outliers_iqr(df, column):
6    Q1 = df[column].quantile(0.25)
7    Q3 = df[column].quantile(0.75)
8    IQR = Q3 - Q1
9    lower, upper = Q1 - 1.5 * IQR, Q3 + 1.5 * IQR
10    return df[(df[column] < lower) | (df[column] > upper)]
11
12outliers = find_outliers_iqr(df, 'population')
13print(f"Outliers in population: {len(outliers)}")

Histograms show the shape of a distribution at a glance.

find_outliers_iqr

applies the IQR (Interquartile Range) method - the classic quartile-based rule of "1.5 x the distance between the first and the third quartile" for flagging values that are far too large or far too small. An arithmetic mean method would fail here, because a single extreme animal drags the mean along with it, while checksums and hashing only confirm that a file arrived intact and say nothing about statistics. And remember: an outlier is not automatically an error. An elephant in weight data is a natural extreme - your job is to decide whether it is a mistake to remove or a real, important animal.

Question 5: what moves together? (correlations)

Finally you check which features move together - when one goes up, the other follows. That tells you what will help with prediction, and which columns are essentially saying the same thing twice.

1correlation = df[numerical].corr()
2
3sns.heatmap(correlation, annot=True, cmap='coolwarm', center=0, fmt='.2f')
4plt.title('Correlation Matrix')
5plt.show()
6
7corr_pairs = correlation.unstack().sort_values(ascending=False)
8print(corr_pairs[corr_pairs < 1].head(10))

The correlation map colors every pair of features: values close to

1

(or

-1

) mean a strong relationship, values close to

0

mean none at all, and unstacking the matrix turns it into a plain ranked list of the strongest pairs. Why is this worth knowing? Two nearly identical features (a correlation of 0.98) carry the same information, so you usually keep only one of them. And a feature strongly tied to whatever you are trying to predict is your best candidate to feed the model.

Question 6: how do the groups differ?

Correlation links numbers to numbers. But often the most interesting question is whether a category changes a number - does population differ between habitats, for instance? A box plot split by category answers exactly that.

1for cat in categorical:
2    for num in numerical:
3        sns.boxplot(data=df, x=cat, y=num)
4        plt.title(f'{num} by {cat}')
5        plt.xticks(rotation=45)
6        plt.show()

Each

boxplot

compares the distribution of a numerical variable across groups. If the boxes for different habitats sit at completely different heights, that is a sign the habitat genuinely explains population - and that the category is worth handing to the model. If the boxes overlap almost entirely, the category adds very little, and you can safely leave it out.

The whole scouting run in one function

Since you ask the same questions of every new dataset, it pays to pack that first report into a single function you can fire the moment the data is loaded.

1def eda_summary(df):
2    print("=" * 50)
3    print("EDA REPORT")
4    print("=" * 50)
5    print(f"Rows: {df.shape[0]}")
6    print(f"Columns: {df.shape[1]}")
7    print(f"Missing data: {df.isnull().sum().sum()}")
8    print(f"Duplicates: {df.duplicated().sum()}")
9    print(df.dtypes.value_counts())
10    print(df.describe())
11
12eda_summary(df)

This function gathers the answers to questions 1-3 into one quick report - size, types, missing values and duplicates. Note

df.duplicated().sum()

:

duplicated

marks every repeated row and

sum

counts those marks, so a single number tells you whether the same animal was written down twice. From now on every new dataset starts from the same trusted scouting routine.

What you should take away from this lesson is the order, not the commands: load the data, check shape and info, analyze missing data, analyze distributions, analyze correlations. You train a model only once that scouting run is behind you - otherwise you are hunting blind in unknown terrain.