We use cookies to enhance your experience on the site

Feature Engineering - turning a field notebook into data a model can read

Your notebook is full, @name. For half a season you wrote waterhole counts into it, then you patched the gaps, drew the charts and used tests to find out which differences were real. Yesterday a radio message came in from headquarters: build a model that predicts whether a species is endangered. You hand the notebook over exactly as it is, press enter, and get back a single line:

could not convert string to float: 'Lion'

The model cannot read the word "Lion". It cannot read "Savanna", it cannot read a date and it cannot read a sentence describing a track. A model sees numbers arranged in a table and nothing whatsoever beyond that. And the moment you do give it nothing but numbers, a second problem starts up: the count column runs from 85 to 5000, so one waterhole with a migrating herd crushes every other measurement, in exactly the way it crushed the mean in the previous lesson.

Both problems are solved by the same piece of work, and that work is what today is about.

A feature is what the model gets on the way in

A feature is one column of values that you hand to the model. Herd size is a feature. Territory area is a feature. Species is not one yet, because it is text, but in a moment we will turn it into three numeric features.

Feature engineering is creating new features from raw data for machine learning models. You take what you carried back from the field and rework it into columns the model can load and can actually learn something from. Sometimes you rescale an old column, sometimes you split one column into several, and sometimes - this is the best part - you invent a column that was never in the notebook at all.

Memorize that definition in precisely those words, because it is easy to confuse with three neighbouring things. Feature engineering is not removing all features from data - that would be absurd, since once every column is gone there is nothing left for a model to learn from. There is a related field called feature selection, which throws out useless columns, but it throws out some of them, never all of them, and it is a separate stage. Feature engineering is not the process of training models either - training is the next step along, the one with the

fit

call, and it begins once the features are ready. And finally, feature engineering is not a data visualization method - drawing is what Matplotlib and Seaborn are for, from the lesson on charts, and not one single picture will come out of today's work, only new columns of numbers.

The notebook we are working on

We start from the raw scribbles. Two familiar libraries are needed: pandas under the shortcut

pd

, which holds tables, and NumPy under the shortcut

np

, which computes on arrays. I build the data frame with the

pd.DataFrame

function from a dictionary, where the key is a column name and the value is a list of measurements. I call it plain

df

, because that is the name you will meet in the practical exercises. The notebook holds five observations and six columns: species, habitat, population, territory area in square kilometres, danger level as recorded in the field, and protection status written out in words as "Yes" and "No".

1import pandas as pd
2import numpy as np
3
4df = pd.DataFrame({
5    'species': ['Lion', 'Elephant', 'Cheetah', 'Lion', 'Elephant'],
6    'habitat': ['Savanna', 'Forest', 'Savanna', 'Forest', 'Savanna'],
7    'population': [120, 450, 85, 200, 5000],
8    'area_km2': [100, 500, 50, 200, 10000],
9    'danger': ['High', 'Low', 'High', 'Medium', 'Low'],
10    'endangered': ['Yes', 'No', 'Yes', 'No', 'No']
11})
12
13print(df)
14#     species  habitat  population  area_km2  danger endangered
15# 0      Lion  Savanna         120       100    High        Yes
16# 1  Elephant   Forest         450       500     Low         No
17# 2   Cheetah  Savanna          85        50    High        Yes
18# 3      Lion   Forest         200       200  Medium         No
19# 4  Elephant  Savanna        5000     10000     Low         No

Five rows, six columns and not one feature more. Four of those columns are pure text, so as far as the model is concerned they do not exist. The remaining two are numbers, but on a scale that does more harm than good. The whole rest of the lesson is about turning those six columns into a dozen or so that a model will genuinely accept.

The original stays untouched

Before we add anything, one rule of the camp: you never scribble over the raw notebook. If a new feature turns out to be nonsense, you want to be able to go back to the field records without reloading the file. Making an independent copy of a frame is the job of the

copy

method, called on the frame as

df.copy()

. It hands back a new table with the same data but living at a different address in memory, so adding a column to the copy does not touch the original. I name the copy

features

, because that is where everything we manufacture today will land.

1features = df.copy()
2
3print(features.shape)
4# (5, 6)

For now the copy is the identical twin of the original: the same five rows and six columns, as the

shape

attribute confirms. Nothing was computed and nothing changed - this is simply a second copy of the notebook, one you are allowed to scrawl on. Keep the name

df.copy()

in mind, because a few paragraphs from now I will come back to it in connection with a certain function that beginners very often mistake for copying a frame.

When one number crushes all the others

Look at the

population

column once more: 120, 450, 85, 200 and then suddenly 5000. The largest value is almost fifty nine times the smallest. For most models that means the first four waterholes blur into one indistinguishable dot right next to zero, while the model's entire attention goes to the fifth. A distribution shaped like that is called skewed - you met it in the statistics lesson as positive skew, the long tail on the right hand side.

The cure is a logarithm. A logarithm grows more and more slowly, so it squeezes big numbers harder than small ones and flattens that tail. NumPy has a ready made function called

log1p

, whose name reads as "the logarithm of one plus x": it takes a number, adds one to it, and only then computes the natural logarithm. That added one is not a whim. A plain

np.log

of zero gives minus infinity, and a waterhole where not a single animal turned up that day is a perfectly ordinary entry in the notebook.

np.log1p(0)

gives a calm zero and nothing breaks.

The call is made of six parts in a fixed order: first the library shortcut

np

, then a dot, then the function name

log1p

, then an opening bracket, then the thing to be converted, which is

df['population']

, and last a closing bracket. The dot tells Python "reach inside NumPy", and the brackets say "do it now". The bare name

np.log1p

without brackets would compute nothing at all - it would only be a pointer to a function. The result of the logarithm carries a dozen or so decimal places, so for printing I attach the

round

method, which trims numbers to the given number of decimal places and changes nothing in the data itself.

1features['log_population'] = np.log1p(df['population'])
2
3print(features[['population', 'log_population']].round(2))
4#    population  log_population
5# 0         120            4.80
6# 1         450            6.11
7# 2          85            4.45
8# 3         200            5.30
9# 4        5000            8.52

The ratio of the largest value to the smallest dropped from 58.8 to 1.9, so the migrating herd has stopped drowning out the rest of the reserve. More importantly, the order of the rows did not change: the smallest waterhole still has the smallest value and the largest still has the largest, because a logarithm is an increasing function and never rearranges data. The

population

column did not change either, and neither did anything at all in the

df

frame - a seventh column simply joined the old six inside

features

The same yardstick for every column: standardization

The logarithm improved the shape of one column, but it did not solve the second complaint: population is measured in hundreds, area in thousands, and density will come out in fractions. A model that computes distances between observations will then treat the column with the bigger numbers as the more important one, even though nobody ever said so. All the features have to be brought onto a common yardstick.

The most common way is standardization, also known as the Z-score. From every value you subtract the mean of the column, and you divide the result by the standard deviation of that column. You know both of those ideas from the previous lesson: the mean is computed by the

mean

method and the standard deviation by the

std

method, both called directly on the column. The result reads very simply: zero means exactly the mean, one means "one deviation above the mean", minus two means "two deviations below".

1mean = df['population'].mean()
2std = df['population'].std()
3
4features['pop_standardized'] = (df['population'] - mean) / std
5
6print(features[['population', 'pop_standardized']].round(2))
7#    population  pop_standardized
8# 0         120             -0.49
9# 1         450             -0.34
10# 2          85             -0.51
11# 3         200             -0.45
12# 4        5000              1.78

The four ordinary waterholes landed just below zero, and the wildebeest herd sits 1.78 deviations above the mean. Notice that the subtraction and the division worked on the whole column at once, with no loop anywhere - that is the NumPy behaviour you already know, where an arithmetic operation spills across every element. Neither the order of the rows nor their number changed in the process: standardization shifts and rescales the axis, it does not touch the data.

That the axis really did move underneath the mean can be checked by hand. The new column has, by definition, a mean of zero and a standard deviation of one, no matter what units the original numbers were measured in.

1print(round(features['pop_standardized'].mean(), 10))
2# -0.0
3print(round(features['pop_standardized'].std(), 10))
4# 1.0

The mean came out as minus zero, which is ordinary floating point litter: the true result is a number of the order of ten to the minus seventeenth, and rounding to the tenth decimal place leaves nothing behind but the minus sign. The deviation came out at exactly 1.0. From now on the

pop_standardized

column speaks the same language as every other standardized column in the table, and the model can compare them fairly.

A scale from zero to one: Min-Max normalization

Standardization offers no guarantee at all about the range - values can come out at minus three and at plus ten. There are situations where you need hard bounds, for instance when you are feeding a neural network, or when you want to show the guides a progress bar running from zero to a hundred percent. That is when you reach for Min-Max normalization.

The formula has three ingredients and all of them come from the same column: from the value you subtract the column minimum, and you divide the result by the range, meaning the difference between the maximum and the minimum. We write it as

(x - min) / (max - min)

. The minimum is given by the

min

method and the maximum by the

max

method, both called on the column exactly the way

mean

was; I park them in the variables

lo

and

hi

so the formula can be read at a single glance. The effect is always the same and follows straight from the arithmetic: the smallest measurement puts a zero in the numerator, so it comes out as exactly 0.0; the largest makes the numerator equal to the denominator, so it comes out as exactly 1.0; everything else falls somewhere in between.

1lo = df['population'].min()
2hi = df['population'].max()
3
4features['pop_normalized'] = (df['population'] - lo) / (hi - lo)
5
6print(features[['population', 'pop_normalized']].round(3))
7#    population  pop_normalized
8# 0         120           0.007
9# 1         450           0.074
10# 2          85           0.000
11# 3         200           0.023
12# 4        5000           1.000

The waterhole with 85 animals got a zero, the one with 5000 got a one, and the other three squeezed into the bottom eight hundredths of the interval. And that is at the same time the biggest weakness of this method: a single outlier eats the entire scale, packing the rest of the data up against zero. Standardization, in the previous section, spread the very same numbers out far more clearly. That is why I recommend standardization as the default choice, and Min-Max normalization for when a range from zero to one is a hard requirement from whoever receives the data - and preferably after taking the logarithm of a skewed column first. What did not change here either: the order of the rows, and the original

population

column, which you can still go back to.

Numbers that would rather be categories

Sometimes the exact head count is not what you want to give the model at all. The reserve office is not asking whether there were 120 or 130 animals at the waterhole - it is asking whether that herd is small, medium, large or huge. Turning a number into an interval is called binning (from bin, a bucket), and in pandas it is done by the

pd.cut

function.

pd.cut

takes three things. The first is the column to be cut. The second is

bins

, the list of bucket edges - give it four edges and you get three intervals, give it five and you get four. The third is

labels

, the list of names for those intervals, and it must be exactly one item shorter than the list of edges. I set the last edge to

np.inf

, NumPy's infinity, so that the top bucket will accept a herd of any size at all. The intervals are open on the left and closed on the right by default, so

(0, 100]

includes one hundred but not zero.

1features['pop_category'] = pd.cut(
2    df['population'],
3    bins=[0, 100, 300, 1000, np.inf],
4    labels=['small', 'medium', 'large', 'huge']
5)
6
7print(features[['population', 'pop_category']])
8#    population pop_category
9# 0         120       medium
10# 1         450        large
11# 2          85        small
12# 3         200       medium
13# 4        5000         huge

Five numbers turned into five labels: 85 fell into

small

, 120 and 200 into

medium

, 450 into

large

, and the wildebeest herd into

huge

. Now for the most important sentence in this section, the one about what did not happen. Not a single row disappeared - there are still five of them. Not a single column disappeared. No copy of the frame was made.

pd.cut

divides numerical data into categories, that is, into bins, and adds a new column made out of them, and it leaves absolutely everything else in peace.

That is worth underlining, because the word cut suggests three untrue things at once. Copying a frame is the job of

df.copy()

, which you met earlier, and it was its result that we named

features

. Removing columns is

df.drop(columns=['area_km2'])

from the pandas lesson, and after it the frame really does have fewer columns. Cutting out the rows that meet a condition is a boolean mask, something written along the lines of

df[df['population'] > 100]

, which hands back a shorter table.

pd.cut

does none of those three things - all it does is stick interval labels onto numbers.

Buckets with equal counts instead of equal widths

pd.cut

slices the number line at the places you point to yourself, so the buckets can end up wildly uneven in terms of how many observations they hold - it can easily happen that one waterhole lands in

huge

and forty land in

medium

. When you want the opposite effect, buckets holding roughly the same number of observations, you reach for

pd.qcut

. The letter "q" stands for quantiles, which you know from the statistics lesson. Instead of edges you pass the

parameter with the number of groups:

q=4

gives quartiles,

q=10

gives deciles. The group names are passed the same way as in

pd.cut

, through

labels

1features['pop_quartile'] = pd.qcut(
2    df['population'],
3    q=4,
4    labels=['Q1', 'Q2', 'Q3', 'Q4']
5)
6
7print(features[['population', 'pop_quartile']])
8#    population pop_quartile
9# 0         120           Q1
10# 1         450           Q3
11# 2          85           Q1
12# 3         200           Q2
13# 4        5000           Q4

The edges computed themselves, out of the data alone, and nobody had to guess where a "small" herd stops being small. Five observations cannot be split evenly into four groups, so

Q1

got two waterholes and the other groups got one each - with bigger datasets that unevenness disappears. Notice that 5000 landed in

Q4

in just the same way that 450 landed in

Q3

: for

pd.qcut

the only thing that counts is your position in the queue, not the distance, which makes it a method that shrugs off outliers. The

population

column, once again, came through untouched.

The feature that was never in the notebook

Everything we have done so far reworked columns that already existed. Now comes the finest part of feature engineering: inventing columns that nobody measured. Look at two numbers from the notebook, the population and the territory area. Each of them on its own says very little: 5000 animals sounds impressive right up until you ask how large an area they are spread across. What is genuinely interesting is the density, the number of animals per square kilometre. It comes from dividing one column by another and it is called a ratio feature.

1features['density'] = df['population'] / df['area_km2']
2
3print(features[['species', 'population', 'area_km2', 'density']])
4#     species  population  area_km2  density
5# 0      Lion         120       100      1.2
6# 1  Elephant         450       500      0.9
7# 2   Cheetah          85        50      1.7
8# 3      Lion         200       200      1.0
9# 4  Elephant        5000     10000      0.5

Read that table carefully, because it turns your conclusions upside down. The largest population, those 5000 animals that were crushing everything, has the lowest density in the whole reserve: 0.5 animals per square kilometre. The smallest population, 85 cheetahs, has the highest: 1.7. Neither of the source columns said any of this on its own - the information was sitting in the relationship between them and only came to light after the division. As a bonus, the new column has no scale problem left, because every value fits between 0.5 and 1.7. Both source columns stayed in the frame unchanged and you can go on using them.

Squares, roots and products

A ratio is not the only way to mould a new feature. When you suspect that a relationship is not a straight line - because a territory twice as big does not feed twice as many animals - you add polynomial features: a column raised to a square with the

**

operator, or its root through the

np.sqrt

function. And when you suspect that only two things taken together carry the signal, you build an interaction feature, the product of two columns.

1features['area_sqrt'] = np.sqrt(df['area_km2'])
2features['pop_x_area'] = df['population'] * df['area_km2']
3
4print(features[['area_km2', 'area_sqrt', 'pop_x_area']].round(2))
5#    area_km2  area_sqrt  pop_x_area
6# 0       100      10.00       12000
7# 1       500      22.36      225000
8# 2        50       7.07        4250
9# 3       200      14.14       40000
10# 4     10000     100.00    50000000

The square root squeezed ten thousand square kilometres down to a hundred and evened out the scale much as the logarithm did. The product went the other way and reached fifty million, so a feature like that wants standardizing before it goes anywhere near a model. And one warning from camp: given that any pair of columns can be multiplied, divided and squared, six columns easily become a hundred. Do not do it. Build the features you can justify in words, the way density was justified in the previous section - the rest is noise, and a model loses the trail in noise. The original columns, needless to say, are still lying in the frame untouched.

The model will not read the word "Lion"

Back to the error from the start of the lesson. Four of the notebook's columns are text, and a model accepts nothing but numbers. They have to be encoded, and the most important encoding method is One-Hot Encoding.

The idea goes like this: every possible value of the category gets its own separate column, and in that column we write a one when the row belongs to the category and a zero when it does not. The name comes from electronics - in every row exactly one column is "hot", that is, lit, and all the others are dark. In pandas this is done by the

pd.get_dummies

function, where dummy means a stand-in column. Let us start with a single column so you can see the mechanism on its own.

1print(pd.get_dummies(df['species']))
2#    Cheetah  Elephant   Lion
3# 0    False     False   True
4# 1    False      True  False
5# 2     True     False  False
6# 3    False     False   True
7# 4    False      True  False

One text column turned into three boolean columns, one per species occurring in the data. In every row exactly one value is

True

- row zero is a lion, so the

Lion

column lit up. Pandas returns the logical values

True

and

False

here by default rather than ones and zeros, but for a model that is the same information, because

True

is treated as 1 and

False

as 0. If you would rather look at numbers, you add the

dtype=int

argument.

1print(pd.get_dummies(df['habitat'], dtype=int))
2#    Forest  Savanna
3# 0       0        1
4# 1       1        0
5# 2       0        1
6# 3       1        0
7# 4       0        1

Two habitats, two binary columns, one single one in every row. It is worth noticing that with only two categories the second column adds nothing new - if

Forest

holds a zero then

Savanna

has to hold a one. For some models that redundancy is harmful, so

pd.get_dummies

accepts a

drop_first=True

argument, which removes the first column from each set.

The whole frame at once

In practice you rarely encode columns one at a time.

pd.get_dummies

accepts a whole frame and finds everything inside it that is text by itself. The call is made of six parts in a fixed order: the shortcut

pd

, a dot, the name

get_dummies

, an opening bracket, the frame name

df

, a closing bracket.

1encoded = pd.get_dummies(df)
2
3print(encoded.shape)
4# (5, 12)

Six columns turned into twelve while the number of rows stayed exactly the same - and that is the heart of this operation. The four text columns unfolded into ten binary ones: three species, two habitats, three danger levels and two protection statuses. The two numeric columns,

population

and

area_km2

, went through
pd.get_dummies
without any change at all - the function touches categories only and leaves numbers alone.

That last sentence is worth remembering, because a few stubborn misunderstandings circle around One-Hot Encoding. It is not used for normalizing numerical data - normalization is the formula

(x - min) / (max - min)

you met earlier, or standardization, and both of those work on numbers; the fact that encoding produces zeros and ones is an accidental resemblance, because nobody scaled those zeros and ones, they were merely switched on. It is not used for removing outliers either - you detect extreme measurements with the interquartile range rule from the data exploration lesson, and One-Hot removes not one single row, it only adds columns. And finally it is not used for merging DataFrames - gluing tables together is what

pd.merge

and

pd.concat

from the pandas lesson are for, while

pd.get_dummies

works on one frame and never even sees a second one. One-Hot Encoding does one job: it converts categorical variables into binary columns.

The temptation to number the categories

There is a shorter road to encoding the species: number them one after another. That is what

LabelEncoder

from the scikit-learn library does, from the package imported as

sklearn

, and more precisely from its

preprocessing

submodule, where all the data preparation tools live. You create a

LabelEncoder()

object and then call the

fit_transform

method on it, which in one movement learns the list of categories and swaps them for numbers. Let us see what comes of it, because the result is instructive.

1from sklearn.preprocessing import LabelEncoder
2
3le = LabelEncoder()
4features['species_encoded'] = le.fit_transform(df['species'])
5
6print(features[['species', 'species_encoded']])
7#     species  species_encoded
8# 0      Lion                2
9# 1  Elephant                1
10# 2   Cheetah                0
11# 3      Lion                2
12# 4  Elephant                1

One column instead of three, so it looks economical. The trouble is that the numbers were handed out alphabetically: Cheetah got 0, Elephant 1, Lion 2. A model given a column like that will read things out of it that nobody ever wrote - that a lion is bigger than an elephant, that an elephant lies exactly halfway between a cheetah and a lion, and that the average of a cheetah and a lion is an elephant. Species have no order, so every number like that is a lie planted in the data. That is why for input columns with no natural ordering I recommend

pd.get_dummies

and nothing else, and I keep

LabelEncoder

for encoding the column being predicted, where the model treats the values as class names rather than as quantities.

When the order really does exist

There are cases, though, where the categories have an order written into their very definition. The

danger

column takes the values "Low", "Medium" and "High" - and here high danger genuinely is greater than low. Categories like that are called ordinal, and assigning them increasing numbers is then an honest thing to do. You do it with a dictionary and the

map

method, which walks along the column and swaps every value for whatever it finds under that key in the dictionary.

1danger_map = {'Low': 0, 'Medium': 1, 'High': 2}
2features['danger_level'] = df['danger'].map(danger_map)
3
4print(features[['danger', 'danger_level']])
5#    danger  danger_level
6# 0    High             2
7# 1     Low             0
8# 2    High             2
9# 3  Medium             1
10# 4     Low             0

The order of the numbers now mirrors an order that exists in reality, so the model can honestly work out that a one step rise in danger changes something. Watch out for one trap:

map

gives no warning about unknown values. If somebody had written "Extreme" into the notebook, and it is not in the dictionary, a silent missing value

NaN

would appear in the resulting column - which is why it is worth checking after every mapping whether any new holes have opened up. The

danger

column in its text form stayed in the frame untouched.

Yes and no in a single column

That leaves the last text column:

endangered

, with the words "Yes" and "No". With only two possible values you need neither one-hot encoding nor a dictionary - a comparison is enough. The expression

df['endangered'] == 'Yes'

checks the condition for the whole column at once and hands back a column of boolean values, exactly the same mask you used for filtering in the pandas lesson. The

astype

method changes the type of a column, and

astype(int)

turns

True

into 1 and

False

into 0.

1features['endangered_binary'] = (df['endangered'] == 'Yes').astype(int)
2
3print(features[['endangered', 'endangered_binary']])
4#   endangered  endangered_binary
5# 0        Yes                  1
6# 1         No                  0
7# 2        Yes                  1
8# 3         No                  0
9# 4         No                  0

The brackets around the comparison are essential, because without them Python would first try to call

astype

on the string

'Yes'

. The result is exactly what One-Hot Encoding with

drop_first=True

would give on a two category column - one column instead of two, with no loss of information. Notice that the original column is still standing right next to it, so in the report for the office you can show a readable "Yes" while handing the model a 1.

A date is not one fact but six

A column holding an observation date is even worse for a model than text: even if you grind it down into a number of seconds, what comes out is a counter rising forever, and you cannot read either the season or the rhythm of the week out of it. A date has to be broken into parts. Pandas keeps them under the

dt

accessor, available on any column of date type. You reach for them with the form

column.dt.name

dt.year

is the year,

dt.month

is the month number,

dt.quarter

is the quarter,

dt.dayofweek

is the day of the week counted from zero for Monday to six for Sunday, and

dt.dayofyear

is the day number within the year. The dates themselves I load with the

pd.to_datetime

function, which turns strings into real dates.

1dates = pd.DataFrame({
2    'observation_date': pd.to_datetime([
3        '2023-01-14', '2023-03-15', '2023-07-04', '2023-12-31'
4    ])
5})
6
7dates['year'] = dates['observation_date'].dt.year
8dates['month'] = dates['observation_date'].dt.month
9dates['day_of_week'] = dates['observation_date'].dt.dayofweek
10dates['quarter'] = dates['observation_date'].dt.quarter
11dates['day_of_year'] = dates['observation_date'].dt.dayofyear
12
13print(dates)
14#   observation_date  year  month  day_of_week  quarter  day_of_year
15# 0       2023-01-14  2023      1            5        1           14
16# 1       2023-03-15  2023      3            2        1           74
17# 2       2023-07-04  2023      7            1        3          185
18# 3       2023-12-31  2023     12            6        4          365

One column became six, and each one carries a different kind of signal: the year catches the multi year trend, the month and the quarter catch the dry and the rainy season, the day of the week catches the working rhythm of the camp. Check it on the second row: 15 March 2023 was a Wednesday, so

day_of_week

is 2, counting Monday as zero. The last row is 31 December, day 365 of the year and the fourth quarter. The

observation_date

column itself stayed in the table - breaking a date into parts takes nothing away from it.

Out of those parts you can immediately glue together a feature more useful than any of them alone. The

isin

method checks whether a value belongs to a given list and hands back a boolean mask - and five and six are precisely Saturday and Sunday. The

astype(int)

on the end turns the mask into zeros and ones.

1dates['is_weekend'] = dates['day_of_week'].isin([5, 6]).astype(int)
2
3print(dates[['observation_date', 'day_of_week', 'is_weekend']])
4#   observation_date  day_of_week  is_weekend
5# 0       2023-01-14            5           1
6# 1       2023-03-15            2           0
7# 2       2023-07-04            1           0
8# 3       2023-12-31            6           1

Two observations fell at the weekend and got a one, two fell midweek and got a zero. It is far easier for a model to pick up one clear zero-one column than to guess that the values 5 and 6 in the

day_of_week

column happen to mean something they share. The same trick works for the dry season, the migration window or the days after a downpour.

December and January are neighbours

There is one problem with a month written as a number from 1 to 12, and it is not obvious at first glance. As far as the model is concerned December, which is 12, lies eleven units away from January, which is 1 - further away than June is. In nature it is the other way round: one month passes between a December and a January observation and the weather is almost the same. The year is a circle, and a number from 1 to 12 is a line segment.

The answer is cyclical encoding: instead of one number you give the model two, being the position of the month on a circle. They are computed by the

np.sin

and

np.cos

functions, and the formula is the same in both cases - you divide the month number by 12 to get the fraction of the year, and multiply by the full angle, which is two times

np.pi

, where

pi

is the constant pi as recorded in NumPy. I pack the month numbers into

pd.Series

, the single column structure you know from the pandas lesson, because four sample months do not need a whole table.

1months = pd.Series([1, 2, 6, 12])
2
3month_sin = np.sin(2 * np.pi * months / 12)
4month_cos = np.cos(2 * np.pi * months / 12)
5
6print(pd.DataFrame({
7    'month': months,
8    'month_sin': month_sin.round(3),
9    'month_cos': month_cos.round(3)
10}))
11#    month  month_sin  month_cos
12# 0      1      0.500      0.866
13# 1      2      0.866      0.500
14# 2      6      0.000     -1.000
15# 3     12     -0.000      1.000

Now compare the pairs of points on that circle. January sits at

(0.5, 0.866)

, December at

(0.0, 1.0)

, February at

(0.866, 0.5)

. The distance from January to December is 0.518 and it is exactly the same as the distance from January to February - so the December and January pair is finally as close together as it is in reality. June, half a year away, lies 1.932 from January, on the opposite side of the circle. The minus in front of the zero in the last row is floating point litter again: the true value of the sine of a full angle is a number of the order of ten to the minus sixteenth. The same trick works on the hour of the day and on the day of the year.

A field description is data too

One thing is left in the notebook, the one least like a table: the sentences the guides pencil in beside an observation. You cannot hand them to a model directly, but you can squeeze numbers out of them. Pandas offers text operations under the

str

accessor, working on the whole column at once:

str.len

gives the length of the string in characters,

str.split

breaks a sentence into a list of words, and

str.contains

checks whether a given fragment occurs in the text and hands back the boolean mask you already know. By attaching one more

.str.len()

str.split()

you get the number of words in each sentence.

1notes = pd.DataFrame({
2    'description': [
3        'Large male lion spotted near river',
4        'Small elephant calf with mother',
5        'Fast cheetah hunting gazelle'
6    ]
7})
8
9notes['text_length'] = notes['description'].str.len()
10notes['word_count'] = notes['description'].str.split().str.len()
11notes['has_river'] = notes['description'].str.contains('river').astype(int)
12
13print(notes[['text_length', 'word_count', 'has_river']])
14#    text_length  word_count  has_river
15# 0           34           6          1
16# 1           31           5          0
17# 2           28           4          0

Three sentences turned into three columns of numbers, and each carries a different signal: the length of a description is often a measure of how unusual the sighting was, while

has_river

is a ready made zero-one feature saying that the animal was seen near water. The

description

column stayed untouched and is still fit for humans to read. There are of course methods that take the entire vocabulary of a text and weigh the importance of every word - the best known is called TF-IDF - but they belong to natural language processing and carry far too many ideas of their own to fit into this lesson. For now three hand invented columns are more than enough.

Last camp: compute your statistics where you are allowed to

One warning remains, and without it all the work above can quietly cheat on you. When we computed the standardization, the mean and the deviation were computed from all five rows. If some of those rows were test data, the data the model is only going to be checked against, then information from the test would have leaked into the data preparation and the results would come out far too beautiful. That is why in real projects the transformations are described up front and only then run on the appropriate slice of data.

The tool for that is

ColumnTransformer

from the

compose

submodule of scikit-learn: it takes a list of jobs, where each job is a triple - a name of your choosing, a transforming tool, and the list of columns it is to work on.

StandardScaler

is ready made standardization, the thing you computed by hand earlier, and

OneHotEncoder

is ready made one-hot encoding, with a

drop='first'

argument matching the

drop_first

you already know. The

fit_transform

method does two things at once: it learns the statistics it needs from the given data and transforms that data immediately.

1from sklearn.compose import ColumnTransformer
2from sklearn.preprocessing import StandardScaler, OneHotEncoder
3
4preprocessor = ColumnTransformer([
5    ('num', StandardScaler(), ['population', 'area_km2']),
6    ('cat', OneHotEncoder(drop='first'), ['habitat'])
7])
8
9matrix = preprocessor.fit_transform(df)
10
11print(matrix.shape)
12# (5, 3)

Out of the frame came a plain NumPy array with five rows and three columns: two numeric columns after standardization plus one binary column for the habitat, because with two categories and

drop='first'

only one is left. The columns you did not name in any job simply did not make it into the result -

ColumnTransformer

passes through only what it was pointed at. The standardized values here differ from the ones we computed by hand - the wildebeest row comes out at 2.00 instead of 1.78 - because scikit-learn divides by the deviation computed for a whole population while pandas divides by the sample deviation by default. That same

preprocessor

can later be dropped in as the first step of a

Pipeline

object, which welds the data preparation and the model into one whole - and then the statistics are computed on the training data only and a leak becomes impossible.

What you take back to camp

Feature engineering is creating new features from raw data for ML models. It is not removing all features from data, it is not the process of training models (that is
```
fit
```
, later on) and it is not a data visualization method (Matplotlib and Seaborn are there for that).
You flatten a skewed column with a logarithm, in the order
```
np
```
,
```
.
```
,
```
log1p
```
,
```
(
```
,
```
df['population']
```
,
```
)
```
. The
```
log1p
```
variant instead of
```
log
```
survives zeros.
Standardization is
```
(x - mean) / std
```
and gives a mean of 0 and a deviation of 1. Min-Max normalization is
(x - min) / (max - min)
and forces the data into exactly the range from 0.0 to 1.0. By default I recommend standardization, because a single outlier does not eat its scale.
pd.cut
divides numerical data into categories, that is, into bins. It does not copy a DataFrame (that is
```
df.copy()
```
), it does not remove columns from a DataFrame (that is
```
df.drop
```
) and it does not cut the rows that meet a condition (that is a boolean mask).
```
pd.qcut
```
does the same job with buckets of equal counts.
The most valuable features come out of relationships between columns - density showed that the largest population has the lowest density in the reserve.
One-Hot Encoding converts categorical variables into binary columns. It does not remove outliers, it does not normalize numerical data and it does not merge DataFrames -
```
pd.merge
```
and
```
pd.concat
```
are there for merging.
The encoding call is six parts:
```
pd
```
,
```
.
```
,
```
get_dummies
```
,
```
(
```
,
```
df
```
,
```
)
```
. Numeric columns pass straight through it unchanged.
```
LabelEncoder
```
numbers categories alphabetically and talks the model into an order that does not exist. Number only genuinely ordinal categories, with a dictionary and the
```
map
```
method.
You break a date apart with the
```
dt
```
accessor into year, month, quarter, day of the week and day of the year, and you rescue the cycle with the pair
```
np.sin
```
and
```
np.cos
```
.
You turn text into numbers with
```
str.len
```
,
```
str.split
```
and
```
str.contains
```
, and you compute the statistics for scaling at the very end through
```
ColumnTransformer
```
, so that they cannot leak in from the test data.

The model will never see what you saw at the waterhole, @name - it will only see the columns you mould for it, so from today it is you who decides how much of the reserve can be read at all.

Go to CodeWorlds→