We use cookies to enhance your experience on the site

Gradient Boosting - XGBoost, LightGBM, CatBoost

In the scikit-learn lesson you built a random forest: a hundred trees, a hundred trackers, each one handed its own sample of specimens and its own set of measurements, and at the end everybody votes. The idea works beautifully for as long as the trackers get things wrong in different places - then the mistakes cancel each other out and the majority is right.

But there is a square of savanna where the whole forest falls apart. A young hyena can weigh exactly as much as a starved lioness, its paw print is about the same size, and it walks the very same corridor. A hundred independent trackers, none of whom has ever spoken to the others, will make the same mistake there a hundred times over. Voting does not save you, because the majority is wrong together. And there is nobody in that forest who says, after the council is over: "look, twenty specimens keep landing in the wrong column, let us go and deal with them".

What you are missing is a mechanism that takes one tracker's list of mistakes and sends the next tracker straight at those mistakes. Not another independent vote, but a correction of what the previous one could not do. That mechanism is called gradient boosting, and it is what stands behind most of the winning entries in competitions on tabular data - data shaped exactly like your field notebook, where every row is one animal and every column one measurement.

A queue instead of a council

Memorize one sentence, because it is the definition of the entire technique: gradient boosting combines weak models sequentially, and each one corrects the errors of the previous one.

Let us take it apart, phrase by phrase. A weak model is a model barely better than guessing - in practice a very shallow decision tree, sometimes literally one question and two answers. A stump like that is useless on its own, but it is fast and it is hard to overfit. Sequentially means one after another, in a queue, not side by side: the second model cannot come into existence before the first has finished its work, because only then do you know what went wrong. And finally, corrects the errors of the previous one - the next model does not learn from the original answers at all. It learns from how far off the previous one was.

That error has a name of its own: the residual, meaning the true value minus whatever the model predicted. The word "gradient" comes from the fact that the residual is, mathematically, the direction in which the error drops fastest - so every new tree walks exactly where the walking pays off most. Happily, you do not need any of that mathematics in order to use boosting. What you need is the intuition of the queue, and the best way to build that intuition is with your own hands.

Boosting by hand: two stumps and a list of mistakes

We are going to do boosting manually, on six lions of known age and known weight, so that you can watch the mechanism work without any library beyond the one you already know. We will use

DecisionTreeRegressor

from scikit-learn - a decision tree in the flavor that predicts numbers, a close cousin of the

DecisionTreeClassifier

from the scikit-learn lesson. The

max_depth=1

parameter limits it to a single question, so we get precisely the stump we have been talking about. The methods are old friends:

fit

teaches,

predict

predicts.

The first tracker receives the raw data: age in years and weight in kilograms. In a moment you will see what it predicted and how badly it missed.

1import numpy as np
2from sklearn.tree import DecisionTreeRegressor
3
4age = np.array([[1], [2], [3], [4], [5], [6]])
5weight_kg = np.array([40.0, 70.0, 100.0, 150.0, 170.0, 190.0])
6
7stump_1 = DecisionTreeRegressor(max_depth=1)
8stump_1.fit(age, weight_kg)
9
10predicted = stump_1.predict(age)
11residuals = weight_kg - predicted
12
13print(predicted)                       # [ 70.  70.  70. 170. 170. 170.]
14print(residuals)                       # [-30.   0.  30. -20.   0.  20.]
15print(np.abs(residuals).sum())         # 100.0

The single question the stump asked was "is this lion younger than three and a half". It gave every youngster 70 kg and every elder 170 kg - and that really is everything a model of depth one can do. The second line is where it gets interesting. It gave the one-year-old 30 kg too many, it shortchanged the three-year-old by 30 kg, and the four-year-old by 20 kg. The sum of the absolute misses is 100 kg, and that is our reference point from here on. Notice that the

weight_kg

array was not touched in any way - we computed the residuals alongside it, with an ordinary array subtraction in NumPy, exactly the way you were subtracting arrays back in the numerical computing module.

Now comes the single most important move in this lesson. The second stump receives the same ages, but in place of the weights it receives the list of the first stump's mistakes. It does not learn how much a lion weighs. It learns where its predecessor missed, and in which direction.

1stump_2 = DecisionTreeRegressor(max_depth=1)
2stump_2.fit(age, residuals)
3
4correction = stump_2.predict(age)
5combined = predicted + correction
6
7print(correction)                          # [-30.   6.   6.   6.   6.   6.]
8print(combined)                            # [ 40.  76.  76. 176. 176. 176.]
9print(np.abs(weight_kg - combined).sum())  # 76.0

The second stump noticed that one specimen sticks out further than all the others, and asked "is this the yearling". From the yearling it subtracts 30 kg, and to everyone else it adds 6 kg. Once you add the two predictions together, the total error has dropped from 100 kg to 76 kg - and all we did was add one extra question.

Now the most valuable sentence in this section: the first stump was not modified and not retrained. It still predicts 70 and 170, exactly as it did a minute ago. Neither

age

nor

weight_kg

changed either - the input data is untouched. The only thing that changed is that a second model now stands next to the first one, and that the answer is now the sum of their predictions. The second stump never saw the real weights of any lion. All it ever saw was somebody else's mistakes. This is precisely what XGBoost does, except that instead of two trees it lines up a hundred or a thousand, and before adding each correction it multiplies that correction by a small number - the

learning_rate

. Had our correction entered with a weight of 0.1, it would have been ten times more cautious, and you would need many such trees to arrive anywhere. That is exactly why the defaults in these libraries are hundreds of shallow trees rather than three deep ones: small steps are harder to overshoot with.

What gradient boosting is not

The name sounds mysterious, so a handful of misunderstandings circle around it. Let us settle them now, before we go anywhere near the real libraries.

It is not a type of neural network. There is not a single neuron in boosting, no layer, no connection weight, and no backpropagation either. There are decision trees, meaning chains of yes-no questions, lined up in a row. You will meet neural networks in the next lesson, with PyTorch, and you will see that they are a completely different construction - and that on tabular data they are usually weaker than boosting.

It is not a method of randomly selecting features. Randomly drawing features and samples is the trick behind the random forest: there, every tree gets a different slice of the data precisely so that the trees stay independent of one another. Boosting does have

subsample

and

colsample_bytree

parameters, which let you randomly restrict the specimens and the columns handed to each tree, but that is only an anti-overfitting extra, not the definition of the technique. Random drawing makes use of nobody's errors, whereas the whole of boosting rests on each tree seeing the mistakes of the one before it.

And it is finally not a clustering algorithm. Clustering is unsupervised learning: there is no

at all, the algorithm groups similar specimens by itself, and nobody tells it what they are called. Boosting is supervised learning in its purest form - without a column of correct answers you cannot compute a single residual, so there is nothing left to correct. Remember from the supervised learning lesson that everything starts with measurement-answer pairs? Here that rule holds doubly.

Three libraries, one idea

Three teams implemented the same idea independently, and that is how we ended up with the three libraries this lesson is about. XGBoost is the oldest and the most widely recognized, the default choice to start from. LightGBM, from Microsoft, was written with speed on large datasets in mind. CatBoost, from Yandex, specializes in text columns, which is to say categorical ones. All three speak the scikit-learn verbs you already know -

fit

predict

score

- so once you master one, you will recognize the other two on sight.

None of them ships as part of scikit-learn, so they have to be installed separately. A single command in the terminal takes care of all three at once.

1pip install xgboost lightgbm catboost

The installation downloads pre-compiled wheels, so it takes a moment and does not require a compiler on your machine. Note that the names you install with do not always match the names you import with: the

lightgbm

package is imported as

lgb

, and from the

catboost

package we pull in the

CatBoostClassifier

class. This is a common source of confusion on the first run. If on macOS you see a message about a missing

libomp

library, install it with

brew install libomp

- XGBoost uses it to spread the work across multiple cores.

The field notebook we will be working on

We need data. We are going to build it ourselves, using the NumPy random number generator you know from the numerical computing module:

np.random.default_rng(42)

creates a generator with a fixed seed, so that the same numbers come out on every run. The

normal

method draws measurements around a given mean with a given spread - just like in nature, where no two lions weigh exactly the same. We put the results into a Pandas frame, because named columns will make the feature importance charts readable later. The

pd.concat

function glues frames one below another, and

np.repeat

produces the label column: two hundred zeros, two hundred ones and two hundred twos.

One column has been slipped in there on purpose.

hour_of_sighting

is the hour at which the tracker took the photograph - drawn at random for every species, so it carries no information whatsoever about who is in the picture. I am leaving it in deliberately, because in a moment we are going to check whether the model notices.

1import numpy as np
2import pandas as pd
3
4rng = np.random.default_rng(42)
5
6def herd(n, weight, speed, tail, bite):
7    return pd.DataFrame({
8        "weight": rng.normal(weight, weight * 0.30, n),
9        "speed": rng.normal(speed, speed * 0.25, n),
10        "tail_length": rng.normal(tail, tail * 0.30, n),
11        "bite_force": rng.normal(bite, bite * 0.35, n),
12        "hour_of_sighting": rng.uniform(5, 19, n),
13    })
14
15X = pd.concat([
16    herd(200, 190, 58, 85, 650),      # lion
17    herd(200, 55, 110, 75, 475),      # cheetah
18    herd(200, 60, 65, 30, 1100),      # hyena
19], ignore_index=True)
20
21y = np.repeat([0, 1, 2], 200)         # 0 = lion, 1 = cheetah, 2 = hyena
22
23print(X.shape, y.shape)               # (600, 5) (600,)

Six hundred specimens described by five numbers, and six hundred labels - the shapes confirm it through

shape

. The species deliberately overlap: a hyena weighs barely more than a cheetah, and a heavy lion's bite reaches well into hyena territory. No single measurement pulls all three apart, and that is exactly what makes this good material for boosting. Notice that

is a Pandas frame while

is a plain NumPy array - all three libraries accept that combination without any conversion, because underneath they read the numeric values anyway.

XGBoost - four steps, always in the same order

Working with XGBoost takes four steps, and their order is not a matter of taste.

Step one: import XGBClassifier. There is nothing to build a model out of until the library has been loaded. Step two: define the hyperparameters, the numbers describing what the model should look like - you type them into the constructor at the moment the object is created, so you have to know them beforehand. Step three: train the model with

fit

, because only now does the model get to look at the data. Step four: evaluate feature importance, which can only be read out of a trained model - before

fit

there is nothing to ask about, because no tree exists yet.

Let us begin with step one. The import has four pieces: the keyword

import

, the package name

xgboost

, the word

as

and the alias

xgb

, which we will use from here on. That alias is the convention adopted throughout the documentation, so stick to it. While we are at it, we split the data into a training part and a test part with

train_test_split

, which you met in the supervised learning lesson:

test_size=0.2

sets every fifth specimen aside for the exam,

random_state=42

makes the split reproducible, and

stratify=y

makes sure the species proportions are the same in both parts.

1import xgboost as xgb
2from sklearn.model_selection import train_test_split
3
4X_train, X_test, y_train, y_test = train_test_split(
5    X, y, test_size=0.2, random_state=42, stratify=y
6)
7
8print(X_train.shape, X_test.shape)    # (480, 5) (120, 5)

Four hundred and eighty animals go to training, one hundred and twenty stay locked up for the exam. Note that

train_test_split

returns four objects in a fixed order: first the two halves of the features, then the two halves of the labels. Mixing that order up is the classic mistake after which the model trains on labels instead of measurements. The

frame itself was not disturbed - the split returned new objects, and the original is still lying there untouched.

Step two is the hyperparameters. The

XGBClassifier

class accepts several dozen of them, but three decide almost everything.

n_estimators

is the length of the queue, meaning how many trees will line up one after another.

max_depth

is the depth of a single tree, meaning how many questions in a row it may ask; the bigger it is, the stronger the individual tree, but also the easier it is to overfit.

learning_rate

is the weight with which each successive correction enters the sum - the very same number we discussed with the stumps. On top of that, two safety parameters:

subsample=0.8

gives each tree a random 80 percent of the specimens, and

colsample_bytree=0.8

a random 80 percent of the columns. And finally

random_state=42

, because without a fixed seed that random drawing would give you a different result on every run.

1model = xgb.XGBClassifier(
2    n_estimators=100,        # length of the queue of trees
3    max_depth=6,             # how many questions one tree may ask
4    learning_rate=0.1,       # how strongly each correction enters
5    subsample=0.8,           # percent of specimens per tree
6    colsample_bytree=0.8,    # percent of columns per tree
7    random_state=42,
8)
9
10print(model.n_estimators, model.max_depth)    # 100 6

The model exists, but it has not seen a single animal yet - the constructor stored nothing but the settings. You can read them back as ordinary object attributes, which is exactly what we just did, and that is the entire content of the model at this stage. If you called

predict

right now, you would get an exception, because there is not one tree inside it.

Step three, and the shortest: training.

fit

builds the whole queue of trees, one after another, each on the residuals of the ones before. Then

score

computes the percentage of hits on the test set - the same accuracy metric you met in the supervised learning lesson.

predict

returns species numbers for new specimens.

1model.fit(X_train, y_train)
2
3print(f"Accuracy: {model.score(X_test, y_test):.2%}")
4print(model.predict(X_test)[:8])

The accuracy will land somewhere around 97 percent - I am not quoting an exact figure, because it depends on the library version, so run the code and check it on your own machine. What matters more is what

fit

did not do. It did not return a new model for you to assign to a variable; it modified the

model

object in place, exactly as scikit-learn does. It did not touch

X_train

y_train

either - a hundred trees moved in inside the model, and your data is lying exactly where it was. And one more thing: the queue is built once and for all. Calling

fit

again does not append trees to the existing ones, it rebuilds everything from scratch.

When to stop adding trees

We typed in a hundred trees off the top of our heads, and nobody guaranteed that a hundred is the right number. With too short a queue the model will not have time to learn; with too long a queue it will start learning the noise in the training data. The answer is early stopping: you set the queue up with plenty of slack, and after every tree XGBoost checks the score on a validation set and breaks off as soon as a given number of rounds passes without improvement. That number is given by the

early_stopping_rounds

parameter - and since XGBoost 2.0 it goes into the constructor rather than into

fit

, which is a frequent source of outdated examples online.

We need a set for the model to check itself against, and here comes the trap: it must not be the test set. If XGBoost picked the number of trees while looking at the exam, that exam would stop being honest - the model would tune itself to the very answers it is later supposed to guess. So we cut a slice off the training data as a separate validation set. That validation set is passed to

fit

through the

eval_set

parameter, as a list of pairs. The

verbose=False

parameter silences the per-tree report, and

best_iteration

tells you afterwards which tree the model stopped on.

1X_fit, X_val, y_fit, y_val = train_test_split(
2    X_train, y_train, test_size=0.2, random_state=42, stratify=y_train
3)
4
5model_es = xgb.XGBClassifier(
6    n_estimators=500,
7    max_depth=6,
8    learning_rate=0.1,
9    early_stopping_rounds=10,
10    random_state=42,
11)
12model_es.fit(X_fit, y_fit, eval_set=[(X_val, y_val)], verbose=False)
13
14print("Best tree:", model_es.best_iteration)
15print(f"Accuracy: {model_es.score(X_test, y_test):.2%}")

We ordered five hundred trees and the model stopped far earlier, without losing any accuracy - the exact number depends on the version, so once again I am sending you off to run it yourself. Pay attention to one thing that did not change: the test set took no part in any of this. Neither

fit

nor the stopping mechanism ever saw it, so the number from

score

is still an honest exam. Notice as well that

X_fit

came out of splitting

X_train

, not out of splitting the whole dataset - the exam was set aside earlier and remains intact.

Which measurements actually do the work

Step four of the ritual. The model is trained, so we can finally ask what it based its decisions on. The

feature_importances_

attribute - with a trailing underscore, like everything scikit-learn computes during training - returns one number per column, and they all add up to one. The higher the number, the more often and the more effectively that measurement separated the species inside the trees. The order of the numbers matches the order of the columns in

X_train

, so we pair them up with the

zip

function you know from the Python basics.

1for name, importance in zip(X_train.columns, model.feature_importances_):
2    print(f"{name:>20}: {importance:.3f}")

At the very bottom of the list you will find

hour_of_sighting

with a value close to zero - exactly the column we slipped in on purpose. The model worked out for itself that the time of the photograph says nothing about the species, and it barely used the column at all. That is the practical value of this step: you get a hint about which measurements are not worth collecting in the field in the first place. Be careful with the interpretation, though - a high importance says the model used the column, not that the column causes anything. If a cage number assigned in increasing order by species crept into the notebook, it would land at the top of this list, and it would mean nothing beyond the fact that the tracker cataloged the animals one species at a time.

A list of numbers is easier to compare by eye than in print, which is why XGBoost ships a plotting function of its own. The call has six pieces: the library alias

xgb

, a dot, the function name

plot_importance

, an opening parenthesis, the trained

model

, and a closing parenthesis. Nothing more is needed - the function pulls the column names out of the model itself and sorts the bars for you. We display the chart with

plt.show

, the command from the visualization module.

1import matplotlib.pyplot as plt
2
3xgb.plot_importance(model)
4plt.show()

You get a horizontal bar chart with the most important feature on top. If you have several dozen features, add the

max_num_features

argument to trim the chart down to the best dozen or so - with five columns there is no point. A small surprise at the end: by default

plot_importance

measures importance differently from

feature_importances_

, because it shows how many times a given column appeared in the trees' questions at all. The ordering can therefore come out slightly different from the printout above, and that is not a bug - it is a different measure of the same phenomenon.

LightGBM - the same idea on a faster machine

XGBoost copes beautifully for as long as the notebook fits in memory. Once the expedition grows to millions of rows - and that is what working with sensor readings or frames from hundreds of camera traps looks like - training time starts to hurt, all the more so because during tuning you train the model dozens of times over. That is when people reach for LightGBM, which on very large datasets is usually faster than XGBoost.

Where does that advantage come from? From two design decisions. First, LightGBM converts continuous measurements into histogram buckets, so instead of considering every possible threshold value it considers a few hundred buckets - and it does that once, not for every tree. Second, it grows trees leaf-wise: instead of filling in an entire level evenly, it picks the leaf that will reduce the error most and expands only that one. As a result, for the same compute budget it drives the error down harder.

Two things have to be said outright here, because a false conclusion is easy to draw. It is not true that both libraries are equally fast - on large datasets the difference can be several-fold and is easy to measure. Nor is it true that either of them cannot handle large data: both were built for large data in the first place, and both can work on millions of rows. The point is simply that on very large datasets LightGBM usually gets there sooner. Do not try to measure this on our six hundred animals - at that size the startup costs dominate and the stopwatch will hand you a random winner. LightGBM's advantage only shows up at hundreds of thousands and millions of rows.

The class is called

LGBMClassifier

and it takes the same three basic hyperparameters as XGBoost. Two of its own join them, both consequences of leaf-wise growth.

num_leaves

caps the number of leaves in a tree, and it is this parameter, not

max_depth

, that is the main complexity dial here.

min_child_samples

says how many specimens must land in a leaf at minimum for it to be allowed to exist - a safeguard against leaves built on two accidental animals. I am also throwing in

verbose=-1

so that the library does not bury the console under training messages.

1import lightgbm as lgb
2
3model_lgb = lgb.LGBMClassifier(
4    n_estimators=100,
5    max_depth=6,
6    learning_rate=0.1,
7    num_leaves=31,          # maximum number of leaves in a tree
8    min_child_samples=20,   # min specimens needed to form a leaf
9    random_state=42,
10    verbose=-1,
11)
12model_lgb.fit(X_train, y_train)
13
14print(f"LightGBM: {model_lgb.score(X_test, y_test):.2%}")

The result will be very close to XGBoost, and that is exactly as it should be - it is the same algorithm, implemented differently. Look at the code once more and see what did not change:

fit

took precisely the same

X_train

and

y_train

score

computed precisely the same metric, and the names

n_estimators

max_depth

and

learning_rate

read identically. One import and one class name changed. This is that shared language we talked about with scikit-learn: switching between boosting libraries costs you two lines. LightGBM also has its own

plot_importance

inside the

lgb

module, working just like the XGBoost version.

CatBoost - when the notebook holds words

Both XGBoost and LightGBM read numbers and nothing else. A real field notebook, meanwhile, is full of words: habitat is "savanna", "bush" or "river", diet is "meat" or "carrion". So far you have dealt with this by encoding -

OneHotEncoder

turned every value into a separate column of zeros and ones. With three habitats that is nothing to worry about, but with a column holding five hundred plant species you get five hundred new columns and the model starts to drown in them.

CatBoost solves the problem differently: it takes text columns as they are and converts them to numbers itself, computing a statistic from the labels for each category - and it does so in a way that is resistant to peeking at the answers. That is why it is the default choice when categories dominate the data.

So let us add two text columns to our notebook. In seventy percent of cases the habitat will agree with the species, and in the rest it will come out at random - a notebook like that is closer to the truth than a perfect one. Diet will separate the hyenas, since they are the ones that live on carrion. To choose between two arrays we use

np.where

, which you know from the NumPy module: it takes a condition and two values, and picks one of them for each element.

1rng_cat = np.random.default_rng(7)
2terrain = np.array(["savanna", "bush", "river"])
3
4matching = rng_cat.random(len(y)) < 0.7
5random_pick = rng_cat.integers(0, 3, len(y))
6
7X_cat = X.copy()
8X_cat["habitat"] = terrain[np.where(matching, y, random_pick)]
9X_cat["diet"] = np.where(y == 2, "carrion", "meat")
10
11print(X_cat[["habitat", "diet"]].head(3))

The frame now has seven columns: five numeric and two textual. What matters is what did not change -

X.copy()

made a copy, so the original

still has five columns and the earlier models know nothing about this addition. Had we skipped

copy

, we would have appended the columns to the very frame we trained XGBoost on, and that earlier code would fall over on the next run.

Now the model itself. The

CatBoostClassifier

class uses its own names for two concepts you already know:

iterations

is the length of the queue of trees, the equivalent of

n_estimators

, and

depth

is the tree depth, the equivalent of

max_depth

. The crucial one is

cat_features

: a list of the names of the columns that are to be treated as categorical. And here is the most common beginner's mistake - passing column names only works if you train on a Pandas frame that actually contains those columns. If you pass names but hand over a NumPy array, CatBoost stops with an error saying that numeric data cannot be treated as categorical. That is why we split

X_cat

and not

. The

verbose=False

parameter silences the per-iteration report, and

random_seed

plays the role that

random_state

plays in the other libraries.

1from catboost import CatBoostClassifier
2
3Xc_train, Xc_test, yc_train, yc_test = train_test_split(
4    X_cat, y, test_size=0.2, random_state=42, stratify=y
5)
6
7model_cat = CatBoostClassifier(
8    iterations=100,
9    depth=6,
10    learning_rate=0.1,
11    cat_features=["habitat", "diet"],
12    verbose=False,
13    random_seed=42,
14)
15model_cat.fit(Xc_train, yc_train)
16
17print(f"CatBoost: {model_cat.score(Xc_test, yc_test):.2%}")

The model trained on text columns without a single line of encoding on our side - and that is the whole promise of CatBoost. Notice that we made the split with the same

random_state=42

and the same

stratify

, so exactly the same animals ended up in the test set as before. That is what makes the comparison with XGBoost meaningful: the difference comes from the model and from the added columns, not from a different draw.

CatBoost exposes feature importance through a method rather than an attribute - it is called

get_feature_importance

and it returns an array of numbers in column order. The scale is different from XGBoost, because the values add up to a hundred, so you read them straight off as percentages.

1for name, importance in zip(Xc_train.columns, model_cat.get_feature_importance()):
2    print(f"{name:>20}: {importance:.2f}")

On that list you will see numeric and textual columns side by side, priced in the same currency -

habitat

and

diet

take their place in the ranking like any other measurement. Diet lands right at the top, because it flags hyenas without a single miss, and habitat lands low, because it lies in one case out of three. Draw a conclusion from this that reaches further than CatBoost itself: one well-chosen categorical column can be worth more than four readings off a measuring tape. And note the parentheses in

get_feature_importance()

: it is a method, so it has to be called, unlike

feature_importances_

in XGBoost, which is a plain attribute and takes no parentheses at all.

Hyperparameter tuning - who is going to click through all of this

Let us come back to the numbers we typed in off the top of our heads. Why should

max_depth

be six rather than three? Why is

learning_rate

0.1 and not 0.3? The honest answer is: nobody knows. Those values depend on the data, and the only trustworthy way to choose them is to check. By hand that means dozens of runs, results scribbled on a scrap of paper, and a mistake somewhere in the middle.

That is what GridSearchCV from the

sklearn.model_selection

module is for. Its one and only job is automatic tuning of a model's hyperparameters: you hand it a grid of values to check, it trains the model on every combination and returns the best one. Let us say straight away what GridSearchCV is not, because the name gives nobody a clue. It is not for data visualization - charts are the business of matplotlib and seaborn from the visualization module, and GridSearchCV draws nothing at all. It is not for data cleaning - you removed gaps and duplicates with Pandas long before training, and here the data has to be clean already. And it certainly does not create neural networks - networks are an entirely different family of models, which you will meet with PyTorch; GridSearchCV can at most pick hyperparameters for them, but it will never build one.

Here is how it works.

param_grid

is a dictionary in which the key is a hyperparameter name and the value is the list to check.

cv=3

is the number of cross-validation folds you met in the supervised learning lesson - every combination is scored three times, on different splits, so that the result is not a lottery.

scoring="accuracy"

says which metric we measure quality with, and

n_jobs=-1

lets it use every core of the processor. After training,

best_params_

returns the winning set,

best_score_

its average score across the folds, and

best_estimator_

a ready, already trained model.

1from sklearn.model_selection import GridSearchCV
2
3param_grid = {
4    "max_depth": [3, 6],
5    "learning_rate": [0.05, 0.1, 0.3],
6    "n_estimators": [50, 100],
7}
8
9grid = GridSearchCV(
10    xgb.XGBClassifier(random_state=42),
11    param_grid,
12    cv=3,
13    scoring="accuracy",
14    n_jobs=-1,
15)
16grid.fit(X_train, y_train)
17
18print(len(grid.cv_results_["params"]))    # 12
19print(grid.best_params_)
20print(f"Best score: {grid.best_score_:.2%}")
21
22best_model = grid.best_estimator_

Twelve combinations, because two times three times two - and that is a number worth working out in your head before you press run. With three folds it means thirty-six trainings. Had you added a fourth parameter with four values and raised

cv

to five, you would be at two hundred and forty, because the grid grows as a product, not as a sum. Note one more thing: we pass the model to

GridSearchCV

before training and we never call

fit

on it ourselves. The

grid

object trains all the copies for us, and the finished winner is waiting in

best_estimator_

and needs no retraining.

Since the grid grows that fast, there is an alternative: instead of checking every combination, draw a fixed number of them at random. That is what

RandomizedSearchCV

from the same module does, and instead of lists of values it takes random distributions from

scipy.stats

randint(3, 10)

draws an integer from 3 to 9 inclusive, whereas

uniform

has surprising parameters: the first is the start of the range, but the second is the width, not the end. So

uniform(0.01, 0.29)

draws from the interval 0.01 to 0.30 - one of the most common slips in tuning. The

n_iter

parameter says how many combinations to draw, and it is that number, not the size of the grid, that decides how long you wait.

1from sklearn.model_selection import RandomizedSearchCV
2from scipy.stats import randint, uniform
3
4param_dist = {
5    "max_depth": randint(3, 10),
6    "learning_rate": uniform(0.01, 0.29),
7    "n_estimators": randint(50, 300),
8}
9
10random_search = RandomizedSearchCV(
11    xgb.XGBClassifier(random_state=42),
12    param_dist,
13    n_iter=20,
14    cv=3,
15    random_state=42,
16    n_jobs=-1,
17)
18random_search.fit(X_train, y_train)
19
20print(random_search.best_params_)

Twenty draws instead of an exhaustive sweep, and the result usually does not fall far behind the best of the grid. I am not quoting the values it finds, because they depend on the draw and on the library version - and that is the more important lesson here than any particular number: with random search,

random_state

stops being cosmetic and becomes the condition for repeating your experiment at all.

My recommendation, @name, is unambiguous: start with

RandomizedSearchCV

over wide ranges, and only reach for
GridSearchCV
at the end, to polish a narrow band around the winner. The reason is simple - most hyperparameters change very little, so an exhaustive sweep burns an enormous share of its time on combinations that were going to lose anyway. Random search in twenty attempts usually lands closer to the optimum than a grid of the same cost, because it tries more distinct values of each individual parameter.

When even that stops being enough, you will reach for the Optuna library. What sets it apart is that it does not draw blindly: it remembers the results of previous attempts and picks the next combinations where it expects an improvement. In it you define an objective function that returns a model score for a given set of parameters, and Optuna calls that function hundreds of times, narrowing the search on its own. That is a tool for later - master the two above first, because in nine cases out of ten they will do.

Take one thing away from this lesson, @name: a random forest is a council of trackers who do not listen to one another, while boosting is a caravan in which each one follows the trail of the previous one's mistakes - and that is why it gets further, even though each member on its own sees less.

Go to CodeWorlds→