Gradient Boosting Libraries (XGBoost, LightGBM & CatBoost)

Introduction

Ask most winners of Kaggle competitions on structured, tabular data what algorithm they used, and the answer is almost always some flavor of gradient boosting. Scikit-learn has its own GradientBoostingClassifier, but three specialized libraries — XGBoost, LightGBM, and CatBoost — have become the industry standard because they are faster, more accurate, and more feature-rich.

This lesson explains what gradient boosting is, then walks through each of the three libraries with an installation command and a short working example.

What is Gradient Boosting?

Gradient boosting builds a model as a sequence of small decision trees, where each new tree is trained specifically to correct the mistakes (residual errors) of the trees before it. The trees are combined into one strong ensemble model that is usually far more accurate than any single tree, and often more accurate than a random forest of independently trained trees.

All three libraries in this lesson implement this same core idea, but differ in speed, memory use, and how they handle things like categorical features and missing values.

XGBoost

XGBoost (Extreme Gradient Boosting) was the library that popularized gradient boosting for competitive machine learning. It is highly optimized, supports regularization to reduce overfitting, and runs on both CPU and GPU.

pip install xgboost

import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42
)

model = xgb.XGBClassifier(n_estimators=100, learning_rate=0.1, random_state=42)
model.fit(X_train, y_train)

predictions = model.predict(X_test)
print(f"XGBoost accuracy: {accuracy_score(y_test, predictions):.2%}")

LightGBM

LightGBM, developed by Microsoft, uses a histogram-based approach to decide where to split trees: instead of evaluating every possible split point, it buckets continuous values into discrete bins first. This makes it noticeably faster and more memory-efficient than XGBoost on large datasets, with comparable accuracy.

pip install lightgbm

import lightgbm as lgb
from sklearn.metrics import accuracy_score

model = lgb.LGBMClassifier(n_estimators=100, learning_rate=0.1, random_state=42)
model.fit(X_train, y_train)

predictions = model.predict(X_test)
print(f"LightGBM accuracy: {accuracy_score(y_test, predictions):.2%}")

CatBoost

CatBoost, developed by Yandex, is built around one standout feature: it handles categorical features natively. With XGBoost or LightGBM you typically need to one-hot or label-encode text categories yourself first; CatBoost accepts them directly and often gets better accuracy with less preprocessing code.

pip install catboost

from catboost import CatBoostClassifier
from sklearn.metrics import accuracy_score

model = CatBoostClassifier(iterations=100, learning_rate=0.1, verbose=0, random_state=42)
model.fit(X_train, y_train)

predictions = model.predict(X_test)
print(f"CatBoost accuracy: {accuracy_score(y_test, predictions):.2%}")

Comparing the Three

Common Mistakes

Best Practices

Frequently Asked Questions

Key Takeaways

Summary

XGBoost, LightGBM, and CatBoost are the three dominant gradient boosting libraries in modern data science, each with its own edge in speed, feature handling, or maturity. On structured, tabular data they frequently outperform both plain scikit-learn models and, for many problems, even deep learning.