Random Forest vs XGBoost

A random forest is a collection of decision trees, but the trees are trained mostly independently and then combined.

A single decision tree

Start with tabular features:

age = 32
income = 140000
years_experience = 8
embedding_similarity = 0.82

The tree repeatedly asks questions:

embedding_similarity > 0.75?
├── yes
│   years_experience > 5?
│   ├── yes → leaf
│   └── no  → leaf
└── no
    income > 100000?
    ├── yes → leaf
    └── no  → leaf

The leaf contains a prediction.

For classification, a tree might output a class or class probability:

Tree 1 → P(relevant) = 0.9
Tree 2 → P(relevant) = 0.8
Tree 3 → P(relevant) = 0.3
Tree 4 → P(relevant) = 0.7
Tree 5 → P(relevant) = 0.4

The forest averages the probabilities (or majority-votes the hard labels):

So the forest predicts relevant.

For regression, every tree directly outputs a number, and the forest averages them:

Why many trees

500 identical trees on identical data would be redundant. Random forests make each tree different in two ways.

First, each tree gets a bootstrap sample of the training data, randomly sampled rows with replacement:

Original:        A B C D E
Tree 1 training: A A C D E
Tree 2 training: B B C E E
Tree 3 training: A B B D E

Second, at each split, the tree only considers a random subset of features. If you have 100 features, a node might only be allowed to consider 10 randomly selected ones.

That creates a diverse population of trees. The intuition:

which reduces variance and overfitting.

Random forest vs XGBoost

Random Forest:

Gradient Boosting / XGBoost:

Both ultimately map:

Both are workhorses for L1 reranking in a candidate retrieval system: cheap tree models over a handful of features (BM25 score, embedding similarity, recency) cutting ~1000 candidates to ~100 before the expensive cross-encoder.