skip to content

How do Gini impurity and entropy differ as split criteria for a classification tree?

level: juniorimportance: must knowfreq 74%

answer

  1. both are zero at a pure node
  2. one squares proportions, one takes logs
  3. binary maxima: 0.5 versus 1 bit
  4. rankings almost always coincide
  5. Gini is entropy's first-order approximation

basics

~20 s

Gini impurity is 1 - sum(p_i^2) and entropy is -sum(p_i * log2 p_i). Both are zero at a pure node and largest when classes are evenly mixed, and they rank splits so similarly that the choice rarely changes the tree.

solid answer

~50 s

Both measure how mixed a node's labels are, and both are used the same way: score each candidate split by the size-weighted impurity of its children and take the biggest drop. Gini is `1 - sum(p_i^2)`; for a two-class node that is `2p(1-p)`, peaking at `0.5` when the classes are even. Entropy is `-sum(p_i * log2 p_i)`, peaking at `1` bit for two even classes, and the drop it produces is called information gain. The curves have the same shape, so they usually select the same split; entropy is a touch steeper near pure nodes and costs a logarithm per class, Gini needs only a square. Their gains are on different scales, so you rank within one criterion and never compare across. In practice depth, minimum leaf size and the amount of data dominate the criterion choice completely.

go deeper

for a junior

Know both formulas cold and what they mean: zero for a pure node, biggest for an even mix. Be able to compute Gini for a 60/40 node in your head and get 0.48.

for a middle

Explain the shape argument - both concave, both symmetric, Gini a first-order approximation of entropy in nats - and state the binary maxima of 0.5 and 1 bit without hesitating.

for a senior

Show judgment about where effort belongs: the criterion is nearly a non-decision, while depth, leaf size and data volume decide whether the tree generalises. Note that gains from different criteria are not comparable.

for a principal

Frame it as a cost question. Argue when a logarithm per class per candidate is worth paying, and push back on teams that treat criterion selection as a tuning knob when the real variance is in how the tree is grown and stopped.

## Two ways to say "this node is mixed" A classification tree needs a number that is **zero when every row in a node shares one label** and **largest when the labels are spread evenly**. Two such numbers are in universal use. **Gini impurity**: `I_G = 1 - sum(p_i^2)`, where `p_i` is the proportion of class `i` in the node. Read it as the probability that you misclassify a randomly drawn row if you label it by drawing a second row at random from the same node. For two classes it simplifies to `2p(1-p)`, which is `0` at `p = 0` or `p = 1` and peaks at `0.5` when `p = 0.5`. With `k` even classes the maximum is `1 - 1/k`, so it never reaches 1. **Entropy**: `I_H = -sum(p_i * log2 p_i)`, measured in bits - the average number of bits needed to encode the label of a row drawn from the node. For two classes it peaks at exactly `1` bit at `p = 0.5`; with `k` even classes it peaks at `log2(k)`. Both are plugged into the same machinery: `gain = I(parent) - (n_L/n)*I(left) - (n_R/n)*I(right)`. When the impurity is entropy, that gain has its own name, **information gain**. ## A worked comparison Take a factory visual-inspection node: 100 units, 60 pass and 40 fail. The parent has `I_G = 1 - 0.36 - 0.16 = 0.48` and `I_H = 0.971` bits. **Candidate A** cuts the node into two halves of 50: one child at 90% pass (45/5), the other at 30% pass (15/35). - Gini: children `0.18` and `0.42`, weighted `0.30`, so the Gini gain is `0.48 - 0.30 = 0.18`. - Entropy: children `0.469` and `0.881` bits, weighted `0.675`, so information gain is `0.971 - 0.675 = 0.296` bits. **Candidate B** cuts off 80 rows at 65% pass and 20 rows at 40% pass. - Gini: children `0.455` and `0.48`, weighted `0.46`, gain `0.02`. - Entropy: children `0.934` and `0.971` bits, weighted `0.942`, gain `0.030` bits. Both criteria prefer A, by a wide margin, and neither `0.18` nor `0.296` is "bigger" in any meaningful sense - they are measured on different scales. That is the whole story in miniature: the ranking agrees, the units do not. ## Why they agree so often Write entropy in nats instead of bits: `-sum(p_i * ln p_i)`. Expand `-ln(p)` around `p = 1` and keep the first term: `-ln(p) is approximately (1 - p)`. Substituting gives `sum(p_i * (1 - p_i)) = 1 - sum(p_i^2)`, which is exactly Gini. **Gini is the first-order Taylor approximation of entropy.** Two impurity functions that are that close, both concave, both symmetric in the classes, both zero at the corners and maximal at the centre, will order candidate splits the same way except in finely balanced cases. Empirical studies of this question find disagreement in only a small minority of splits, and where they do disagree the two resulting trees usually perform indistinguishably after pruning. Where a difference is visible at all: entropy is steeper as a node approaches purity, so it is marginally more willing to pay for splits that clean up an already-good node; Gini is often described as more inclined to isolate the largest class into its own branch. These are tendencies, not rules, and they are swamped by tree depth and sample size. ## Which algorithms use which The CART family grows binary trees and uses Gini for classification by default (and variance for regression). The ID3 line uses information gain, and its successor C4.5 uses **gain ratio**, which divides information gain by the entropy of the *split itself* - a correction aimed at multi-valued features, which naturally produce many small children and therefore inflated raw gain. ## Practical guidance - Do not tune the criterion first. If a tree overfits, the depth and leaf-size controls are the lever; swapping Gini for entropy will not save it. - Do not compare gains across criteria, and do not report an impurity drop as if it were an accuracy improvement. - Do remember the two maxima - `0.5` and `1` bit for a balanced binary node - because interviewers use them as a quick sanity check on whether you have actually computed either quantity. - Entropy costs a logarithm per class per candidate; on very large candidate sets that is a real, if modest, training-time difference.

  • Why do the two criteria almost never disagree on which split to take?
    Because they are nearly the same function. Take entropy in nats and approximate -ln(p) by (1 - p); the result is exactly 1 - sum(p_i^2), Gini. Both are concave, symmetric across classes, zero at purity and maximal at the uniform mix, so they order candidate splits identically unless two candidates are almost tied. Where they do differ, the fitted trees usually score the same after pruning.
  • Can you compare a Gini gain of 0.18 against an information gain of 0.30 bits?
    No. They are drops in two different impurity functions with different units and different maxima - 0.5 for binary Gini, 1 bit for binary entropy. A gain is only meaningful as a ranking device within one criterion at one node. Comparing across criteria, or across nodes with different sample sizes, is a category error.
  • What does C4.5's gain ratio add on top of plain information gain?
    It divides the information gain by the entropy of the split itself - how evenly the rows are spread across the branches. Raw information gain rises mechanically when a feature carves the node into many small children, so gain ratio penalises splits whose apparent gain comes from fragmentation rather than from real separation of the classes.

saying these in an interview costs you the question

  • Claims entropy reliably produces a more accurate tree
  • Confuses Gini impurity with the Gini coefficient of inequality
  • Says a balanced two-class node has Gini impurity 1
  • Defines information gain as the children's entropy alone
  • Tunes the criterion instead of depth to fix overfitting

context

resolving links…