How does automatic differentiation differ from symbolic and numerical differentiation?

## Three different products Ask for 'the derivative' and you can get three different things. **Symbolic differentiation** takes an expression and returns another expression. Give it `f(x) = x * sin(x)` and it returns `sin(x) + x * cos(x)`, valid for every `x`. That generality is genuinely useful, and where a closed form exists and stays small it is the best of the three. **Numerical differentiation** takes a function you can only call, and estimates the derivative by perturbing the input: the forward difference `(f(x+h) - f(x)) / h`, or the central difference `(f(x+h) - f(x-h)) / (2h)`. It needs no access to the internals at all. **Automatic differentiation** takes the *program* and returns derivative *values* at the input you supplied. It is exact in the same sense the program is exact — up to floating-point rounding — but it produces numbers, not a formula. ## Why symbolic differentiation breaks down on deep compositions A network is a deep composition `f_k( ... f_2( f_1(x) ) ... )`. Apply the product and chain rules symbolically and each level distributes over the level below, duplicating subterms rather than naming them. The printed derivative can grow exponentially in the nesting depth even though the function itself is cheap to evaluate. This is **expression swell**, and it is the classic reason a hand-derived or computer-algebra derivative of a deep model is unusable while the same derivative computed op-by-op is trivial. Automatic differentiation avoids it by construction: the derivative of each primitive is a fixed local rule with no growth of its own, and every shared intermediate is computed once and reused, because it is a *value* sitting in a variable rather than a subterm sitting in an expression tree. Cost stays a small constant multiple of evaluating the function. There is a second limit. Symbolic differentiation needs a closed-form expression. A program with data-dependent branches or loops whose trip count depends on the input has no single formula; automatic differentiation simply differentiates the sequence of primitive operations that actually executed for this input, which is the derivative of the piece of the function you are standing on. Worth knowing that this is only meaningful away from kinks: at a point where the branch flips, the returned derivative is one side's, not a subgradient of the whole. ## Why finite differences are the wrong default Two errors pull in opposite directions. **Truncation error** comes from the Taylor remainder and shrinks with `h`: order `h` for a forward difference, order `h^2` for a central one. **Rounding error** comes from subtracting two nearly equal floating-point numbers and grows like machine epsilon divided by `h`. The total error is minimised at a sweet spot — around the square root of machine epsilon for a forward difference, times the scale of `x` — and even at that optimum you keep only about half the significant digits. Shrinking `h` further makes things *worse*, which is the single most common misconception here. Cost is the other problem. A finite-difference gradient needs one extra evaluation per input direction, so a model with millions of parameters is out of the question. Automatic differentiation's reverse sweep gets the whole gradient for a constant multiple of one evaluation. Finite differences still have one excellent use: as an **independent check**. If you write a new derivative rule by hand, compare its directional derivative against a central difference along a few random directions in double precision, and expect agreement to several digits, not to machine precision. A curiosity worth knowing: for real analytic functions the complex-step estimate `Im(f(x + i*h)) / h` has no subtractive cancellation, so it is accurate for very small `h`. ## Is automatic differentiation just symbolic differentiation in disguise? They are cousins — both apply exact chain-rule identities, and reverse mode can be described as symbolic differentiation with aggressive sharing of common subexpressions, evaluated eagerly. The distinctions that matter in an answer: automatic differentiation carries numeric values rather than expressions, so nothing swells; it has a bounded cost per primitive, so the total cost is predictable; and it follows the executed path, so control flow is not an obstacle. What it gives up is generality — you get the derivative at one point, and you must run it again at the next point. ## A compact summary - Symbolic: expression in, expression out. Exact everywhere, can swell, needs a closed form. - Numerical: black box in, approximation out. Trivial to apply, error-prone, one evaluation per direction. - Automatic: program in, numbers out. Exact at the point, fixed cost multiple, handles arbitrary control flow — which is why every trained network relies on it.