How Does a Neural Network Know Which Way to Move?

English · 中文 · Français

First Principles of LLMs & RL · Article 03 · Previous article

Writing note: The questions and initial draft material are mine. ChatGPT assisted with structuring, editing, figures, and translation.

From local approximation to gradient descent and backpropagation

The previous article explained how cross-entropy measures a model’s prediction loss. But computing a loss does not change the model’s parameters.

A number can tell us how poor the current prediction is. It does not arrive with instructions about which parameter to increase, which to decrease, or by how much.

Between measuring an error and modifying the model, we need an account of how the error changes.

I want to follow that missing link. First, what does a small parameter change do to the loss? Then, which changes would be favorable? Finally, how do we calculate those sensitivities when the parameters are separated from the loss by a long chain of computations? Taylor expansion, gradient descent, and backpropagation answer these questions in sequence.

1. A loss value has no direction. A derivative does.

Set the network aside for a moment. Suppose there is only one real parameter, and the loss is

At parameter values 1 and 5, the loss is 4 in both cases. Yet to approach the minimum, we should increase the former and decrease the latter. The loss value alone cannot distinguish these situations.

The derivative supplies the missing information:

At parameter value 1, it is −4. That does not mean the loss is negative. It means a sufficiently small increase in the parameter will decrease the loss at this point.

This interpretation follows almost directly from the definition of a derivative. Let δ be a small change to the parameter:

For sufficiently small δ, we can approximate the change in loss by

For example, starting at 1 and increasing the parameter by 0.01, the first-order approximation predicts a loss of 3.96. Evaluating the original function gives 3.9601.

The approximation is not exact, but it captures the effect of this small change.

A derivative is not the size of the current error. It is the local sensitivity of the loss to a change in a parameter. Equal loss values may have entirely different sensitivities.

2. Taylor expansion: using local information to reconstruct nearby behavior

The approximation above is already a first-order Taylor approximation. It replaces the function near the current point with a line having the same value and slope there.

If a line matches height and slope, could a polynomial also match curvature?

Let f be a scalar-valued function, a the point around which we expand, and δ the displacement from a. Start with a quadratic polynomial:

Require that its value and its first two derivatives at zero match those of f at a:

Then

The factor one half is not an arbitrary convention. Differentiating a quadratic term twice produces a factor of 2, which has to be canceled when we match the second derivative. Matching higher derivatives produces the corresponding factorials.

More generally, let n be the highest polynomial degree and denote the remainder by Rₙ:

Here n is a nonnegative integer and the zeroth derivative is the function itself. This is an identity with a remainder. It becomes a useful approximation only when that remainder can be controlled.

For instance, suppose the (n+1)st derivative is continuous on the interval of interest, and its absolute value is at most M. Taylor’s theorem gives

Under these assumptions, small displacements control the error. None of this says that the infinite Taylor series of every smooth function converges to that function.¹

The point of Taylor expansion is to construct a simple approximation to nearby behavior from derivatives at one point.

Consider the exponential function expanded around zero:

The approximations match increasingly many derivatives at the expansion point. They are not the same function everywhere. The horizontal axis is the displacement from zero.

At x = 0.2, the first-order approximation gives 1.2, the second gives 1.22, and the true value is about 1.2214. At x = 1, the same approximations give 2 and 2.5, while the true value is about 2.7183.

I deliberately chose a nonquadratic function for this graph. A quadratic’s second-order Taylor polynomial equals the original function everywhere, hiding the limitations of approximation.

Now return to learning. If a local approximation lets us predict the effect of a small parameter change, which change should we choose?

3. Why the negative gradient, and why does step size still matter?

A network has many parameters. Write them as an m-dimensional real vector in bold, and for now hold the training data, random choices, and the current computation fixed, so that the loss is a deterministic function of that vector.

The partial derivatives form the gradient:

At a differentiable point, the multivariable first-order approximation becomes

The right side is an inner product: each parameter change is multiplied by its local sensitivity, and the contributions are added. A gradient therefore evaluates an entire small update, not merely separate coordinates.

To talk about the steepest descent direction, we must decide how to compare the lengths of directions. Use Euclidean length, with u denoting a unit vector. By Cauchy–Schwarz,

If the gradient is nonzero, equality holds when

Among equally small changes measured by Euclidean length, the negative gradient has the greatest instantaneous rate of decrease. “Steepest” depends on how length is defined; it is not a coordinate-free absolute statement.²

The simplest gradient descent rule scales the negative gradient by a positive learning rate η:

Putting the proposed change back into the first-order approximation gives

For nonzero g, the right side is negative. But it only tells us about sufficiently small steps: the approximation was built at the current point, not at every point we might reach.

The gradient indicates which way a small step should go. The learning rate determines how far the actual step goes.

Return to the parabola, starting at parameter value 1. The derivative is −4, so the next parameter is

These arrows compare alternative single updates from the same starting point. They are not successive iterations in one training run, and the line connecting endpoints is not a continuous optimization trajectory.

For this particular quadratic, we can even derive exactly when the error shrinks:

A fixed learning rate strictly between 0 and 1 makes the error shrink; η = 1 makes a nonoptimal point oscillate between two positions, and η > 1 makes the error magnitude grow. This range belongs to this function and must not be used as a universal learning-rate rule.

The difference between local and global optimization is just as important. At a differentiable point with nonzero gradient, some sufficiently small step along the negative gradient decreases the loss. That alone does not guarantee convergence, or even convergence to a local minimum. A zero gradient may also indicate a local maximum or saddle point.²

Convexity supplies additional global structure: for an unconstrained differentiable convex objective, a point with zero gradient is globally optimal. Whether a particular algorithm reaches such a point requires further conditions. In general, “gradient descent at least guarantees a local optimum” is not a valid claim.

So far, we have assumed the gradient is already available. But a parameter does not directly produce the loss. Many computations lie between them. How are all those derivatives obtained?

4. Backpropagation: tracing sensitivity through a computation

Suppose a parameter first passes through φ to produce u, and the final loss is obtained by applying f:

A small parameter change first changes u and then changes the loss:

Combining the steps and taking the limit gives the calculus chain rule:

Sensitivities multiply along a computational path because the change produced at one stage is magnified, reduced, or reversed at the next.

This chain rule is not the probability chain rule of Article 01. One differentiates compositions of functions; the other factorizes joint probabilities. Both operate stepwise, but they are different statements.

If a parameter influences the loss through several paths, we must include all of them. The paths’ contributions add where they meet.

Backpropagation organizes these relations on a computational graph. In the forward pass, each node computes an intermediate value. Working backward from the loss, we propagate derivatives in the reverse order of dependencies. Derivatives already computed for a shared intermediate node are reused instead of being recomputed from scratch for each parameter.³

Backpropagation also does not mean perturbing one parameter at a time and rerunning the network to estimate slopes. That would be finite differences, which can check a small example. Here we differentiate the operations composing the graph and assemble their derivatives with the chain rule.

Backpropagation computes gradients; an optimizer uses those gradients to update parameters. A model does not move simply because its gradients have been computed.

We can now make that distinction concrete by returning to the very same distribution used in the first two articles.

5. One real update for cats, dogs, and pizza

Keep the context “I like”. Five observed continuations gave the empirical distribution

The three candidates are cats, dogs, and pizza, in that order. The model produces logits z and a temperature-1 Softmax distribution:

Holding P fixed and using natural logarithms, define average negative log-likelihood as a function of logits:

The average matters here. The total NLL of the five observations would have a loss and gradient five times as large; using the same learning rate would produce a different step.

Write Softmax inside the loss and use the fact that P sums to one:

Differentiating the first term with respect to the kth logit gives Qₖ; differentiating the second gives −Pₖ. Consequently,

This is an exact result under the stated Softmax, fixed-target, and natural-log conventions, not a property of every imaginable loss.

Take the logits from the previous two articles:

They give approximately

If the three logits could be adjusted independently, the negative-gradient step would raise cats and lower the other two scores. We now have a direction corresponding to the earlier intention of assigning observations more appropriate probabilities.

In a real network, however, we usually update shared parameters, not independent logits permanently stored for each context. For a worked example, construct a model with only one trainable scalar parameter:

This is an intentionally limited teaching model, not a description of a full language-model architecture. At θ = 0, it reproduces the logits above.

One parameter affects both cats and dogs. Define the parameter-level objective

By the chain rule, the contributions along every output path must be added:

At θ = 0,

The three paths have different signed contributions. The bottom bar is their sum, not a fourth path. The pizza logit does not depend on θ, so that path’s local derivative is zero.

This is “multiply along a path, add at the point where paths meet” in numerical form. The cats path favors increasing the parameter; the dogs path favors decreasing it. Their total determines the parameter’s local update direction.

Now take one gradient descent step with learning rate 0.5:

Recompute the logits, probabilities, and loss:

There is a detail worth noticing. Dogs already had a model probability above the empirical target 0.2; after the update, it increases even further.

That is not a contradiction. One shared parameter affects several logits at once; we do not have an independent dial for every probability. The objective minimizes the total P-weighted log-loss, not each class’s absolute probability error. This step reduces the former without improving every class’s match to P.

And although the pizza logit does not move, its probability does: the Softmax denominator depends on all logits.

A parameter update can reduce the overall loss without bringing every single predicted probability closer to its target in that step.

These are calculations from the specified equations, not claims about measured training performance.

6. Separating differentiation from an update in code

The same toy model fits into a short, typed PyTorch example:

import torch


def toy_logits(theta: torch.Tensor) -> torch.Tensor:
    """Map a scalar parameter to three shared logits."""
    return torch.stack((2.0 * theta, theta + 1.0, theta.new_tensor(2.0)))


def toy_loss(theta: torch.Tensor, target: torch.Tensor) -> torch.Tensor:
    """Return the average natural-log loss for a fixed target distribution."""
    log_probs = toy_logits(theta).log_softmax(dim=-1)
    return -(target * log_probs).sum()


target: torch.Tensor = torch.tensor([0.6, 0.2, 0.2], dtype=torch.float64)
theta: torch.Tensor = torch.tensor(0.0, dtype=torch.float64, requires_grad=True)
learning_rate: float = 0.5

loss_before = toy_loss(theta, target)
loss_before.backward()  # Compute gradients; theta is still zero.

if theta.grad is None:
    raise RuntimeError("The parameter gradient was not computed.")

with torch.no_grad():
    theta -= learning_rate * theta.grad  # Perform the parameter update.

theta.grad = None  # Do not accumulate this step's gradient into the next one.
loss_after = toy_loss(theta, target)

print(f"{loss_before.item():.6f} -> {loss_after.item():.6f}")

The checked output is

1.807606 -> 1.389241

backward() traverses the graph and accumulates derivatives in the leaf tensor’s .grad. It does not update θ. We perform the update inside no_grad(), and then run a fresh forward computation to obtain the new loss. Before the next backward pass, any gradients that should not accumulate must be cleared.⁴

Some parts of this story use approximations and some do not. Taylor approximates the effect of a finite displacement by a local polynomial. Backpropagation through this smooth computational graph instead computes derivatives at the current point by applying differentiation rules; it does not use a tangent line to guess each derivative. Floating-point computations naturally still incur numerical error.³

Only now do we have a training loop: a forward pass produces a loss, a backward pass produces sensitivities, an update rule changes parameters, and the next forward pass evaluates the new model.

We have not obtained a complete map to the optimal model. We have obtained a local procedure we can repeat.

The first two articles explained how to represent a distribution and how to define a loss. This one allowed that loss to influence shared parameters. Yet our toy model reduced the mapping from context to logits to something tiny.

The next article will open that box: how can a token retrieve useful information from earlier tokens, in a way that can be trained using the same loss?