Where Do Probabilities Come From?

English · 中文 · Français

First Principles of LLMs & RL · Article 01 · Why this series?

Writing note: The ideas and original draft material are mine. ChatGPT assisted with structuring, editing, and translation.

From observations to Softmax and maximum likelihood

I understood how to expand the probability chain rule. What stopped me was a question before the algebra: where did the probability come from in the first place?

A training dataset contains things that happened: a sentence, a label, the next token after a particular prefix. A language model assigns probabilities to things that might happen. The dataset does not hand it a probability for every possible continuation.

There are three linked problems. A neural network naturally gives us unconstrained real numbers, but we want a probability distribution over possible next tokens. One next-token distribution is still only local; a language model must assign probability to an entire sequence. And once the model can assign probability to an observed sequence, we still need a principle for changing its parameters.

Rather than treating Softmax, the probability chain rule, and maximum likelihood as separate topics, I will follow the same object as it changes form:

This is a conceptual route, not a literal forward pass: the learning objective at the end will eventually loop back and change the shared parameters near the beginning. Each step appears because the previous one leaves a question unanswered.

I. How does a model get probabilities?

Observations, counts, and shared parameters

Consider five sentences. For this toy example, treat each word as one token:

I like cats
I like dogs
I like cats
I like pizza
I like cats

After I like, cats occurs three times; dogs and pizza occur once each. In that order, the empirical conditional distribution is

The hat marks an estimate constructed from the sample. The observations are the five sentences. The empirical distribution is a description we construct from them. It is not a complete account of how language is generated.

Counting continuations is closest to a count-based language model, not to Word2Vec. For example, let C(u,v) count how often token v follows token u, and let C(u) count occurrences of u with a following token. An unsmoothed bigram estimate is

Longer contexts make direct counting sparse. A prefix may occur once, or not at all. Count-based models can address sparsity through smoothing and shorter contexts; neural language models instead learn distributed representations and probability functions that share information across examples.¹

Word2Vec belongs on the parameter-sharing side of this divide: it learns vector parameters from word-context observations rather than storing an exact-context frequency table. “Static” describes how a word’s lookup vector is used, not a table of counts.²

For our language model, let c denote a context, y a candidate next token, and θ all model parameters. We want a function

Parameter sharing means that different contexts use the same parameter set, not that they receive the same output. An update from one example changes parameters also used for other examples. This creates an opportunity to generalize; it does not guarantee that generalization will be good.

Sharing and initialization are separate choices. In the from-scratch setting considered here, weight matrices start from a chosen random initialization; some other parameters can start at fixed values such as zero or one. Sharing is already part of the architecture before any training happens. Fine-tuning, by contrast, starts from previously learned parameters.

Parameter sharing lets one function compute outputs for many different contexts, including unseen ones. It does not by itself guarantee that those outputs generalize correctly. But the object we ultimately need is not an arbitrary real vector. It is a normalized distribution over possible next tokens. That gap is why logits and Softmax enter.

What exactly is a logit?

Calling a logit a “score” leaves an important question unanswered: a score of what?

Suppose a context has been mapped to a hidden vector h of dimension d. For candidate token i, an output layer can compute

Here wᵢ is a learned d-dimensional vector and bᵢ is a scalar bias. With K possible tokens, the outputs form

These are the logits. At this point they are just unconstrained real numbers. They do not have a predefined unit of truth, confidence, or frequency. Their probabilistic interpretation comes from the way we choose to use them.

One useful choice is to interpret each logit as a log-weight. Exponentiating gives a positive weight:

Normalize those weights to obtain a categorical distribution:

That is Softmax. More concretely, the probabilities are the conditional distribution we wanted from the start. If the network computes a logit vector from context c, then the probability assigned to vocabulary item i is

For a finite vector of finite real logits, exact arithmetic gives strictly positive probabilities whose sum is one. A probability distribution in general may contain zeros; finite-logit Softmax represents its strictly positive interior.

A small example makes the transformation concrete. The entries below are rounded, but calculated from the same vector:

The rows now stand for the same three continuations as at the beginning: cats, dogs, and pizza. This is a toy model snapshot, not the empirical distribution. The model assigns cats only about 0.0900, even though cats appeared in three of our five observations. Softmax has produced valid probabilities, but they do not fit these observations well. We will return to that discrepancy once we have a learning principle.

Think of allocating a fixed budget in proportion to positive weights. The weights need not sum to one; their shares do. Logits record those weights on a logarithmic scale. This is an analogy for the parameterization, not a claim that the network contains literal votes or evidence counters.

For a precise relationship, define the normalization constant

Using natural logarithms throughout this article,

A logit is a log-probability plus an offset shared by all candidates in the same distribution. That offset depends on the full logit vector. A logit by itself is therefore not a probability or a log-probability.

Softmax does not discover hidden probabilities inside arbitrary numbers. We choose it as a map from unrestricted scores to a distribution, then train the scores through that map. Even an untrained network can produce a valid distribution. Equal logits, for example, give

Random initialization need not produce exactly equal logits. The point is that a valid distribution can exist before the model has learned useful predictions. Normalization guarantees neither accuracy nor calibration. A next-token probability is also not automatically a probability that a statement is true.

We have now answered the first half of the local problem: logits give the model unconstrained coordinates, and Softmax turns those coordinates into a valid distribution. The next question is why this particular map has the structure it does.

Differences are probability ratios

The most informative Softmax identity comes from comparing two candidates. Their normalization constant cancels:

Equivalently,

A one-unit logit advantage means an e-fold probability ratio, not one extra percentage point of probability. Other candidates affect each individual probability, but not this pairwise ratio when the two logits are held fixed.

For two candidates, define

Then

The graph below plots these two probabilities against the difference. Equal logits give equal probabilities. A difference of 1 gives approximately 0.7311 and 0.2689.

Move the slider to compare the probabilities. Open the interactive graph · Marimo source.

In the binary case only, the second probability is the complement of the first, so

This is the usual log-odds. With more classes, pairwise log-probability ratio is the less ambiguous description.

Why exponentials, and why this base?

Now that score differences have a probability-ratio interpretation, the exponential map is worth examining. An obvious alternative is to divide scores by their sum. With arbitrary real scores, this can fail immediately:

The entries sum to one, but they are not probabilities. The denominator could also be zero.

Many positive functions avoid that problem. So positivity alone does not select the exponential. The more interesting property is how Softmax handles a shared offset. If a is any real constant and 𝟙 is the all-ones vector, then

This follows because every exponential gains the same factor:

That factor cancels during normalization. Scores (1, 2, 3) and (101, 102, 103) therefore define the same distribution. Choosing to ignore a common score baseline is a modeling preference; Softmax satisfying that preference is an algebraic fact.

There is a useful conditional characterization here. Suppose we restrict ourselves to maps that apply the same continuous positive function g to each score and then normalize:

If every shared shift must leave the probabilities unchanged, then for two scores u and 0 we obtain

After defining

this becomes the additive equation

Continuity then gives a linear h and therefore

Requiring higher scores to receive higher weights gives α > 0, while C cancels during normalization. So the exponential is singled out within this construction and these assumptions; Softmax is not the only conceivable map into a probability distribution.

The base is a separate issue. For any b > 1,

Changing the base rescales the logits; it does not leave the probabilities of a fixed score vector unchanged. Natural exponentials and natural logarithms are convenient partners in calculus, but the important structure here is exponential weighting.

Temperature changes the scale of this relationship. For a positive temperature τ,

Therefore,

A smaller positive temperature amplifies nonzero differences; a larger one compresses them. Temperature controls how strongly a score difference becomes a probability ratio. It does not change which score is largest.

That completes the local problem. Given one prefix, the model can now produce a normalized distribution over the next token. But language data are sequences, not isolated prediction steps. How does a collection of next-token distributions become a probability for an entire sequence?

II. From next-token conditionals to sequence probability

Conditional probability is where the chain rule comes from

We now have a distribution for the next token. In a sequence, the context c is the already observed prefix, and the next token y is the token at the current position. The conditional model from Part I therefore supplies exactly the factors a sequence model needs. To combine them, return to conditional probability itself.

Let A and B be events, with P(A) positive. The definition is

We restrict attention to the probability mass inside A and ask what fraction also belongs to B. Rearranging gives

For example, if A has probability 0.4 and B occurs within A with conditional probability 0.25, their intersection has probability 0.1. No independence assumption was needed.

For a fixed-length sequence, let T be the length. Write its observed token values as x₁, …, x_T, and define the prefix notation

Repeatedly apply the same conditional-probability identity to prefixes, assuming the conditioning prefixes have positive probability:

Expanding recursively yields

The first factor is the probability of the first token, with an empty prefix. The notation abbreviates probabilities of token-valued random variables taking these particular values.

The chain rule does not require independent tokens. It keeps their dependencies in the conditioning. Independence would allow us to remove those prefixes, which is a different statement.

We choose a model for each conditional distribution and multiply them to define a sequence model:

For fixed T, normalized conditionals produce a normalized joint distribution: summing over the last token gives one, then summing over the preceding token does the same, recursively. This identity does not assert that the model equals the data-generating distribution. Variable-length generation would also require accounting for how sequences end; we keep the length fixed here.

We have now moved from a local conditional distribution to a probability for a whole sequence. That creates the bridge to learning: different parameter settings can assign different probabilities to the same observed sequence. Hold the observation fixed and vary the parameters, and sequence probability becomes likelihood.

III. How do observations shape the distribution?

Likelihood: hold the observations fixed

A model can now assign probabilities. We still need a principle for choosing its parameters.

Consider a hypothetical sequence of ten independent coin flips, with eight heads and two tails:

H H H T H H T H H H

Let p be the same head probability on every flip. For this particular ordered dataset D,

Fix p and consider different possible datasets: this is a probability model. Fix the observed D and compare different p values: the same expression becomes a likelihood function,

Probability holds the model fixed and varies the data. Likelihood holds the data fixed and varies the parameters. Likelihood is not a probability distribution over parameters and does not tell us the probability that a parameter value is true.

Maximum likelihood prefers the parameter setting that assigns the highest likelihood to the fixed observations:

The graph makes the direction of comparison visible. We move along the parameter axis while keeping the same ten flips throughout.

The maximum likelihood is approximately 0.00671: even at its best-fitting parameter, this particular ordered sequence has a small probability. MLE compares different parameters for the same data; the likelihood does not need to be close to one.

Setting p to 1 would make the two tails impossible, so its likelihood is zero. Seeing more heads does not mean assigning heads probability one.

If we recorded only the number of heads, rather than the ordered sequence, a binomial coefficient would also appear. That factor is independent of p and leaves the maximizing parameter unchanged.

Maximum likelihood is an estimation principle, not a consequence forced on us by Softmax. It also does not guarantee a well-generalizing model. Here we are constructing a training objective, not proving that its optimum is the best possible predictor outside the sample.

Why the logarithm, and why the minus sign?

For positive numbers,

Since the natural logarithm is strictly increasing, taking it preserves the maximizing parameters wherever likelihood is positive:

Zero likelihood can be assigned log-likelihood minus infinity by a limiting convention. For the coin, at interior p values,

Differentiating and setting the derivative to zero gives

Hence

The second derivative is negative throughout the interval, and the endpoint likelihoods are zero:

So this is the unique maximum. In this Bernoulli example, MLE recovers the empirical frequency.

The logarithm also helps computation: products of many small probabilities can underflow, whereas we can accumulate their logs without first forming the product. Taking a log after the product has already rounded to zero cannot recover the lost information.

Finally, a maximization can be rewritten as a minimization by changing the sign:

This is negative log-likelihood. The log turns products into sums; the minus sign turns maximization into minimization.

For one observed outcome assigned probability p, the contribution is

The curve is drawn from p = 0.001 to p = 1. At p = 0 the loss has no finite value. The marked probabilities 0.1, 0.5, and 0.9 give losses approximately 2.303, 0.693, and 0.105.

A high assigned probability gives a small loss. A tiny assigned probability gives a large loss. The curve describes the training penalty, not a separate guarantee that the prediction was calibrated.

Putting the chain together: the language-model objective

The coin example showed maximum likelihood with one parameter. A language model uses the same principle, but its sequence probability has already been decomposed into next-token factors. We can now substitute the chain rule into the dataset likelihood.

Take N training sequences of the fixed length T used above, retaining repeated sequences as repeated observations:

Modeling them as independent draws from the same sequence model gives

The chain rule then gives

The outer product uses an independence assumption across examples. The inner product uses the chain rule within a sequence. They are not the same assumption written twice.

Taking the negative log produces

Each position supplies an observed prefix and the token that followed it. The model predicts a full distribution; the loss selects the probability assigned to that observed token. Observing cats once does not imply that dogs was impossible.

Return to the five observations at the beginning. Let c denote their common context, “I like”. If we model their next-token outcomes as conditionally independent given c and θ, their conditional likelihood is

In the illustrative table, cats received a probability of about 0.0900. Each observed cats continuation therefore contributes approximately

to the negative log-likelihood. The five outcomes together give

These figures are calculations from the toy scores, not experimental training results. Counts, Softmax probabilities, and likelihood now refer to the same observations. The observations constrain the entire distribution; not every candidate probability can increase at once.

Because the probabilities came from Softmax, the negative log-likelihood of a single observed token also has a direct expression in terms of the logits:

This is an algebraic identity for negative log-Softmax, not a new learning principle. It also explains why numerical implementations often compute log-probabilities directly.

Under this model and loss, next-token training is sequence maximum-likelihood training expressed as a sum of local terms. Dividing by the fixed number of predicted tokens rescales the objective without changing its minimizers. We have not yet derived the optimizer or promised that it will find a global optimum.

The entire argument can now be read from left to right:

The learning objective will then provide the signal used to update θ; the optimization machinery itself belongs later in the series. Softmax supplies the local distribution; the chain rule turns local conditionals into a sequence probability; likelihood turns that sequence probability into a function of the parameters; negative log-likelihood turns the estimation principle into a loss.

Implementation note: stable Softmax

The conceptual chain is complete at this point. One numerical detail is worth keeping separate from the main argument. Softmax’s shift invariance lets us subtract the largest logit before exponentiating. For finite float32 or float64 inputs with a nonempty last axis, this teaching implementation also handles batches:

import torch

def softmax_stable(x: torch.Tensor) -> torch.Tensor:
    """Normalize finite float32/float64 logits along a nonempty last axis."""
    max_x = x.max(dim=-1, keepdim=True).values
    shifted_x = x - max_x

    exp_x = torch.exp(shifted_x)
    row_sums = exp_x.sum(dim=-1, keepdim=True)

    return exp_x / row_sums

The tensor returned by .values contains the maxima rather than their indices. Keeping that dimension lets subtraction broadcast along the candidate axis. In exact arithmetic this computes the same Softmax; in floating point it avoids large positive exponential arguments. Extremely small weights may still underflow.

When the next operation is a logarithm, computing log(softmax(x)) separately can lose information through those tiny probabilities. PyTorch’s log_softmax computes log-probabilities directly using a more numerically stable formulation.³ Mathematical equivalence alone does not make two floating-point implementations equally reliable.

The companion llm-from-first-principles project is where I work through such implementation details.

Where this leaves us

The distinction worth keeping is simple: parameterization makes a distribution available; the learning objective tells us how observations should shape it.

One expression now deserves another look:

So far it emerged from likelihood and a convenient transformation. Why is it also called surprisal, or self-information? And what changes when we average it over a distribution rather than evaluate one observation?

That is where the next article begins.