Suppose we're trying to train a classifier $\pi$ for $k$ classes that takes as input a feature vector $x\in\mathbb{R}^n$ and outputs a probability vector $\pi(x)\in\mathbb{R}^k$ such that $\sum_{v=1}^k \pi(x)_v = 1$ and $\pi(x)_v\in [0,1]$ is the probability that the object with features $x$ belongs to the $v$th class.

The standard logistic model for such a classifier uses the function $\pi(x)$ whose $v$th component is given by

$$\pi(x)_v = \frac{e^{\lambda_v\cdot x}}{\sum_{u=1}^k e^{\lambda_u\cdot x}}$$

for some weight vector $\lambda_v\in\mathbb{R}^k$. If $k = 2$ and $\lambda_2 = 0$, then this is just the sigmoid function in $\lambda_1\cdot x$. If our training data consists of feature vectors $x(1),\ldots,x(m)\in\mathbb{R}^n$ and classifications $y(1),\ldots,y(n)\in\{1,\ldots,k\}$, then logistic regression would have us choose the weights $\lambda_v$ to maximize the product $$\prod_{i=1}^m\pi(x(i))_{y(i)}$$ which is the probability that the model correctly classifies each item in the training data. Apparently, under mild hypotheses, the maximum value is attained at a unique $\lambda = (\lambda_v)_{v = 1,\ldots,k}$.

My basic question is: In what sense is this an optimal model? One answer, which I learned from these notes by John Mount, is that when restricted to a subclass of classifiers satisfying a certain "balance condition" (described below), this model is the one that maximizes entropy over the training set. If the training set consists of the feature vectors $x(1),\ldots,x(m)$, then this model maximizes the quantity $$-\sum_{i=1}^m\sum_{v=1}^k \pi(x(i))_v\log \pi(x)_v$$ I accept that maximizing entropy makes sense philosophically, but this shifts our attention to the balance condition, which would require: $$\sum_{i=1}^m \pi(x(i))_u x(i)_j = \sum_{i=1}^m \delta(u,y(i))x(i)_j\qquad\text{for all $u,j$}$$ where $\delta(a,b) = 1$ if $a = b$, and 0 otherwise.

Why is it reasonable to restrict our attention to functions which satisfy these balance conditions?

  • $\begingroup$ Normally there's an intercept parameter, thus: $$\pi(x)_v = \frac{e^{\alpha + \lambda_v\cdot x}}{\sum_{u=1}^k e^{\alpha + \lambda_u\cdot x}}$$ This is not trivially equivalent to the form that omits $\alpha$ because $\alpha$ is not part of the observed data but, like $\lambda,$ must be estimated by maximizing the product that you mentioned. $\endgroup$ Sep 15 at 1:40
  • 1
    $\begingroup$ I'll probably post an answer here within a few days. $\endgroup$ Sep 16 at 22:07
  • $\begingroup$ @MichaelHardy That would be great! $\endgroup$ Sep 17 at 1:15
  • $\begingroup$ This question has caused me to realize that I'm rusty in certain things. $\endgroup$ Sep 17 at 20:26

1 Answer 1


On the space $\{(x_1,\ldots,x_v)\in(0,1)^v : x_1+\cdots + x_v=1\},$ introduce an operation of addition as follows: $$ \mathbf a \mathbin{\text{“}{+}\text{”}} \mathbf b =(a_1,\ldots,a_v) \mathbin{\text{“}{+}\text{”}} (b_1,\ldots,b_v) = \frac{(a_1b_1,\ldots, a_vb_v)}{a_1b_1+\cdots +a_vb_v}. $$ Now we have a vector space (where it is now easy to see what the scalar multiplication will be).

If $\mathbf a = \big( \Pr(A_1), \ldots, \Pr(A_v) \big)$ and $\mathbf b \propto \big( \Pr(D\mid A_1),\ldots, \Pr(D\mid A_v) \big) $ then $\mathbf a \mathbin{\text{“}{+}\text{”}} \mathbf b = \big( \Pr(A_1\mid D),\ldots, \Pr(A_v\mid D) \big).$

Thus the arrival of new data $D$ corresponds to a translation in this vector space.

With this structure, the mapping to be estimated, from $x\in\mathbb R^n$ to the smaller $(v-1)$-dimensional vector space, and that is estimated by maximizing the product that was mentioned, is affine.

(If I'm not mistaken, the numerical algorithm used for this is "iteratively reweighted least squares", but the weights to which the algorithm converges depend on the data—the observed values of the $x$s and the categories, so this is not the same as what is usually called weighted least squares.)

  • $\begingroup$ Maybe I'll add some more here later. $\endgroup$ Sep 22 at 2:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.