7
votes
Who introduced the term hyperparameter?
In 1996 Irving Good himself recalls:
One of the related problems close to philosophy is the estimation of the probability of one category of a multinomial when the order of the cells is irrelevant. [....
- 188k
5
votes
What does the KL being symmetric tell us about the distributions?
We're looking at the equation $$-\sum_{x\in\mathcal{X}} P(x) \log\left(\frac{Q(x)}{P(x)}\right)
=
-\sum_{x\in\mathcal{X}} Q(x) \log\left(\frac{P(x)}{Q(x)}\right).
$$
In the Bernoulli case,
$$-p \log\...
- 24.8k
5
votes
What does the KL being symmetric tell us about the distributions?
One example where this happens is when $P$ and $Q$ are "antipodal" distributions -- say, $p=(p_1,\ldots,p_n)$, $q=(q_1,\ldots,q_n)$,
$q_i=1-p_i$, and
$$P=\mathrm{Ber}(p_1)\times\mathrm{Ber}(p_2)\...
- 6,541
4
votes
Accepted
Multivariate normal concentration
Let $ Y := Z^T \Sigma Z $. We have $ \operatorname{var}(Y) = \mathbb{E}(Y^2) - \mathbb{E}(Y)^2 $ and $ \mathbb{E}(Y) = \sum_i \sigma_{i, i} $. We thus need to compute $ \mathbb{E}(Y^2) $. For this, ...
- 593
3
votes
Accepted
Derive equation for regularized logistic regression with batch updates
I found the solution (with the help of a friend: cudos!). The posterior is
$$\begin{align*}
-\log p(\boldsymbol{w}|\boldsymbol{x}) &= -\log p(\boldsymbol{x}|\boldsymbol{w}) - \log p(\boldsymbol{w})...
- 59
3
votes
A quantity associated to a probability measure space
The probability -- say $p$ -- for the experiment of rolling two different colored dice is
$$\frac{162601421574468954588}{2^{2\times36}}\approx0.0344322.$$
Here it is assumed that the random sets $A$ ...
- 128k
3
votes
Accepted
Updating Geman and Geman (1984) on image restoration
Given that the paper has accumulated a stunning 21.850 citations at Google scholar to this date and 487 are from 2018 the work is clearly influential (zbMATH lists 913 citations and 17 from 2018, ...
- 12.7k
3
votes
Accepted
Parametrising a sparse orthogonal matrix
The smallest number of nonzero entries in an $n\times n$ fully indecomposable$^*$ orthogonal matrix is $4n−4$. A method to construct such a matrix is described in Sparse orthogonal matrices (2003).
$^...
- 188k
3
votes
Parametrising a sparse orthogonal matrix
The four-argument version of Matlab's sprand can generate a sparse orthogonal matrix, with suitable arguments. According to the documentation,
[the matrix] is ...
- 20.2k
2
votes
Accounting for unobserved events in baysian learning
After observing $n$ events where $y=0$, I would have thought that your posterior distribution for $\theta$ should be $\operatorname{Beta}(1,n+1)$ which has expected value $\frac{1}{n+2}$ and so might ...
- 842
2
votes
Accepted
Posterior expected value for squared Fourier coefficients of random Boolean function
To begin with, I'll be switching to considering functions $\Omega^n = \{-1,1\}^n\to \{-1,1\} = \Omega$. This is of course entirely isomorphic to the $\{0,1\}$-valued bit setting, it just makes the ...
- 516
2
votes
Convolution of two Gaussian mixture model
The pdf of the sum $X+Y$ of independent random variables $X$ and $Y$ is the convolution of the pdf's of $X$ and $Y$. The convolution operation is bilinear. The convolution of Gaussian pdf's is ...
- 128k
2
votes
A problem with elementary inequality involving probabilities and Brier scoring rule
Clearly $\sum_{i=1}^{n-1}p_i - (\sum_{i=1}^{n-1}p_i)^2\geq 0$, so we can ignore those terms in the inequality. Since $(\frac{1}{1-p_n})\sum_{i=1}^{n-1}p_i^2 \geq \sum_{i=1}^{n-1}p_i^2$ it suffices to ...
- 1,338
2
votes
Bayesian inverse problems on non-separable Banach spaces
$\newcommand\B{\mathscr B}\newcommand\C{\mathscr C}$There is hardly any particular intuition behind the concept of the cylindrical $\sigma$-algebra. This is just the smallest $\sigma$-algebra with ...
- 128k
1
vote
Accepted
How does this Bayesian updating work $z_i=f+a_i+\epsilon_i$
$\newcommand{\bR}{\mathbb{R}}
\newcommand{\one}{\mathbf{1}}
\newcommand{\diag}{\textrm{diag}}
\newcommand{\Id}{\textrm{Id}}$
I guess you assume independence of the random variables $f,a,\varepsilon$, ...
- 211
1
vote
Accepted
Conditional Gaussians in infinite dimensions
$\newcommand\Si\Sigma\newcommand\X{\mathbf X}$If $Y,X_1,\dots,X_n$ are jointly normal zero-mean (real-valued) random variables, then
$$E(Y|X_1,\dots,X_n)=\Si_{12}\Si_{22}^{-1}\X,$$
where $\X:=[X_1,\...
- 128k
1
vote
Do these distributions have a name already?
The pushforward measure of a Gaussian distribution in $\mathbb R^{n+1}$ under the map $\mathbb R^{n+1}\ni(x_0,x_1,\ldots,x_n)\mapsto(e^{x_0},\ldots,e^{x_n})$ is called a multivariate lognormal ...
- 128k
1
vote
Gaussian process kernel parameter tuning
I do not know a mathmatical answer, but can give some intuition from the side of (Bayesian) machine learning.
If possible, one would like to avoid fitting parameters at all, and instead use strict ...
1
vote
Accepted
Lower bound for reduced variance after conditioning
The LHS of the expression you wrote is the minimum mean square error (i.e., the expectation $\inf E(X-\hat X)^2$ where $\hat X$ is measurable on $Y$). On the other hand, the expression
you wrote ($\...
- 7,514
1
vote
Learning a Gaussian from noisy observations
If I understand your question, $X$ has a Gaussian distribution and that there is added noise independent from the data, so actually your observation is $X=Z+e$ where $e$ is independent and identically ...
1
vote
Proving the existence of a symmetric Bayesian Nash equilibrium
You may try the following approach: for every strategy $x$,
calculate the set of best responses of a player who faces $I-1$
players who all play $x$. Show that the set-valued function just
defined ...
- 745
1
vote
Accepted
Conditional density for random effects prediction in GLMM
This has nothing to do with GLMM's per se. All what is done here is using the definition
$$f_{Y|X}(y|x):=\frac{f_{X,Y}(x,y)}{f_X(x)}$$
(if $f_X(x)\ne0$) to write
$$f_{Y|X}(y|x)f_X(x)=f_{X,Y}(x,y),$$
...
- 128k
1
vote
Optimal solution to cross entropy loss in the continuous case
$\newcommand{\Si}{\Sigma}$
It does not matter whether the random variable (r.v.) $R:=\Phi$ is discrete or continuous or neither; it can be any r.v. whatsoever, with values in any measurable space $(S,\...
- 128k
1
vote
What does the KL being symmetric tell us about the distributions?
I doubt much can be said. One example is where $p$ is a translation of $q$, but there are many others. I will use the notation and results from my answer at https://stats.stackexchange.com/questions/...
- 2,600
1
vote
Bayesian Inference with Student-t likelihood
Section 2.1 of this paper gives expressions for the posterior mean of location parameters. This may be helpful in your context.
- 2,791
1
vote
Bayesian methods in online setting
The model you are discribing is a Markovian state space model. You have a hidden state (p_k). A common approach is to use particle filtering (aka sequential Monte Carlo). The idea is to keep current ...
- 11
1
vote
Accepted
Shannon problem
The concept of weighted entropy with weight function $\varphi$ defined as
$$
H_\varphi = -\sum_i \phi(A_i) p(A_i) \log p(A_i)
$$
is not so new. However, this recent reference seems to give a good ...
- 10.4k
Only top scored, non community-wiki answers of a minimum length are eligible