All Questions
Tagged with it.information-theory machine-learning
11 questions
1
vote
0
answers
45
views
Existence of optimal entropic weights for empirical modeling
Let $\mathcal{X} = [0,1]^n$ be the input space and $\mathcal{Y} = \{1, ..., n_c\}$ be a discrete output space. Let $D = \{(x_i, y_i)\}_{i=1}^N \subset \mathcal{X} \times \mathcal{Y}$ be a training ...
- 111
8
votes
2
answers
693
views
Information theoretical interpretation of Free Energy
When exploring the concept of free energy from an information-theoretic perspective, I often come across statements like:
"Free energy measures the degree of surprise an agent experiences when ...
- 253
1
vote
0
answers
40
views
From a constraint satisfaction problem (CSP) to a sudoku grid [closed]
one of the existing methods of solvin a sudoku grid is via constraints satisfaction (CSP), but can we do the inverse ie convert a CSP problem into a sudoku grid and then solve it ?
2
votes
1
answer
138
views
How to lower bound the absolute value of the difference of two Kullback-Leibler divergences given the constrains below?
Given that $\min (D[p_1||p_3],D[p_2||p_4])=a$, how to find a lower bound of the difference $\left \vert D[p_1\parallel p_2]-D[p_3\parallel p_4] \right\vert$ with respect to $a$? If the condition is ...
- 31
1
vote
0
answers
35
views
A network to transform/predict one probability distribution to another
I have a random variable of a particular density (e.g., normal), and a known probability distribution (e.g., mixture Gaussian). I used a simple KL measure to predict/transform one another. Now I need ...
- 11
6
votes
2
answers
344
views
Entropy & difference between max and min values of probability mass
Let $X$ be a random variable with probability mass function $p(x) = \mathbb{P}[X = x]$.
I know entropy $H(X)$ of $X$ measures the uncertainty of $X$ and
a large value of $H(X)$ means $p(x)$ is nearly ...
- 163
1
vote
0
answers
34
views
Correlating two matrices $A,B$ with stochastic dependency structure imposed by cross-validation
Consider a labelled data set
$$D = \{(x_1, y_1),...,(x_n, y_n)\} $$
on which we want to evaluate a machine learning algorithm using $k$-fold cross validation with $m$ different random seeds. This ...
- 153
7
votes
1
answer
660
views
Books to develop a unified view of statistics and information theory?
I hope to understand the connection between statistics and information theory in a deep philosophical sense.
I suppose the best place to start would be David MacKay's Information Theory, Inference, ...
- 73
2
votes
0
answers
217
views
Inequality on the Kullback-Leibler divergence
Let us define the arithmetic, geometric, and harmonic means of $x,y \in \mathbb{R}$ weighted by $\alpha =(\alpha_x,\alpha_y) \in [0,1]$, respectively as
\begin{equation}
a_\alpha(x,y) = \frac{\...
- 245
0
votes
2
answers
299
views
Statistical divergence
Does anyone know about a statistical divergence of this type?
\begin{equation}
\text{D}(P||Q) = \frac{1}{2} \left[\text{KL}(M||P) + \text{KL}(M||Q)\right]
\end{equation}
where $M = \frac{1}{2} [P+Q]$....
- 245
10
votes
1
answer
421
views
How sensitive are Neural Networks to weight change?
Let's consider the space of feedforward neural networks with a given structure: $L$ layers, $m$ neurons per layer, ReLu activation, input dimension $d$, output dimension $k$.
Which means I'm ...
- 899