# Questions tagged [machine-learning]

The machine-learning tag has no usage guidance.

77
questions

**2**

votes

**1**answer

43 views

### Uniform Lipschitz function approximation by shallow neural networks

Fix $d\in \mathbb{N}$. Let $F_1$ be the set of all 1-Lipschitz functions mapping $[0, 1]^d$ to $\mathbb{R}$.
For $\varphi: \mathbb{R} \rightarrow \mathbb{R}$ and $m \in \mathbb{N}$, let $N_\varphi^m$ ...

**0**

votes

**1**answer

88 views

### Independence in a sequential problem with observations getting added to buckets

Consider a sequence of random observations $(O(t))_{t\geq 1}$, with $O(t)=(D(t),J(t),Y(t))$. Denote $\mathcal{F}(t) := \sigma(O(1),\ldots,O(t))$, the filtration induced by the first $t$ observations.
...

**7**

votes

**1**answer

120 views

### Books to develop a unified view of statistics and information theory?

I hope to understand the connection between statistics and information theory in a deep philosophical sense.
I suppose the best place to start would be David MacKay's Information Theory, Inference, ...

**3**

votes

**1**answer

161 views

### Abstract mathematical concepts/tools appeared in machine learning research

I am interested in knowing about abstract mathematical concepts, tools or methods that have come up in theoretical machine learning. By "abstract" I mean something that is not immediately related to ...

**2**

votes

**0**answers

93 views

### Inequality on the Kullback-Leibler divergence

Let us define the arithmetic, geometric, and harmonic means of $x,y \in \mathbb{R}$ weighted by $\alpha =(\alpha_x,\alpha_y) \in [0,1]$, respectively as
\begin{equation}
a_\alpha(x,y) = \frac{\...

**0**

votes

**2**answers

138 views

### Statistical divergence

Does anyone know about a statistical divergence of this type?
\begin{equation}
\text{D}(P||Q) = \frac{1}{2} \left[\text{KL}(M||P) + \text{KL}(M||Q)\right]
\end{equation}
where $M = \frac{1}{2} [P+Q]$....

**1**

vote

**0**answers

53 views

### Comparison of concentrations of different $L^p$-norms of (sub) Gaussian distributions

It's well-known that the Euclidean $2$-norm of subgaussian random vectors concentrates in high dimensions, e.g. when $X \sim \mathcal{N}(0,I_n),$ (or in general $X$ is subgaussian with independent co-...

**0**

votes

**2**answers

160 views

### Help with a definition of a two-person game in a referenced paper

In the paper "Finding Mixed Nash Equilibria of Generative Adversarial Networks" the authors write in equation (1) on page 2:
Consider the classical formulation of a two-player game with
finitely ...

**0**

votes

**0**answers

29 views

### What is the number of iterations needed for the message passing algorithm to converge when applied to an acyclic factor graph?

I understand that the message passing algorithm (Belief Propagation algorithm), when applied to a factor graph consists in an exchange in messages between the factor nodes and the variable nodes, ...

**8**

votes

**4**answers

2k views

### mathematical physics without partial derivatives

Remark: All the answers so far have been very insightful and on point but after receiving public and private feedback from other mathematicians on the MathOverflow I decided to clarify a few notions ...

**2**

votes

**0**answers

46 views

### Variational forms of non-convex functions

I am trying to understand what kind of variational forms exist for non-convex functions. Alternatively, are there conjugate forms which attain strong duality? For a non-convex function $f$, I am ...

**1**

vote

**0**answers

28 views

### sequential learning reference request [closed]

I would like to find a book for master math student about the following topics.
I don't know the field and I don't want to be lost in details so if it could be an straight forward please.
I'm a ...

**8**

votes

**0**answers

90 views

### Positive definite kernels on categories

I'm wondering if there is any work on studying positive definite kernels on (the objects of a) category. By this I mean for a category $\mathcal{C}$, find a function
$$
K: Ob\mathcal{C} \times Ob\...

**0**

votes

**0**answers

47 views

### References for recent books, self-contained papers or downloadable notes that compares spectra of kernel integral operators and kernel matrices

Let $\{x_1,...x_n\}:\Omega \to \mathbb{R}^p$ be $n$ number of iid random vectors, with common probability law $P$, we can assume $dP(x)=f(x)dx$ for nice cases.
I'm trying to find some books/lecture ...

**8**

votes

**2**answers

583 views

### Physical interpretation of the Manifold Hypothesis

Motivation:
Most dimensionality reduction algorithms assume that the input data are sampled from a manifold $\mathcal{M}$ whose intrinsic dimension $d$ is much smaller than the ambient dimension $D$. ...

**0**

votes

**0**answers

317 views

### Artificial intelligence simulating mathematicians (what a distopia!)

This is kind of soft and naive question, so feel free to shame on me :)
I start from the fact that, in my opinion, what humans are interested in about mathematics are things that we find deep and ...

**23**

votes

**6**answers

4k views

### How useful is differential geometry and topology to deep learning?

After seeing this article https://www.quantamagazine.org/an-idea-from-physics-helps-ai-see-in-higher-dimensions-20200109/ I wanted to ask myself how useful of an endeavor would it be if one goes ...

**15**

votes

**1**answer

714 views

### Are primes linearly separable?

Let $X_1,\cdots,X_n$ be finite subsets of some set $Z$. Then the symmetric difference metric space:
$$d(X_i,X_j) = \sqrt{ |X_i|+|X_j|-2|X_i\cap X_j|}$$
can be embedded in Euclidean space. The value $|...

**0**

votes

**1**answer

72 views

### a probability density algorithm that is not sensitive to the initial condition

There are many algorithms to estimate the density of probability distributions. I am looking for one that is not sensitive to the initial condition. For instance, Expectation–maximization algorithm ...

**1**

vote

**1**answer

47 views

### Perturbation of the value of a general-sum game at a equilibirium

Consider a general-sum game with $N$ players. Let $u_i(a_1, \ldots, a_N)\colon \prod_{i=1}^N A_i \rightarrow \mathbb{R} $ be the payoff of the player $i\in \{ 1, \ldots, N \}$ when each player takes ...

**0**

votes

**1**answer

105 views

### Optimal solution to cross entropy loss in the continuous case

This could be a simple question but I don't have a satisfying answer.
Setup. Suppose that we have $K$ different classes, and consider cross entropy loss which maps a probability vector in the ...

**1**

vote

**0**answers

73 views

### Extension of universal approximation theorem

Let $I_d:=[0,1]^d$ with $d\ge 2$. Define $C(I_d):=\{F: I_d\to\mathbb R \mbox{ is continuous}\}$ and
$$N(I_d):=\{F\in C(I_d): F(x)=\sum_{k=1}^n f_k(v_k\cdot x), \mbox{ where } n\ge 1 \mbox{ and } f_1,\...

**0**

votes

**1**answer

124 views

### What subjects of Fourier analysis have had more effect on machine learning? [closed]

What is the salient uses of Fourier analysis in machine learning? What subjects of Fourier analysis have had more effect on machine learning?
Please mention the references.

**1**

vote

**1**answer

163 views

### Statistical model vs. Statistical Learning Theory

I am interested in the relation between a statistical model $(\Omega, \mathcal{F}, (\mathbb{P}^\theta : \theta\in\Theta))$,
where the hypotheses are "$\mathbb{P}^\theta$ is a good approximation of the ...

**1**

vote

**1**answer

102 views

### Why we use Rademacher complexity for generalization error when we can have a trained function?

Let $G$ be a family of functions mapping from $Z$ to $[a, b]$ and $S=\left(z_{1}, \ldots, z_{m}\right)$ a fixed sample of size $m$ with elements in $Z$ . Then, the empirical Rademacher complexity of $...

**12**

votes

**0**answers

489 views

### Relation between information geometry and geometric deep learning

Disclaimer: This is a cross-post from a very similar question on math.SE. I allowed myself to post it here after reading this meta
post about cross-posting between mathoverflow and math.SE, I did
...

**1**

vote

**0**answers

152 views

### Solution to a Strongly Convex Non-smooth Minimization Problem involving an L1 Norm

Let $X \in \mathbb{R}^{n \times d}, w \in \mathbb{R}^d, y \in \{\pm 1\}^{n}, \alpha \in [0,1], \lambda \in \mathbb{R}$. I have an expression that looks as follows
$\frac{1}{2}\|Xw -y \|_{2}^2 + \...

**1**

vote

**0**answers

80 views

### Plethora of variant neural networks?

Since a decade ago when new life was breathed in to neural networks in the form of deep learning a plethora of different architectures have come about. Is there a reference that gives compendium of ...

**0**

votes

**0**answers

59 views

### How to derive formula (10) norm to obtain formula (11) in Uncorrelated Group LASSO?

In Uncorrelated Group LASSO, Eq. (10) and Eq. (11) are as follows:
$J_2(W)=f(W)+\alpha Tr(W^TFW)$. (10)
$F_{ii}=\sum_{g}\frac{(I_{G_{g}})_i||W_{G_g}||_{2,1}}{||W_{G_g}^i||_2}$. (11)
where $w_{...

**1**

vote

**0**answers

68 views

### A mathematical area capable of describing nonstationary game-like problem [closed]

Here is my definition of the problem that I am trying to model:
Let's have two agents and an environment. Each agent can do two types of actions. They are either supporting the environment or don't. ...

**3**

votes

**0**answers

67 views

### Use of Asymptotics in Diffusion Maps

Question for brevity: Suppose $\varepsilon >0$ is small and that
$$
f(\varepsilon) = f_1(\varepsilon) + \mathcal{O}(\varepsilon^k)
$$
where $f_1$ has order $\varepsilon^{-\delta}$ for small fixed ...

**3**

votes

**0**answers

167 views

### How to inference the dual form of perceptron?

The model of perceptron is a linear binary classifier, which is $f(x)=\mathbb{sign}(w^Tx+b)$. $x$ is the datapoint as $w$ as well as $b$ are the parameters.
The cost function of Primal Perceptron is $...

**1**

vote

**1**answer

99 views

### Can we order random variables in a measurable way in a general setup?

Let
$(\Omega,\mathcal A,\operatorname P)$ be a probability space
$(E,\mathcal E)$ be a measurable space
$n\in\mathbb N$
$X_1,\ldots,X_n$ be $(E,\mathcal E)$-valued random variables on $(\Omega,\...

**52**

votes

**1**answer

4k views

### Mathematics of imaging the black hole

The first ever black hole was "pictured" recently, per an announcement made on 10th April, 2019. See for example: https://www.bbc.com/news/science-environment-47873592 .
It has been claimed that ...

**4**

votes

**1**answer

169 views

### Generating function for lattice paths making aribitrary (i,j)-up-right move in one step and fitting rectangular (m,n)?

There is the following beautiful formula (see Qiaochu Yuan excellent blog):
$$ \sum_{\lambda \in Young~diagrams~fitting~rectangle~m~n} q^{Box~count(="area~under~the~curve")~of~\lambda} = \binom{n+m}{...

**1**

vote

**0**answers

98 views

### Clarification about the ϵ -net argument

I have been reading the paper Do GANs learn the distribution? Some theory and empirics.
In Corollary D.1, they reference the paper Generalization and Equilibrium in Generative Adversarial Nets which ...

**1**

vote

**1**answer

59 views

### Error metric for joint estimation of mean and variance

Background:
Let $\mu:\mathbb{R}^n\to\mathbb{R}$ and $\sigma:\mathbb{R}^n\to\mathbb{R}_+$ be two unknown functions, and consider a stochastic model of the form
$$
\mathbb{E}[Y|\mathbf{x}] = \mu(\...

**5**

votes

**0**answers

164 views

### Algebraic/relational structures produced using evolutionary/machine learning algorithms?

Are there examples of algebraic structures which have been constructed using evolutionary algorithms and possibly machine learning algorithms? I am looking for algebraic structures like lattices ...

**3**

votes

**0**answers

150 views

### Simple (?) question on inner product in reproducing kernel Hilbert space

I'm following the gentle introduction to Reproducing Kernel Hilbert Spaces From Zero to Reproducing Kernel Hilbert Spaces in Twelve Pages or Less by Hal Daumé III. I believe the author fully ...

**0**

votes

**1**answer

110 views

### Cross entropy loss is not twice differentiable?

I was reading a recent theory paper in machine learning by Kenji Kawaguchi and Leslie Pack Kaelbling
https://arxiv.org/pdf/1901.00279.pdf
and the authors seem to suggest in section 2.2 that cross-...

**3**

votes

**4**answers

1k views

### Proof of Bellman optimality equation for finite Markov Decision Processes

This question has already been posed in Cross Validated without receiving a correct formal answer, so I reformulate it here to gain attention of mathematicians. I am referring to chapter 3 of Sutton ...

**9**

votes

**0**answers

197 views

### How sensitive are Neural Networks to weight change?

Let's consider the space of feedforward neural networks with a given structure: $L$ layers, $m$ neurones per layer, ReLu activation, input dimension $d$, output dimension $k$.
Which means I'm ...

**9**

votes

**1**answer

2k views

### Is There an Induction-Free Proof of the 'Be The Leader' Lemma?

This lemma is used in the context of online convex optimisation. It is sometimes called the 'Be the Leader Lemma'.
Lemma:
Suppose $f_1,f_2,\ldots,f_N$ are real valued functions with the same ...

**1**

vote

**0**answers

89 views

### Hypergraph partitioning and bipartite graph partitioning

Are hypergraph partitioning, and bipartite graph partitioning related, or equivalent, given that hypergraphs can be represented as bipartite graphs?
In the first case, we want to partition the set of ...

**0**

votes

**3**answers

120 views

### Clustering on tree

I am looking for a method that would identify clusters in a tree-like structure. In the figure below you can see a very simple example where one can visually identify distinct branches with a lot of ...

**1**

vote

**1**answer

412 views

### Minimum number of support vectors? [closed]

I'm learning SVM and its written everywhere that the minimal number of support vectors is 2? But I couldn't find any formal proofs of that. Why cant there be less than 2 support vectors? Can somebody ...

**3**

votes

**0**answers

212 views

### Universal approximation theorem for whole R^d

The well-known universal approximation theorem states that neural network with one hidden layer can approximate any continuous function on every compact subset of $\mathbb{R}^d$.
My question is ...

**0**

votes

**1**answer

75 views

### Reconstructing Euclidian space from distance matrix

The setup. Let's say that we have a set of objects $O_i$ for which we have a dissimilarity measure $M(O_1,O_2)$. With this we can build a distance matrix $D_{ij}$.
Let's also assume that we have NO ...

**4**

votes

**1**answer

366 views

### Why the Fisher information matrix is equal to the Hessian matrix of the Kullback–Leibler distance at the true parameter?

I'm reading 《Algebraic geometry and statistical learning theory》.My problem is why the Fisher information matrix is equal to the Hessian matrix of the Kullback–Leibler distance at the true parameter?...

**0**

votes

**2**answers

183 views

### Is it possible to “solve” iterative (convex/non-convex) optimization problems via learning (one-shot)?

I posted a following question in MSE, but I think it should be posted here in MO. Since I don't know how to transfer the post from MSE to MO, I have pasted the question below. Thank you in advance and ...