Questions tagged [machine-learning]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2
votes
1answer
43 views

Uniform Lipschitz function approximation by shallow neural networks

Fix $d\in \mathbb{N}$. Let $F_1$ be the set of all 1-Lipschitz functions mapping $[0, 1]^d$ to $\mathbb{R}$. For $\varphi: \mathbb{R} \rightarrow \mathbb{R}$ and $m \in \mathbb{N}$, let $N_\varphi^m$ ...
0
votes
1answer
88 views

Independence in a sequential problem with observations getting added to buckets

Consider a sequence of random observations $(O(t))_{t\geq 1}$, with $O(t)=(D(t),J(t),Y(t))$. Denote $\mathcal{F}(t) := \sigma(O(1),\ldots,O(t))$, the filtration induced by the first $t$ observations. ...
7
votes
1answer
120 views

Books to develop a unified view of statistics and information theory?

I hope to understand the connection between statistics and information theory in a deep philosophical sense. I suppose the best place to start would be David MacKay's Information Theory, Inference, ...
3
votes
1answer
161 views

Abstract mathematical concepts/tools appeared in machine learning research

I am interested in knowing about abstract mathematical concepts, tools or methods that have come up in theoretical machine learning. By "abstract" I mean something that is not immediately related to ...
2
votes
0answers
93 views

Inequality on the Kullback-Leibler divergence

Let us define the arithmetic, geometric, and harmonic means of $x,y \in \mathbb{R}$ weighted by $\alpha =(\alpha_x,\alpha_y) \in [0,1]$, respectively as \begin{equation} a_\alpha(x,y) = \frac{\...
0
votes
2answers
138 views

Statistical divergence

Does anyone know about a statistical divergence of this type? \begin{equation} \text{D}(P||Q) = \frac{1}{2} \left[\text{KL}(M||P) + \text{KL}(M||Q)\right] \end{equation} where $M = \frac{1}{2} [P+Q]$....
1
vote
0answers
53 views

Comparison of concentrations of different $L^p$-norms of (sub) Gaussian distributions

It's well-known that the Euclidean $2$-norm of subgaussian random vectors concentrates in high dimensions, e.g. when $X \sim \mathcal{N}(0,I_n),$ (or in general $X$ is subgaussian with independent co-...
0
votes
2answers
160 views

Help with a definition of a two-person game in a referenced paper

In the paper "Finding Mixed Nash Equilibria of Generative Adversarial Networks" the authors write in equation (1) on page 2: Consider the classical formulation of a two-player game with finitely ...
0
votes
0answers
29 views

What is the number of iterations needed for the message passing algorithm to converge when applied to an acyclic factor graph?

I understand that the message passing algorithm (Belief Propagation algorithm), when applied to a factor graph consists in an exchange in messages between the factor nodes and the variable nodes, ...
8
votes
4answers
2k views

mathematical physics without partial derivatives

Remark: All the answers so far have been very insightful and on point but after receiving public and private feedback from other mathematicians on the MathOverflow I decided to clarify a few notions ...
2
votes
0answers
46 views

Variational forms of non-convex functions

I am trying to understand what kind of variational forms exist for non-convex functions. Alternatively, are there conjugate forms which attain strong duality? For a non-convex function $f$, I am ...
1
vote
0answers
28 views

sequential learning reference request [closed]

I would like to find a book for master math student about the following topics. I don't know the field and I don't want to be lost in details so if it could be an straight forward please. I'm a ...
8
votes
0answers
90 views

Positive definite kernels on categories

I'm wondering if there is any work on studying positive definite kernels on (the objects of a) category. By this I mean for a category $\mathcal{C}$, find a function $$ K: Ob\mathcal{C} \times Ob\...
0
votes
0answers
47 views

References for recent books, self-contained papers or downloadable notes that compares spectra of kernel integral operators and kernel matrices

Let $\{x_1,...x_n\}:\Omega \to \mathbb{R}^p$ be $n$ number of iid random vectors, with common probability law $P$, we can assume $dP(x)=f(x)dx$ for nice cases. I'm trying to find some books/lecture ...
8
votes
2answers
583 views

Physical interpretation of the Manifold Hypothesis

Motivation: Most dimensionality reduction algorithms assume that the input data are sampled from a manifold $\mathcal{M}$ whose intrinsic dimension $d$ is much smaller than the ambient dimension $D$. ...
0
votes
0answers
317 views

Artificial intelligence simulating mathematicians (what a distopia!)

This is kind of soft and naive question, so feel free to shame on me :) I start from the fact that, in my opinion, what humans are interested in about mathematics are things that we find deep and ...
23
votes
6answers
4k views

How useful is differential geometry and topology to deep learning?

After seeing this article https://www.quantamagazine.org/an-idea-from-physics-helps-ai-see-in-higher-dimensions-20200109/ I wanted to ask myself how useful of an endeavor would it be if one goes ...
15
votes
1answer
714 views

Are primes linearly separable?

Let $X_1,\cdots,X_n$ be finite subsets of some set $Z$. Then the symmetric difference metric space: $$d(X_i,X_j) = \sqrt{ |X_i|+|X_j|-2|X_i\cap X_j|}$$ can be embedded in Euclidean space. The value $|...
0
votes
1answer
72 views

a probability density algorithm that is not sensitive to the initial condition

There are many algorithms to estimate the density of probability distributions. I am looking for one that is not sensitive to the initial condition. For instance, Expectation–maximization algorithm ...
1
vote
1answer
47 views

Perturbation of the value of a general-sum game at a equilibirium

Consider a general-sum game with $N$ players. Let $u_i(a_1, \ldots, a_N)\colon \prod_{i=1}^N A_i \rightarrow \mathbb{R} $ be the payoff of the player $i\in \{ 1, \ldots, N \}$ when each player takes ...
0
votes
1answer
105 views

Optimal solution to cross entropy loss in the continuous case

This could be a simple question but I don't have a satisfying answer. Setup. Suppose that we have $K$ different classes, and consider cross entropy loss which maps a probability vector in the ...
1
vote
0answers
73 views

Extension of universal approximation theorem

Let $I_d:=[0,1]^d$ with $d\ge 2$. Define $C(I_d):=\{F: I_d\to\mathbb R \mbox{ is continuous}\}$ and $$N(I_d):=\{F\in C(I_d): F(x)=\sum_{k=1}^n f_k(v_k\cdot x), \mbox{ where } n\ge 1 \mbox{ and } f_1,\...
0
votes
1answer
124 views

What subjects of Fourier analysis have had more effect on machine learning? [closed]

What is the salient uses of Fourier analysis in machine learning? What subjects of Fourier analysis have had more effect on machine learning? Please mention the references.
1
vote
1answer
163 views

Statistical model vs. Statistical Learning Theory

I am interested in the relation between a statistical model $(\Omega, \mathcal{F}, (\mathbb{P}^\theta : \theta\in\Theta))$, where the hypotheses are "$\mathbb{P}^\theta$ is a good approximation of the ...
1
vote
1answer
102 views

Why we use Rademacher complexity for generalization error when we can have a trained function?

Let $G$ be a family of functions mapping from $Z$ to $[a, b]$ and $S=\left(z_{1}, \ldots, z_{m}\right)$ a fixed sample of size $m$ with elements in $Z$ . Then, the empirical Rademacher complexity of $...
12
votes
0answers
489 views

Relation between information geometry and geometric deep learning

Disclaimer: This is a cross-post from a very similar question on math.SE. I allowed myself to post it here after reading this meta post about cross-posting between mathoverflow and math.SE, I did ...
1
vote
0answers
152 views

Solution to a Strongly Convex Non-smooth Minimization Problem involving an L1 Norm

Let $X \in \mathbb{R}^{n \times d}, w \in \mathbb{R}^d, y \in \{\pm 1\}^{n}, \alpha \in [0,1], \lambda \in \mathbb{R}$. I have an expression that looks as follows $\frac{1}{2}\|Xw -y \|_{2}^2 + \...
1
vote
0answers
80 views

Plethora of variant neural networks?

Since a decade ago when new life was breathed in to neural networks in the form of deep learning a plethora of different architectures have come about. Is there a reference that gives compendium of ...
0
votes
0answers
59 views

How to derive formula (10) norm to obtain formula (11) in Uncorrelated Group LASSO?

In Uncorrelated Group LASSO, Eq. (10) and Eq. (11) are as follows: $J_2(W)=f(W)+\alpha Tr(W^TFW)$. (10) $F_{ii}=\sum_{g}\frac{(I_{G_{g}})_i||W_{G_g}||_{2,1}}{||W_{G_g}^i||_2}$. (11) where $w_{...
1
vote
0answers
68 views

A mathematical area capable of describing nonstationary game-like problem [closed]

Here is my definition of the problem that I am trying to model: Let's have two agents and an environment. Each agent can do two types of actions. They are either supporting the environment or don't. ...
3
votes
0answers
67 views

Use of Asymptotics in Diffusion Maps

Question for brevity: Suppose $\varepsilon >0$ is small and that $$ f(\varepsilon) = f_1(\varepsilon) + \mathcal{O}(\varepsilon^k) $$ where $f_1$ has order $\varepsilon^{-\delta}$ for small fixed ...
3
votes
0answers
167 views

How to inference the dual form of perceptron?

The model of perceptron is a linear binary classifier, which is $f(x)=\mathbb{sign}(w^Tx+b)$. $x$ is the datapoint as $w$ as well as $b$ are the parameters. The cost function of Primal Perceptron is $...
1
vote
1answer
99 views

Can we order random variables in a measurable way in a general setup?

Let $(\Omega,\mathcal A,\operatorname P)$ be a probability space $(E,\mathcal E)$ be a measurable space $n\in\mathbb N$ $X_1,\ldots,X_n$ be $(E,\mathcal E)$-valued random variables on $(\Omega,\...
52
votes
1answer
4k views

Mathematics of imaging the black hole

The first ever black hole was "pictured" recently, per an announcement made on 10th April, 2019. See for example: https://www.bbc.com/news/science-environment-47873592 . It has been claimed that ...
4
votes
1answer
169 views

Generating function for lattice paths making aribitrary (i,j)-up-right move in one step and fitting rectangular (m,n)?

There is the following beautiful formula (see Qiaochu Yuan excellent blog): $$ \sum_{\lambda \in Young~diagrams~fitting~rectangle~m~n} q^{Box~count(="area~under~the~curve")~of~\lambda} = \binom{n+m}{...
1
vote
0answers
98 views

Clarification about the ϵ -net argument

I have been reading the paper Do GANs learn the distribution? Some theory and empirics. In Corollary D.1, they reference the paper Generalization and Equilibrium in Generative Adversarial Nets which ...
1
vote
1answer
59 views

Error metric for joint estimation of mean and variance

Background: Let $\mu:\mathbb{R}^n\to\mathbb{R}$ and $\sigma:\mathbb{R}^n\to\mathbb{R}_+$ be two unknown functions, and consider a stochastic model of the form $$ \mathbb{E}[Y|\mathbf{x}] = \mu(\...
5
votes
0answers
164 views

Algebraic/relational structures produced using evolutionary/machine learning algorithms?

Are there examples of algebraic structures which have been constructed using evolutionary algorithms and possibly machine learning algorithms? I am looking for algebraic structures like lattices ...
3
votes
0answers
150 views

Simple (?) question on inner product in reproducing kernel Hilbert space

I'm following the gentle introduction to Reproducing Kernel Hilbert Spaces From Zero to Reproducing Kernel Hilbert Spaces in Twelve Pages or Less by Hal Daumé III. I believe the author fully ...
0
votes
1answer
110 views

Cross entropy loss is not twice differentiable?

I was reading a recent theory paper in machine learning by Kenji Kawaguchi and Leslie Pack Kaelbling https://arxiv.org/pdf/1901.00279.pdf and the authors seem to suggest in section 2.2 that cross-...
3
votes
4answers
1k views

Proof of Bellman optimality equation for finite Markov Decision Processes

This question has already been posed in Cross Validated without receiving a correct formal answer, so I reformulate it here to gain attention of mathematicians. I am referring to chapter 3 of Sutton ...
9
votes
0answers
197 views

How sensitive are Neural Networks to weight change?

Let's consider the space of feedforward neural networks with a given structure: $L$ layers, $m$ neurones per layer, ReLu activation, input dimension $d$, output dimension $k$. Which means I'm ...
9
votes
1answer
2k views

Is There an Induction-Free Proof of the 'Be The Leader' Lemma?

This lemma is used in the context of online convex optimisation. It is sometimes called the 'Be the Leader Lemma'. Lemma: Suppose $f_1,f_2,\ldots,f_N$ are real valued functions with the same ...
1
vote
0answers
89 views

Hypergraph partitioning and bipartite graph partitioning

Are hypergraph partitioning, and bipartite graph partitioning related, or equivalent, given that hypergraphs can be represented as bipartite graphs? In the first case, we want to partition the set of ...
0
votes
3answers
120 views

Clustering on tree

I am looking for a method that would identify clusters in a tree-like structure. In the figure below you can see a very simple example where one can visually identify distinct branches with a lot of ...
1
vote
1answer
412 views

Minimum number of support vectors? [closed]

I'm learning SVM and its written everywhere that the minimal number of support vectors is 2? But I couldn't find any formal proofs of that. Why cant there be less than 2 support vectors? Can somebody ...
3
votes
0answers
212 views

Universal approximation theorem for whole R^d

The well-known universal approximation theorem states that neural network with one hidden layer can approximate any continuous function on every compact subset of $\mathbb{R}^d$. My question is ...
0
votes
1answer
75 views

Reconstructing Euclidian space from distance matrix

The setup. Let's say that we have a set of objects $O_i$ for which we have a dissimilarity measure $M(O_1,O_2)$. With this we can build a distance matrix $D_{ij}$. Let's also assume that we have NO ...
4
votes
1answer
366 views

Why the Fisher information matrix is equal to the Hessian matrix of the Kullback–Leibler distance at the true parameter?

I'm reading 《Algebraic geometry and statistical learning theory》.My problem is why the Fisher information matrix is equal to the Hessian matrix of the Kullback–Leibler distance at the true parameter?...
0
votes
2answers
183 views

Is it possible to “solve” iterative (convex/non-convex) optimization problems via learning (one-shot)?

I posted a following question in MSE, but I think it should be posted here in MO. Since I don't know how to transfer the post from MSE to MO, I have pasted the question below. Thank you in advance and ...