# Questions tagged [machine-learning]

The machine-learning tag has no usage guidance.

127
questions

1
vote

0
answers

16
views

### Convergent algorithm for minimizing nonconvex smooth function

Let $\Phi$ be the Gaussian CDF and for $\gamma\ge 0$ and $h>0$, define a loss function $\ell_h:\{\pm 1\} \times \mathbb R$ by
$$
\ell_{\gamma,h}(y,y') := \phi_{\gamma,h}(yy') := \Phi((yy'-\gamma)/h)...

0
votes

0
answers

21
views

### Normalizing a parameter in a regression

I am thinking about the possibility of making a parameter in my regression, let's say the $\lambda$ in a ridge regression, somehow, inside a range : $\lambda \in [0,1]$. Do you have any ideas how I ...

0
votes

0
answers

54
views

### Shattering of a set of binary classifiers

Let $S$ be a set, and let $\mathcal{F}_{S}=\{f:S\to\{-1,+1\}\}$ be a set of different label assignments. Show that $\mathcal{F}_{S}$ shatters at least $|\mathcal{F}_{S}|$ subsets of $S$.
Here is what ...

1
vote

0
answers

51
views

### Converting an indexed equation to a matrix one

I am helping a friend with a project involving neural networks and he wants to convert this equation into matrix notation:
$$w_{ij} = \sum_{n=1}^N\left[\sum_{i=1}^I(r_{in}-y_{in})v_{ih}\right](1-z_{hn}...

3
votes

0
answers

81
views

### What is the VC-dimension of regular convex k-gons in the plane?

Recall the relevant definitions:
Let $H$ be a family of sets in $\mathbb{R}^d$. The intersection of $H$ with a point set $C$ is defined as $H\cap C = \{h\cap C\mid h\in H\}$. The VC-dimension of $H$ (...

2
votes

1
answer

68
views

### Derive equation for regularized logistic regression with batch updates

I am trying to understand this paper by Chapelle and Li "An Empirical Evaluation of Thompson Sampling" (2011). In particular, I am failing to derive the equations in algorithm 3 (page 6). ...

4
votes

1
answer

121
views

### The ODE modeling for gradient descent with decreasing step sizes

The gradient descent (GD) with constant stepsize $\alpha^{k}=\alpha$ takes the form
$$x^{k+1} = x^{k} -\alpha\nabla f(x^{k}).$$
Then, by constructing a continuous-time version of GD iterates ...

2
votes

1
answer

94
views

### Representer theorem for a loss / functional of the form $L(h) := \sum_{i=1}^n (|h(x_i)-y_i|+t\|h\|)^2$

Let $K:X \times X \to \mathbb R$ be a (positive-definite) kernel and let $H$ be the induced reproducing kernel Hilbert space (RKHS). Fix $(x_1,y_1),\ldots,(x_n,y_n) \in X \times \mathbb R$. For $t \ge ...

1
vote

0
answers

27
views

### Correlating two matrices $A,B$ with stochastic dependency structure imposed by cross-validation

Consider a labelled data set
$$D = \{(x_1, y_1),...,(x_n, y_n)\} $$
on which we want to evaluate a machine learning algorithm using $k$-fold cross validation with $m$ different random seeds. This ...

2
votes

1
answer

54
views

### How to fit a set of parametrized data to a parametrized distribution?

I have a time series $d_i(a)$ which depends on the parameter $a$. On the other hand, I have a sequence of normal distributions $\mathcal{N}(0,Q_i(a))$, where the variance $Q_i$ depends on time and ...

2
votes

0
answers

29
views

### Stochastic gradient descent in 'stronger' settings

I am minimzing a function $F(x) = \mathbb E(f(x,\Xi))$ where $\Xi$ is some random value, by a stochastic gradient descent that generates a random number $\xi$ from the distribution of $\Xi$ at each ...

6
votes

0
answers

214
views

### Mathematical questions or areas amenable to AI [duplicate]

This question regards the new paper "Advancing mathematics by guiding human intuition with AI" by Davies et al. (Nature, 2021) (DOI link in open access) in which researchers at Deepmind ...

6
votes

0
answers

261
views

### Signature and cusp geometry of hyperbolic knots

Nature recently published a paper titled “Advancing mathematics by guiding human intuition with AI”. Using the power of linear algebra and calculus machine learning, the authors link "geometric&...

2
votes

0
answers

64
views

### A question about fundamental invariants in the context of neural networks

I'm reading in depth the first part of the following paper: https://arxiv.org/pdf/1804.10306.pdf, paying specific attention to the following result, that I re-write here for the sake of convenience:
[...

0
votes

0
answers

28
views

### Regrouping the leaf nodes of the WordNet DAG

Motivation
I am trying to find a criterion to regroup the classes of the ImageNet challenge dataset, one of the most important datasets used in Machine Learning.
The ImageNet dataset has 1000 classes ...

23
votes

1
answer

1k
views

### Conjectures inspired by AI

Today in Nature a paper described how AI guided mathematicians to make highly non-trivial conjectures, which they managed to prove, one in Knot Theory involving a new invariant, the other in ...

12
votes

2
answers

2k
views

### Is it ever unnecessary to mathematically formalize a concept?

From my understanding, mathematics sometimes gives rise to new physical/tangible laws and the converse is also true. In particular, physical phenomena give rise to new mathematics.
In all of the cases ...

9
votes

3
answers

800
views

### Deep learning for knot theory. Classification

As far as I know, there is a classification of all prime knots with less than 16 crossings.
It seems that there is already a fast enough algorithm to distinguish a knot from an unknot.
So in principle ...

0
votes

1
answer

117
views

### Why is squared exponential kernel often used in Gaussian Process regression when the most standard case is time-like X?

I might be confused about something.
Consider doing inference on $Y'\mid X',Y,X$ using standard Gaussian Process Regression with 1d $Y$ and 1d $X$. Suppose $X$ is time-like (target is stationary or ...

15
votes

3
answers

1k
views

### Theoretical results on neural networks

With this question I'd like to have a recollection of theoretical rigorous results on neural networks.
I'd like to have results that have been settled, as opposed to hypothesis. As an example, this ...

1
vote

1
answer

189
views

### VC-dimensions of functions from real numbers to {0,1}

I'm currently studying VC-dimensions. Suppose I have the hypothesis class $H$, where $H = \{ h \in \mathbb{R} \rightarrow \{ 0,1 \} \}$. I think this means that the hypothesis class $H$ is a set of ...

9
votes

1
answer

327
views

### Neural networks over gadgets other than $\mathbb{R}$

Recently, I learned that neural networks (NN) can be defined over fields other than $\mathbb{R}$: for example, Khrennikov and Tirozzi wrote a paper in 1999 (!) on $p$-adic neural networks, or neural ...

1
vote

0
answers

79
views

### Continuous decomposition of permutation-invariant set functions

The seminal machine learning paper Deep Sets (Zaheer et al., 2017) discusses representations of permutation-invariant functions on real tuples, or (multi)set functions.
Given a countable set $X$ and a ...

2
votes

0
answers

58
views

### What is known about gradient descent on quadratic models (not loss functions!)

Let $\mathcal X$ be any set, and $f:\mathcal X\times\mathbb R^n\to\mathbb R$ be a differentiable model, meaning that for any fixed first argument, $f$ is differentiable in its second argument. Then we ...

0
votes

0
answers

57
views

### EM algorithm for Factor Analysis

I've just learned EM algorithm and Factor Analysis. However, when it comes to applying EM algorithm to Factor analysis, I get confused. It would be much appreciated if someone could help me out.
Based ...

4
votes

1
answer

171
views

### Why is this nonlinear transformation of an RKHS also an RKHS?

I came across this paper (beginning of page 6) where they stated that if $f,h\in \mathcal{H}$, where $\mathcal{H}$ is an RKHS, then $l_{h,f}=\left|f(x)-h(x)\right|^q$ where $q\geq 1$ also belongs to ...

7
votes

0
answers

208
views

### Monotonicity of log determinant of Gaussian kernel matrix

Let \begin{equation} k({x},{y}) = \sigma \exp\left(-\frac{(x-y)^2}{2\theta^2}\right)\end{equation}
be a squared-exponential (Gaussian) kernel, with $\sigma,\vartheta>0$. Consider, for a set of $N$ ...

75
votes

10
answers

9k
views

### What are possible applications of deep learning to research mathematics?

With no doubt everyone here has heard of deep learning, even if they don't know what it is or what it is good for. I myself am a former mathematician turned data scientist who is quite interested in ...

13
votes

1
answer

2k
views

### Various authors of the Bourbaki's books

As far as I understand, each chapter of the Bourbaki's collection was written by one (or two?) specific authors. The book itself was reviewed, corrected and after all approved by the whole Bourbaki ...

3
votes

1
answer

230
views

### Ricci flow for manifold learning

I know that mean curvature and diffusion-type flows are common in manifold learning because of their smoothing effects. I haven't seen Ricci flow used as much. Given that Ricci and diffusion-type ...

0
votes

0
answers

231
views

### Derivative of the function of random variable

Suppose we have a function $\phi(X)$ of random variable $X$. Suppose both of $\phi(X)$ and $X$ are random variables. If $\phi$ is differentiable, how to calculate the derivative of $\phi(X)$ w.r.t. $...

1
vote

1
answer

66
views

### Bayes risk of binary classification problem with conditionally independet covariates

In the setting of this problem, $\eta(\vec{x})$ is $P(Y=1|\vec{X}=\vec{x})$, $Y \in {0,1}$, $X \in R^d$. Being the true probability know, the classification rule is simply $\eta(\vec{x})>0.5 \...

0
votes

0
answers

111
views

### UCB1 for Multiarmed bandits: understanding the proof

Regarding Multiarmed Bandits, I am analysing algorithm UCB1 presented in this paper and specifically the proof of Theorem 1, concerning the regret of the algorithm. At page 243 there is a step I do ...

7
votes

1
answer

265
views

### Iterating projections to random halfspaces

Consider the following process:
Start with a set $S = \mathbb R^n$. Repeat $L$ times: choose a random orthonormal basis $u_1, \ldots, u_n$, and consider the cone $C = \{ \sum \alpha_i u_i : \alpha_i \...

3
votes

1
answer

219
views

### Games and the right mathematical framework for GANs

Generative Adversarial Networks were introduced in http://papers.nips.cc/paper/5423-generative-adversarial-nets and has more than 20000 citations.
It is an important topic within deep learning.
Are ...

7
votes

2
answers

860
views

### Mathematics of GANs (generative adversarial networks)

Generative Adversarial Networks were introduced in http://papers.nips.cc/paper/5423-generative-adversarial-nets and has more than 20000 citations.
The paper introduced key paradigm changes which ...

0
votes

1
answer

2k
views

### In linear regression, we have 0 training error if data dimension is high, but are there similar results for other supervised learning problems?

I tried posting this question on Cross Validated (the stack exchange for statistics) but didn't get an answer, so posting here:
Let's consider a supervised learning problem where $\{(x_1,y_1) \dots (...

1
vote

1
answer

166
views

### Finite VC dimension > the number of free parameters

I'm looking for an example of the following:
A hypothesis class $\mathcal{H}$ such that
$\forall h \in \mathcal{H}$, the number of free parameters of $h$ is equal to $n \in \mathbb{N}$ (where $n < ...

2
votes

0
answers

47
views

### What are some beginner's references on algebraically structured (statistical) models, and their connection with group actions and Fourier transform?

I asked this question on Cross Validated a few days ago, but didn't really get a favorable response, so asking here to see if I get any.
I'm looking at the description of a short-term position in ...

1
vote

0
answers

63
views

### Relation between minimizer of regularized risk & risk in statistical learning theory

In supervised machine learning, we typically take a Risk Minimization (RM) point of view when formulating a problem. So, what we typically solve for is the following:
$$ R^L(h) = \underset{h\in\...

0
votes

1
answer

247
views

### How large sample $m$ is enough [closed]

I have a $D$ probability distribution over $X =R^d$, i have two samples $s_1$ and $s_2$ from $D$, each having size $m_1$, $m_2$, a unit ball centered at origin $B(0)$, defined by $B(0)=\{x \in R^2: \|...

4
votes

0
answers

104
views

### Can we show equivalence of two distributions based on their statistics?

Let $p,q$ be two distributions on $\mathbb{R}^d$. Let $f:\mathbb{R}^d\times\mathbb{R}^d\rightarrow\mathbb{R}$. Under what conditions does $\mathbb{E}_{x\sim p}f(x,z)=\mathbb{E}_{x\sim q}f(x,z)\ \...

0
votes

1
answer

111
views

### Using a poset or directed graph as input for a neural network [closed]

I'm not sure if this is the right community to post this in but I would appreciate any help. As the title states, I'm trying to train a neural network using some unconventional input. I'm wondering if ...

1
vote

0
answers

36
views

### Restriction of Rademacher Complexity

Let $F\subseteq C([0,1]^n,\mathbb{R})$ be a finite family of functions, which is non-empty. Let $A,B$ be subseteq of $[0,1]^n$, again non-empty, and let $Rad(C)$ denote the Rademacher complexity of ...

1
vote

0
answers

61
views

### Bayesian inference of stochastically evolving model parameters

I have a question related to self-calibration in radio interferometry, but I will try to phrase it as generic as possible. I have a set of data points, $D = \{ d_{0, t_0}, d_{1, t_0}, ..., d_{M, t_0}, ...

0
votes

0
answers

52
views

### Relation between test and train error with gradient descent iterates

My question is about establishing an inequality between population error and expected training error (i.e, expected training error < population error) for a model trained with gradient descent on a ...

1
vote

1
answer

641
views

### References for "second order" random walk on graphs (used in "node2vec" paper)?

The "word2vec" family of methods provided a great breakthrough in natural language processing.
The methods assign to each word a vector in $R^{n}$, such that the "similar" words ...

5
votes

0
answers

134
views

### Distance of two points in Grassmannian using Plücker coordinate

Let $G(q,D)$ be the Grassmannian of $q$-dimensional vector spaces in $\mathbb{R}^D$, where $q \le D$ are positive integers. In the paper, a distance between two points of $G(q,D)$ are defined as ...

8
votes

4
answers

2k
views

### How to learn a continuous function?

Let $\Omega \subset \mathbb{R}^m$ be an open subset bounded with a smooth boundary.
Problem : Given any bounded continuous function $f:\Omega\to\mathbb{R}$, can we learn it to a given accuracy $\...

4
votes

1
answer

133
views

### Hermite polynomial after rotation

When we consider the $n$-dimensional standard normal distribution, the orthogonal basis is $\{H_S(x)\}_{S}$ where $H_S(x) = \prod_{k=1}^n H_{s_k}(x_k)$. Here $H_*(x)$ is the normalized probabilist's ...