Questions tagged [machine-learning]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
16 views

Convergent algorithm for minimizing nonconvex smooth function

Let $\Phi$ be the Gaussian CDF and for $\gamma\ge 0$ and $h>0$, define a loss function $\ell_h:\{\pm 1\} \times \mathbb R$ by $$ \ell_{\gamma,h}(y,y') := \phi_{\gamma,h}(yy') := \Phi((yy'-\gamma)/h)...
user avatar
  • 5,580
0 votes
0 answers
21 views

Normalizing a parameter in a regression

I am thinking about the possibility of making a parameter in my regression, let's say the $\lambda$ in a ridge regression, somehow, inside a range : $\lambda \in [0,1]$. Do you have any ideas how I ...
user avatar
0 votes
0 answers
54 views

Shattering of a set of binary classifiers

Let $S$ be a set, and let $\mathcal{F}_{S}=\{f:S\to\{-1,+1\}\}$ be a set of different label assignments. Show that $\mathcal{F}_{S}$ shatters at least $|\mathcal{F}_{S}|$ subsets of $S$. Here is what ...
user avatar
  • 101
1 vote
0 answers
51 views

Converting an indexed equation to a matrix one

I am helping a friend with a project involving neural networks and he wants to convert this equation into matrix notation: $$w_{ij} = \sum_{n=1}^N\left[\sum_{i=1}^I(r_{in}-y_{in})v_{ih}\right](1-z_{hn}...
user avatar
3 votes
0 answers
81 views

What is the VC-dimension of regular convex k-gons in the plane?

Recall the relevant definitions: Let $H$ be a family of sets in $\mathbb{R}^d$. The intersection of $H$ with a point set $C$ is defined as $H\cap C = \{h\cap C\mid h\in H\}$. The VC-dimension of $H$ (...
user avatar
  • 131
2 votes
1 answer
68 views

Derive equation for regularized logistic regression with batch updates

I am trying to understand this paper by Chapelle and Li "An Empirical Evaluation of Thompson Sampling" (2011). In particular, I am failing to derive the equations in algorithm 3 (page 6). ...
user avatar
4 votes
1 answer
121 views

The ODE modeling for gradient descent with decreasing step sizes

The gradient descent (GD) with constant stepsize $\alpha^{k}=\alpha$ takes the form $$x^{k+1} = x^{k} -\alpha\nabla f(x^{k}).$$ Then, by constructing a continuous-time version of GD iterates ...
user avatar
  • 63
2 votes
1 answer
94 views

Representer theorem for a loss / functional of the form $L(h) := \sum_{i=1}^n (|h(x_i)-y_i|+t\|h\|)^2$

Let $K:X \times X \to \mathbb R$ be a (positive-definite) kernel and let $H$ be the induced reproducing kernel Hilbert space (RKHS). Fix $(x_1,y_1),\ldots,(x_n,y_n) \in X \times \mathbb R$. For $t \ge ...
user avatar
  • 5,580
1 vote
0 answers
27 views

Correlating two matrices $A,B$ with stochastic dependency structure imposed by cross-validation

Consider a labelled data set $$D = \{(x_1, y_1),...,(x_n, y_n)\} $$ on which we want to evaluate a machine learning algorithm using $k$-fold cross validation with $m$ different random seeds. This ...
user avatar
  • 153
2 votes
1 answer
54 views

How to fit a set of parametrized data to a parametrized distribution?

I have a time series $d_i(a)$ which depends on the parameter $a$. On the other hand, I have a sequence of normal distributions $\mathcal{N}(0,Q_i(a))$, where the variance $Q_i$ depends on time and ...
user avatar
2 votes
0 answers
29 views

Stochastic gradient descent in 'stronger' settings

I am minimzing a function $F(x) = \mathbb E(f(x,\Xi))$ where $\Xi$ is some random value, by a stochastic gradient descent that generates a random number $\xi$ from the distribution of $\Xi$ at each ...
user avatar
  • 623
6 votes
0 answers
214 views

Mathematical questions or areas amenable to AI [duplicate]

This question regards the new paper "Advancing mathematics by guiding human intuition with AI" by Davies et al. (Nature, 2021) (DOI link in open access) in which researchers at Deepmind ...
6 votes
0 answers
261 views

Signature and cusp geometry of hyperbolic knots

Nature recently published a paper titled “Advancing mathematics by guiding human intuition with AI”. Using the power of linear algebra and calculus machine learning, the authors link "geometric&...
user avatar
  • 2,113
2 votes
0 answers
64 views

A question about fundamental invariants in the context of neural networks

I'm reading in depth the first part of the following paper: https://arxiv.org/pdf/1804.10306.pdf, paying specific attention to the following result, that I re-write here for the sake of convenience: [...
user avatar
0 votes
0 answers
28 views

Regrouping the leaf nodes of the WordNet DAG

Motivation I am trying to find a criterion to regroup the classes of the ImageNet challenge dataset, one of the most important datasets used in Machine Learning. The ImageNet dataset has 1000 classes ...
user avatar
23 votes
1 answer
1k views

Conjectures inspired by AI

Today in Nature a paper described how AI guided mathematicians to make highly non-trivial conjectures, which they managed to prove, one in Knot Theory involving a new invariant, the other in ...
12 votes
2 answers
2k views

Is it ever unnecessary to mathematically formalize a concept?

From my understanding, mathematics sometimes gives rise to new physical/tangible laws and the converse is also true. In particular, physical phenomena give rise to new mathematics. In all of the cases ...
user avatar
9 votes
3 answers
800 views

Deep learning for knot theory. Classification

As far as I know, there is a classification of all prime knots with less than 16 crossings. It seems that there is already a fast enough algorithm to distinguish a knot from an unknot. So in principle ...
user avatar
  • 321
0 votes
1 answer
117 views

Why is squared exponential kernel often used in Gaussian Process regression when the most standard case is time-like X?

I might be confused about something. Consider doing inference on $Y'\mid X',Y,X$ using standard Gaussian Process Regression with 1d $Y$ and 1d $X$. Suppose $X$ is time-like (target is stationary or ...
user avatar
  • 313
15 votes
3 answers
1k views

Theoretical results on neural networks

With this question I'd like to have a recollection of theoretical rigorous results on neural networks. I'd like to have results that have been settled, as opposed to hypothesis. As an example, this ...
1 vote
1 answer
189 views

VC-dimensions of functions from real numbers to {0,1}

I'm currently studying VC-dimensions. Suppose I have the hypothesis class $H$, where $H = \{ h \in \mathbb{R} \rightarrow \{ 0,1 \} \}$. I think this means that the hypothesis class $H$ is a set of ...
user avatar
  • 11
9 votes
1 answer
327 views

Neural networks over gadgets other than $\mathbb{R}$

Recently, I learned that neural networks (NN) can be defined over fields other than $\mathbb{R}$: for example, Khrennikov and Tirozzi wrote a paper in 1999 (!) on $p$-adic neural networks, or neural ...
user avatar
  • 561
1 vote
0 answers
79 views

Continuous decomposition of permutation-invariant set functions

The seminal machine learning paper Deep Sets (Zaheer et al., 2017) discusses representations of permutation-invariant functions on real tuples, or (multi)set functions. Given a countable set $X$ and a ...
user avatar
2 votes
0 answers
58 views

What is known about gradient descent on quadratic models (not loss functions!)

Let $\mathcal X$ be any set, and $f:\mathcal X\times\mathbb R^n\to\mathbb R$ be a differentiable model, meaning that for any fixed first argument, $f$ is differentiable in its second argument. Then we ...
user avatar
  • 623
0 votes
0 answers
57 views

EM algorithm for Factor Analysis

I've just learned EM algorithm and Factor Analysis. However, when it comes to applying EM algorithm to Factor analysis, I get confused. It would be much appreciated if someone could help me out. Based ...
user avatar
4 votes
1 answer
171 views

Why is this nonlinear transformation of an RKHS also an RKHS?

I came across this paper (beginning of page 6) where they stated that if $f,h\in \mathcal{H}$, where $\mathcal{H}$ is an RKHS, then $l_{h,f}=\left|f(x)-h(x)\right|^q$ where $q\geq 1$ also belongs to ...
user avatar
7 votes
0 answers
208 views

Monotonicity of log determinant of Gaussian kernel matrix

Let \begin{equation} k({x},{y}) = \sigma \exp\left(-\frac{(x-y)^2}{2\theta^2}\right)\end{equation} be a squared-exponential (Gaussian) kernel, with $\sigma,\vartheta>0$. Consider, for a set of $N$ ...
user avatar
75 votes
10 answers
9k views

What are possible applications of deep learning to research mathematics?

With no doubt everyone here has heard of deep learning, even if they don't know what it is or what it is good for. I myself am a former mathematician turned data scientist who is quite interested in ...
13 votes
1 answer
2k views

Various authors of the Bourbaki's books

As far as I understand, each chapter of the Bourbaki's collection was written by one (or two?) specific authors. The book itself was reviewed, corrected and after all approved by the whole Bourbaki ...
user avatar
  • 6,159
3 votes
1 answer
230 views

Ricci flow for manifold learning

I know that mean curvature and diffusion-type flows are common in manifold learning because of their smoothing effects. I haven't seen Ricci flow used as much. Given that Ricci and diffusion-type ...
user avatar
  • 143
0 votes
0 answers
231 views

Derivative of the function of random variable

Suppose we have a function $\phi(X)$ of random variable $X$. Suppose both of $\phi(X)$ and $X$ are random variables. If $\phi$ is differentiable, how to calculate the derivative of $\phi(X)$ w.r.t. $...
user avatar
1 vote
1 answer
66 views

Bayes risk of binary classification problem with conditionally independet covariates

In the setting of this problem, $\eta(\vec{x})$ is $P(Y=1|\vec{X}=\vec{x})$, $Y \in {0,1}$, $X \in R^d$. Being the true probability know, the classification rule is simply $\eta(\vec{x})>0.5 \...
user avatar
0 votes
0 answers
111 views

UCB1 for Multiarmed bandits: understanding the proof

Regarding Multiarmed Bandits, I am analysing algorithm UCB1 presented in this paper and specifically the proof of Theorem 1, concerning the regret of the algorithm. At page 243 there is a step I do ...
user avatar
  • 1
7 votes
1 answer
265 views

Iterating projections to random halfspaces

Consider the following process: Start with a set $S = \mathbb R^n$. Repeat $L$ times: choose a random orthonormal basis $u_1, \ldots, u_n$, and consider the cone $C = \{ \sum \alpha_i u_i : \alpha_i \...
user avatar
3 votes
1 answer
219 views

Games and the right mathematical framework for GANs

Generative Adversarial Networks were introduced in http://papers.nips.cc/paper/5423-generative-adversarial-nets and has more than 20000 citations. It is an important topic within deep learning. Are ...
user avatar
  • 12.8k
7 votes
2 answers
860 views

Mathematics of GANs (generative adversarial networks)

Generative Adversarial Networks were introduced in http://papers.nips.cc/paper/5423-generative-adversarial-nets and has more than 20000 citations. The paper introduced key paradigm changes which ...
user avatar
  • 12.8k
0 votes
1 answer
2k views

In linear regression, we have 0 training error if data dimension is high, but are there similar results for other supervised learning problems?

I tried posting this question on Cross Validated (the stack exchange for statistics) but didn't get an answer, so posting here: Let's consider a supervised learning problem where $\{(x_1,y_1) \dots (...
user avatar
1 vote
1 answer
166 views

Finite VC dimension > the number of free parameters

I'm looking for an example of the following: A hypothesis class $\mathcal{H}$ such that $\forall h \in \mathcal{H}$, the number of free parameters of $h$ is equal to $n \in \mathbb{N}$ (where $n < ...
user avatar
2 votes
0 answers
47 views

What are some beginner's references on algebraically structured (statistical) models, and their connection with group actions and Fourier transform?

I asked this question on Cross Validated a few days ago, but didn't really get a favorable response, so asking here to see if I get any. I'm looking at the description of a short-term position in ...
user avatar
  • 223
1 vote
0 answers
63 views

Relation between minimizer of regularized risk & risk in statistical learning theory

In supervised machine learning, we typically take a Risk Minimization (RM) point of view when formulating a problem. So, what we typically solve for is the following: $$ R^L(h) = \underset{h\in\...
user avatar
0 votes
1 answer
247 views

How large sample $m$ is enough [closed]

I have a $D$ probability distribution over $X =R^d$, i have two samples $s_1$ and $s_2$ from $D$, each having size $m_1$, $m_2$, a unit ball centered at origin $B(0)$, defined by $B(0)=\{x \in R^2: \|...
user avatar
4 votes
0 answers
104 views

Can we show equivalence of two distributions based on their statistics?

Let $p,q$ be two distributions on $\mathbb{R}^d$. Let $f:\mathbb{R}^d\times\mathbb{R}^d\rightarrow\mathbb{R}$. Under what conditions does $\mathbb{E}_{x\sim p}f(x,z)=\mathbb{E}_{x\sim q}f(x,z)\ \...
user avatar
0 votes
1 answer
111 views

Using a poset or directed graph as input for a neural network [closed]

I'm not sure if this is the right community to post this in but I would appreciate any help. As the title states, I'm trying to train a neural network using some unconventional input. I'm wondering if ...
user avatar
1 vote
0 answers
36 views

Restriction of Rademacher Complexity

Let $F\subseteq C([0,1]^n,\mathbb{R})$ be a finite family of functions, which is non-empty. Let $A,B$ be subseteq of $[0,1]^n$, again non-empty, and let $Rad(C)$ denote the Rademacher complexity of ...
user avatar
1 vote
0 answers
61 views

Bayesian inference of stochastically evolving model parameters

I have a question related to self-calibration in radio interferometry, but I will try to phrase it as generic as possible. I have a set of data points, $D = \{ d_{0, t_0}, d_{1, t_0}, ..., d_{M, t_0}, ...
user avatar
  • 29
0 votes
0 answers
52 views

Relation between test and train error with gradient descent iterates

My question is about establishing an inequality between population error and expected training error (i.e, expected training error < population error) for a model trained with gradient descent on a ...
user avatar
  • 1
1 vote
1 answer
641 views

References for "second order" random walk on graphs (used in "node2vec" paper)?

The "word2vec" family of methods provided a great breakthrough in natural language processing. The methods assign to each word a vector in $R^{n}$, such that the "similar" words ...
user avatar
5 votes
0 answers
134 views

Distance of two points in Grassmannian using Plücker coordinate

Let $G(q,D)$ be the Grassmannian of $q$-dimensional vector spaces in $\mathbb{R}^D$, where $q \le D$ are positive integers. In the paper, a distance between two points of $G(q,D)$ are defined as ...
user avatar
  • 5,865
8 votes
4 answers
2k views

How to learn a continuous function?

Let $\Omega \subset \mathbb{R}^m$ be an open subset bounded with a smooth boundary. Problem : Given any bounded continuous function $f:\Omega\to\mathbb{R}$, can we learn it to a given accuracy $\...
user avatar
  • 694
4 votes
1 answer
133 views

Hermite polynomial after rotation

When we consider the $n$-dimensional standard normal distribution, the orthogonal basis is $\{H_S(x)\}_{S}$ where $H_S(x) = \prod_{k=1}^n H_{s_k}(x_k)$. Here $H_*(x)$ is the normalized probabilist's ...
user avatar