Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [machine-learning]

The tag has no usage guidance.

2
votes
0answers
78 views

3-dimensional machine learning

How to define are continuous scalar functions $X_i, Y_i, Z_i (i \in \{1, 2, ..., 10 \})$ on $[0, 1]$ so that any continuous scalar function $f$ on $[0, 1]^3$ could be represented as: $$f(x, y, z) = \...
0
votes
0answers
36 views

Computing derivative of certain path integrals

Consider a function F (think of neural networks) with two sets of parameters: (1) model parameters $\mathbf{w}$, and (2) input data ${\bf x} \in {\mathbb R}^d$. Fix $i \in [d]$, consider the following ...
0
votes
3answers
93 views

Clustering on tree

I am looking for a method that would identify clusters in a tree-like structure. In the figure below you can see a very simple example where one can visually identify distinct branches with a lot of ...
1
vote
1answer
73 views

Minimum number of support vectors? [on hold]

I'm learning SVM and its written everywhere that the minimal number of support vectors is 2? But I couldn't find any formal proofs of that. Why cant there be less than 2 support vectors? Can somebody ...
2
votes
0answers
56 views

Universal approximation theorem for whole R^d

The well-known universal approximation theorem states that neural network with one hidden layer can approximate any continuous function on every compact subset of $\mathbb{R}^d$. My question is ...
0
votes
1answer
59 views

Reconstructing Euclidian space from distance matrix

The setup. Let's say that we have a set of objects $O_i$ for which we have a dissimilarity measure $M(O_1,O_2)$. With this we can build a distance matrix $D_{ij}$. Let's also assume that we have NO ...
3
votes
1answer
96 views

Why the Fisher information matrix is equal to the Hessian matrix of the Kullback–Leibler distance at the true parameter?

I'm reading 《Algebraic geometry and statistical learning theory》.My problem is why the Fisher information matrix is equal to the Hessian matrix of the Kullback–Leibler distance at the true parameter?...
0
votes
2answers
137 views

Is it possible to “solve” iterative (convex/non-convex) optimization problems via learning (one-shot)?

I posted a following question in MSE, but I think it should be posted here in MO. Since I don't know how to transfer the post from MSE to MO, I have pasted the question below. Thank you in advance and ...
14
votes
1answer
885 views

Is there any paper which summarizes the mathematical foundation of deep learning?

Is there any paper which summarizes the mathematical foundation of deep learning? Now, I am studying about the mathematical background of deep learning. However, unfortunately I cannot know to what ...
2
votes
1answer
202 views

Why the VC dimension of triangles in 2D space is not greater than 7?

I understand that there are sets of 7 points on a circle that can be fully shattered using triangles.But, it is not clear to me why it cannot shatter 8 points. Is there any intuitive way of arriving ...
3
votes
1answer
143 views

Why does the assumption $|U_t| \le \frac1{p_{\min}}$ work in this paper?

I am reading a 2009 paper right now "Importance Weighted Active Learning" and on page 5, there is a theorem that uses the inequality $|U_t| \leq \frac{1}{p_{\min}}$. I am not sure how the paper found ...
5
votes
1answer
84 views

Clustering distance

Is there a good notion of distance between partitions of a (fixed, finite) set? The context is this: suppose I have a clustering algorithm, which clusters points using some method or other. Now, I ...
0
votes
0answers
51 views

Using mollifiers (or some other idea) to solve constrained minimax problem

Sorry in advance if this sounds like a more SE question. Consider a continuously parametrized family of $L$-Lipschitz continuous $f_\theta: X \rightarrow \mathbb R_+$ on a metric space $X=(X,d)$. Let ...
3
votes
1answer
53 views

How to value the extent of separation or mixing of point sets in plane?

As the image presented below, the reddish point set is totally separated from the blueish one and the greenish one, while the blueish point set is quite mixed with the greenish one. A number of ...
5
votes
1answer
238 views

Approximation of Wasserstein distance between $p_\theta$ and $p_{\theta + d\theta}$

Given a parametric family of distributions $\{p_\theta\mid\theta \in \Theta\}$, one can show that under some regularity conditions, the following approximation is valid $$\operatorname{KL}(p_\theta\...
1
vote
2answers
151 views

Lower bound on misclassification rate of Lipschitz functions in terms of Lipschitz constant

Important note @MateuszKwaśnicki in the comment section has raised a fundamental issue with the current statement of the problem. I'm trying to bugfix it. Setup I wish to show that a Lipschitz ...
5
votes
2answers
192 views

VC dimension, fat-shattering dimension, and other complexity measures, of a class BV functions

I wish to show that a function which is "essentially constant" (defined shortly) can't be a good classifier (machine learning). For this i need to estimate the "complexity" of such a class of ...
3
votes
0answers
37 views

Theoretical justification of time-series forecasting using Takens' embedding

This is a cross-posting where I couldn't get an answer. In the meantime I have tried to improve the original logic: As in Takens original paper about his embedding theorem, consider a compact $m$-...
13
votes
0answers
186 views

Profiles of very high dimensional functions

This question comes from trying to understand the recent success of deep neural nets. Neural networks just (crudely speaking) create a very complicated function of very many variables, and then ...
1
vote
1answer
89 views

Packing number of Lipschitz functions

For some $L>0$ say ${\cal L}$ is the space of all $L-$Lipschitz functions mapping $(X,\rho) \rightarrow [0,1]$ where $(X,\rho)$ is a metric space. For any $\alpha >0$ do we know of a ...
11
votes
2answers
476 views

Reference Request: Theoretical Mixing Times Research in Machine Learning / Artificial Intelligence (AI)

I'm doing a PhD in probability theory, focusing mostly on mixing times. It's a pure maths PhD, considering precise models and showing rigorous mixing results. I'm also interested in stuff like machine ...
7
votes
1answer
591 views

graph signal processing

I have read this article https://arxiv.org/abs/1307.5708 about vertix-frequency analysis on graph. David IShuman in this article claims that,"we generalize one of the most important signal ...
2
votes
1answer
152 views

Covering number of Lipschitz functions

What do we know about the covering number of $L$-Lipschitz functions mapping say, $\mathbb{R}^n \rightarrow \mathbb{R}$ for some $L >0$? Only 2 results I have found so far are, That the $\infty$-...
0
votes
0answers
35 views

Are there sharper training sample bounds than the Clopper-Pearson bound?

Consider a hypothesis $h: \mathbb{R}^n \rightarrow \mathbb{R}$ and random variables $X \in \mathbb{R}^n, Y \in \mathbb{R}$. We set $R = P(h(X) \neq Y)$ and $T = (X_i,Y_i)_{i \leq m}$ iid. test ...
-2
votes
1answer
52 views

Find a columns of matrix $A$ which form a basis of columns space of matrix $A$ [closed]

We have a matrix $A$ whose rows are data records and whose columns are features. We would like to omit useless features such as zero or constant columns, duplicate columns, columns that are equal to ...
1
vote
0answers
161 views

What intuitive concepts in pure math can be used to understand Big data? [closed]

I am not a mathematician but I need mathematicians' general knowledge and that is why I chose this community to ask my question from. As a student/researcher in Data Mining, with background in Pure ...
1
vote
0answers
171 views

On proof of Sauer's lemma

Let $X$ be a (possibly infinite) set. We consider a subset $H$ of the set $\{0,1\}^X$ of functions $X\to\{0,1\}$. Given a finite subset $B\subset X$, we denote by $H_B$ the set of restrictions to $B$ ...
0
votes
0answers
33 views

In paper “Neighbourhood Components Analysis”, how to determine the optimal number of neighbours (K)?

Recently, I am reading the paper "Neighbourhood Components Analysis". At the last paragraph on the second page of the paper, "Notice that by learning the overall scale of A as well as the relative ...
0
votes
0answers
66 views

About Rademacher complexity

The following two questions of mine might be related, Q1 Are there examples of non-trivial function classes known in which some norm bound (like the $L^p$ norm) can be used to carve out a subset of ...
0
votes
0answers
31 views

Best Optimization Algorithm: SPSA vs RL and in RL

If I understand correctly, Simultaneous Perturbation Stochastic Approximation is an optimization method whose input parameter is basically just an initial guess given that you can find obtain a "...
0
votes
0answers
45 views

proof in contextual bandits paper on epoch greedy algorithm

I'm reading the following paper on the epoch greedy algorithm for the contextual bandits problem. I have two questions http://hunch.net/~jl/projects/interactive/sidebandits/bandit.pdf I'm unsure ...
5
votes
2answers
360 views

Derivatives of Gaussian processes

A Gaussian process $X$ on Euclidean space $\mathbb R^d$ has a radial basis kernel if for any $u,w\in\mathbb R^d$, we have $$ \mathrm{Cov}(X_u, X_w) = \sigma^2 \exp\left ( -\frac{\left\lVert u-w \right ...
0
votes
1answer
41 views

General results regarding linear separability?

I'm reading up on the theory behind support vector machines and would like a good reference with some general results about linear separability. Specifically, questions like below: Given two ...
3
votes
3answers
277 views

Automatic vs numerical differentiation of a function known from samples

Suppose I have $n$ samples $(x_i, f(x_i))_{i=1}^n$ from an unknown function $f$. I need to approximate (estimate) the derivative $f'(x^*)$ at some new test point $x^*$, that is not necessarily one of ...
1
vote
0answers
28 views

Latent Dirichlet Allocation on Contrived Data

I am doing a project that seems like it might be susceptible to Latent Dirichlet Allocation. However, my data is highly contrived (both in test cases and use cases) and my "words" don't come close to ...
0
votes
1answer
60 views

Learning a Gaussian from noisy observations

Is it possible to learn a distribution over the parameters ($K=\Sigma^{-1}$ and $\mu$) of a Gaussian from noisy measurements of $X$? (Starting with some appropriate prior over the parameters) I know ...
2
votes
1answer
78 views

What is the most appropriate ML technology or math tool for classifying/recognizing 2D scatter plot shapes? [closed]

I would like to be able to automatically classify an input scatter plot into a limited, predefined set of 2-D scatter plots (see attached image), such as a Circle, a Cross, a Straight Line and a Curvy ...