5
$\begingroup$

We consider a probability measure supported on the whole space $\mathbb{R}^n$, whose density is $p(x)$. We also consider a (one-layer) neural network function class $\mathcal{C}$, whose elements have the form $$ f(x)=\sum_{i=1}^rv_i\sigma(w_i^{\top}x).\tag{neural network} $$ Here $r$ is the number of nodes in the hidden layer, $\sigma$ is the active function (such as ReLu, sigmoid), $v_i$ and $w_i$ are parameters. Such a class can also be viewed as a discretization of the integral $$ g(x) = \int_t v(t) \sigma(w(t)^\top x) d t,\tag{continuous version} $$ which is common in harmonic analysis. A familiar example is when $\sigma$ is a trigonometric function and $v(t)$ is a constant, it resembles the Fourier integral, where $w(t)$ is the frequency.

My question is that, under what conditions on the probability measure $p(x)$ and the structure of function class $\sigma, r,v_i,w_i $, weighted reverse Poincare inequality holds over $\mathcal{C}$? That is to say, there exists a constant $K$, such that for any $f,g \in \mathcal(C)$, the following inequality holds. $$ \int_{\mathbb{R}^n}\|\nabla f(x)-\nabla g(x)\|^2 p(x) dx\le K\int_{\mathbb{R}^n}| f(x)-g(x)|^2 p(x)dx.%\tag{reverse Poincare} $$
In other words, we are able to upper bound the difference in gradients with the difference in function values (which is in the reverse direction of the commonly known Poincare inequality).

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.