Definition $1$. A probability measure $\mu$ on $\mathbb{R}^{d}$ satisfies c-isoperimetry if for any bounded L-Lipschitz $f: \mathbb{R}^{d} \rightarrow \mathbb{R}$, and any $t \geq 0$, \begin{align} \mathbb{P}[|f(x)-\mathbb{E}[f]| \geq t] \leq 2 e^{-\frac{d t^{2}}{2 c L^{2}}} \end{align} In general, if a scalar random variable $X$ satisfies
\begin{align} \mathbb{P}[|X| \geq t] \leq 2 e^{-t^{2} / 2C} \end{align}
then we say $X$ is $C$-subgaussian.
Can I derive the following definition directly?
Definition $1^{*}$. A probability measure $\mu$ on $\mathbb{R}^{d_1} \times \mathbb{R}^{d_2}$ satisfies c-isoperimetry if for any non-Lipschitz $h_{\omega_b \omega_t}: \mathbb{R}^{d_1} \times \mathbb{R}^{d_2} \rightarrow \mathbb{R}^{q}$, and any $t \geq 0$, \begin{align} \mathbb{P}[|h_{\omega_b \omega_t}(s,p)-\mathbb{E}[h_{\omega_b \omega_t}]| \geq t] \leq 2 e^{-\frac{d_1 d_2 t^{2}}{2 c L^{2}}} \end{align} In general, if a scalar random variable $X$ satisfies
\begin{align} \mathbb{P}[|X| \geq t] \leq 2 e^{-t^{2} / 2C} \end{align}
then we say $X$ is $C$-subgaussian.
Info about $h(s,p)$
all was here are weights. All functions are neural net parameterized functions and hence following the Lipschitz property. $L_B$ and $L_T$ are Lipschitz constant.
$$ \mathcal{B}:=\left\{\mathrm{B}_{\boldsymbol{w}} \mid \mathrm{B}: \mathbb{R}^{d_{1}} \rightarrow \mathbb{R}^{q}, \operatorname{Lip}\left(\mathrm{B}_{\boldsymbol{w}}\right) \leq L_{B} \&\|\boldsymbol{w}\| \leq W_{B}\right\} $$ $$ \mathcal{T}:=\left\{\mathbf{T}_{\boldsymbol{w}} \mid \mathbf{T}: \mathbb{R}^{d_{2}} \rightarrow \mathbb{R}^{q}, \operatorname{Lip}\left(\mathbf{T}_{\boldsymbol{w}}\right) \leq L_{T} \&\|\boldsymbol{w}\| \leq W_{T}\right\} $$ Now consider the following function class of all possible inner-products between the two, $$ \mathcal{H}:=\left\{h_{\boldsymbol{w}_{b}, \boldsymbol{w}_{t}} \mid \mathbb{R}^{d_{1}} \times \mathbb{R}^{d_{2}} \ni(s, p) \mapsto h(s, p):=\left\langle\mathrm{B}_{\boldsymbol{w}_{b}}(s), \mathbf{T}_{\boldsymbol{w}_{t}}(p)\right\rangle \in \mathbb{R}, \mathrm{B}_{\boldsymbol{w}_{b}} \in \mathcal{B} \& \mathbf{T}_{\boldsymbol{w}_{t}} \in \mathcal{T}\right\} $$
Is $h(s,p)$ a Lipschitz?
NO! There is no way (as per my knowledge) to show it as a Lipschitz function as,
$f(x) = x$ is a Lipschitz function with Lipschitz constant $1$, $f*f = x^2$ which is not Lipschitz
However I have proven this inequality,
\begin{align} &B_\omega(s)-B_\omega\left(s^{\prime}\right) \leq L_B\left\|s-s^{\prime}\right\|\\ &B_\omega(s) \cdot T_\omega(p)-B_\omega\left(s^{\prime}\right) \cdot T_\omega(p) \leq L_B\left\|s-s^{\prime}\right\| T_\omega(p)\\ &\left\langle B_\omega(s)-B_\omega\left(s^{\prime}\right), T_\omega(p)\right\rangle \leq L_B M_T\left\|s-s^{\prime}\right\| \quad \text { where } M_T=\sup _{\omega_t, p} T_{\omega_t}(p) \text {1.}\\ &T_\omega(p)-T_\omega\left(p^{\prime}\right) \leqslant L_T\left\|p-p^{\prime}\right\|\\ &T_\omega(p) \cdot B_\omega\left(s^{\prime}\right)-T_\omega\left(p^{\prime}\right) \cdot B_\omega\left(s^{\prime}\right) \leq L_T\left\|p-p^{\prime}\right\| \cdot B_\omega\left(s^{\prime}\right)\\ &\left\langle B_\omega\left(s^{\prime}\right), T_\omega(p)-T_\omega\left(p^{\prime}\right)\right\rangle \leqslant L_T M_B\left\|p-p^{\prime}\right\| \quad \text { Where, } M_B=\sup_{\omega_b s^{\prime}} B_{\omega_b}\left(s^{\prime}\right) \text {2.}\\ \end{align}
Adding (1) + (2),
$\left\langle B_{\omega_b}(s), T_{\omega_t}(p)\right\rangle-\left\langle B_{\omega_b}\left(s^{\prime}\right), T_{\omega_t}\left(p^{\prime}\right)\right\rangle$ $\leq L_B M_T\left\|s-s^{\prime}\right\|$ $+L_T M_B\left\|p-p^i\right\|$
So for $h_{\omega_b \omega_t}$ the ?(I guess this is not Lipschitz) constant will be at most $$ \therefore\left(L_B M_T+L_T M_B\right) $$
I am struggling to come up with a solution for $h_{\omega_b \omega_t}$. Any help will be appreciated! Thanks.
Thanks to @dohmatob. I find the mentioned paper fascinating. But there I am facing some issues. If one can give any direction,will be a great help.
I was reading this paper. In that paper, I find Theorem 3 fascinating!
Theorem $3.3$ Assume that $X$ is a random vector in $\mathbb{R}^n$, such that for some constants $L>0, \gamma \geq 1 / 2$, all smooth bounded functions $f$ and all $p \geq 2$, $$ \|f(X)-\mathbb{E} f(X)\|_p \leq L p^\gamma\||\nabla f(X)|\|_p $$ For any smooth function $f: \mathbb{R}^n \rightarrow \mathbb{R}$ of class $\mathcal{C}^D$ and $p \geq 2$ if $\mathbf{D}^D f(X) \in L^p$, then $$ \begin{aligned} \|f(X)-\mathbb{E} f(X)\|_p \leq & C_D\left(\sum_{\mathcal{J} \in P_D} L^D p^{(\gamma-1 / 2) D+\# \mathcal{J} / 2}\|\| \mathbf{D}^D f(X)\left\|_{\mathcal{J}}\right\|_p\right.\\ &\left.+\sum_{1 \leq d \leq D-1} \sum_{\mathcal{J} \in P_d} L^d p^{(\gamma-1 / 2) d+\# \mathcal{J} / 2}\left\|\mathbb{E} \mathbf{D}^d f(X)\right\|_{\mathcal{J}}\right) \end{aligned} $$ Moreover, if $\mathbf{D}^D f$ is bounded uniformly on $\mathbb{R}^n$, then for all $t>0$, $$ \mathbb{P}(|f(X)-\mathbb{E} f(X)| \geq t) \leq 2 \exp \left(-\frac{1}{C_D} \eta_f(t)\right) $$ where $$ \begin{aligned} \eta_f(t) &=\min (A, B), \\ A &=\min _{\mathcal{J} \in P_D}\left(\left(\frac{t}{L^D \sup _{x \in \mathbb{R}^n}\left\|\mathbf{D}^D f(x)\right\|_{\mathcal{J}}}\right)^{2 /((2 \gamma-1) D+\# \mathcal{J})}\right) \\ B &=\min _{1 \leq d \leq D-1} \min _{\mathcal{J} \in P_d}\left(\left(\frac{t}{L^d\left\|\mathbb{E} \mathbf{D}^d f(X)\right\|_{\mathcal{J}}}\right)^{2 /((2 \gamma-1) d+\# \mathcal{J})}\right) \end{aligned} $$
I am particularly interested to calculate that $\eta$ for my function. Also, I have not understood the partition norm quite well.
The definition of the norm
Their $D^{d} f(X)$ definition is like this :
I am interested to find $\eta$ for a multivariable function $h (s,p) := \left\langle\sigma(B_1(s)), \sigma(T_1(p))\right\rangle$. I am assuming $ \sigma = \text{Sigmoid} ,\gamma=1$ and $L=C \sqrt{D_{\text {Poin }}}, D = 2$.
Now in this setting, I need to find Jacobian of that h(s,p) and then I have to take the partition norm of it. How to do that?
Sigmoid function $\varsigma_\alpha(x)$ $$ \begin{aligned} \varsigma_\alpha(x) &=\frac{1}{1+e^{-\alpha x}}=\frac{\tanh (\alpha x / 2)+1}{2} \\ \varsigma_\alpha^{\prime}(x) &=\alpha \varsigma_\alpha(x)\left\{1-\varsigma_\alpha(x)\right\} \\ \varsigma_\alpha^{\prime \prime}(x) &=\alpha^2 \varsigma_\alpha(x)\left\{1-\varsigma_\alpha(x)\right\}\left\{1-2 \varsigma_\alpha(x)\right\} \end{aligned} $$
TIA
