Skip to main content
3 of 3
added 187 characters in body
RaphaelB4
  • 4.4k
  • 12
  • 25

$$\mathbb{E}(|\sigma(x+\eta)-\sigma(x)|)=\mathbb{E}(|J_\sigma\eta|)+O(|\eta|^2)$$where $J_\sigma$ is the Jacobi matrix $$ (J_\sigma)_{i,j}=\frac{e^{x_i}}{\sum_k e^{x_k}}\delta_{i,j} - \frac{e^{x_i+x_j}}{(\sum_k e^{x_k})^2}$$ and $$\mathbb{E}(|J_\sigma\eta|)^2\leq \mathbb{E}(|J_\sigma\eta|^2)=\mathbb{E}(\langle \eta J_\sigma^2\eta\rangle)=\frac{|\eta|^2}{d}\text{Tr}(J_\sigma^2)$$with $$\text{Tr}(J_\sigma^2 )=\sum_i \frac{e^{2x_i}}{(\sum_k e^{x_k})^2}+\sum_{i,j} \frac{e^{2x_i}e^{2x_j}}{(\sum_k e^{x_k})^4} -2\sum_{i}\frac{e^{3x_i}}{(\sum_k e^{x_k})^3} \\ = \sum_i (\sigma(x)_i)^2+\big(\sum_i (\sigma(x)_i)^2\big)^2-2\sum_i (\sigma(x)_i)^3$$

Conclusion : $$\mathbb{E}(|\sigma(x+\eta)-\sigma(x)|)\leq \frac{\eta}{\sqrt{d}}\sqrt{\sum_i (\sigma(x)_i)^2+\big(\sum_i (\sigma(x)_i)^2\big)^2-2\sum_i (\sigma(x)_i)^3 }+O(|\eta|^2)$$

RaphaelB4
  • 4.4k
  • 12
  • 25