# How sensitive are Neural Networks to weight change?

Let's consider the space of feedforward neural networks with a given structure: $$L$$ layers, $$m$$ neurones per layer, ReLu activation, input dimension $$d$$, output dimension $$k$$.

Which means I'm considering the map $$F: \mathcal{W}_1 \times \mathcal{W}_2 \times \dots \times \mathcal{W}_L \times \mathbb{R}^d \to \mathbb{R}^k$$, where $$\mathcal{W}_i$$ is the space of possible weights for layer $$i$$. We also assume, per simplicity, that every weight matrix has norm bounded by a constant $$M$$. Let's now fix the parameters so that we obtain $$v = F(W_1, \dots, W_L, x^*) \in \mathbb{R}^k$$ (note that $$x^*$$ is fixed as well).

Image now that I inject some random noise $$\eta \in \mathbb{R}^{m \times m}$$ in a weight matrix $$W_i$$, where the norm of the noise is 10% the norm of the matrix, e.g. $$||\eta|| = ||W_i||/10$$ . How does it affect my final output?

which means, what's the expected value of $$||v - v_*||$$, where $$v_*$$ is the output of the network obtained after the small change in the weights describer before?