Let's consider the space of feedforward neural networks with a given structure: $L$ layers, $m$ neurones per layer, ReLu activation, input dimension $d$, output dimension $k$.

Which means I'm considering the map $F: \mathcal{W}_1 \times \mathcal{W}_2 \times \dots \times \mathcal{W}_L \times \mathbb{R}^d \to \mathbb{R}^k$, where $\mathcal{W}_i$ is the space of possible weights for layer $i$. We also assume, per simplicity, that every weight matrix has norm bounded by a constant $M$. Let's now fix the parameters so that we obtain $v = F(W_1, \dots, W_L, x^*) \in \mathbb{R}^k$ (note that $x^*$ is fixed as well).

Image now that I inject some random noise $\eta \in \mathbb{R}^{m \times m} $ in a weight matrix $W_i$, where the norm of the noise is 10% the norm of the matrix, e.g. $||\eta|| = ||W_i||/10$ . How does it affect my final output?

which means, what's the expected value of $||v - v_*||$, where $v_*$ is the output of the network obtained after the small change in the weights describer before?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.