I'm trying to understand neural networks formally a little better and it was always my understanding that the miracle that happens during backpropagation is that "performing a pass in gradient descent results in a new neural network where one just updates the weights", however going through a simple example, I don't really see how that works….
Let's just do a simple shallow case $\mathbb{R}^2\longrightarrow \mathbb{R}^2$, in other words we have a supervised learning problem with a dataset $\Delta \in \mathbb{R}^2 \times \mathbb{R}^2$ of sources $\begin{bmatrix}x_i \\ y_i \end{bmatrix} \in \mathbb{R}^2$ and targets $\begin{bmatrix}u_i \\ v_i \end{bmatrix}\in \mathbb{R}^2$ as well.
Now, a shallow neural network will look like $\mathbb{R}^2\longrightarrow \mathbb{R}^2: \begin{bmatrix}x \\ y \end{bmatrix}\mapsto \begin{bmatrix}(\sigma(ax+by)\\ (\sigma(cx+dy) \end{bmatrix}$, or put differently $\sigma \cdot \eta_{(a,b,c,d)} \biggl(\begin{bmatrix}x\\ y \end{bmatrix}\biggr)$ where $\eta_{(a,b,c,d)} = \begin{bmatrix} a & b \\ c & d\end{bmatrix}$ and $\sigma$ is your favorite activation function applied componentwise.
Next we introduce the cost function which takes a matrix $\eta_{(a,b,c,d)}$ in $\mathbb{R}^{2 \times 2}$ and returns the cost $c \in \mathbb{R}$,
$$c: \mathbb{R}^{2\times 2} \longrightarrow \mathbb{R}:\eta_{(a,b,c,d)} \mapsto \sum_{(x_i,y_i), (u_i, v_i) \in \Delta}\big(u_i - \sigma(ax_i + by_i)\big)^2 + \big(v_i - \sigma(cx_i + dy_i)\big)^2$$…. So far so good….
Next we perform a gradient descent pass in the 4-d space of neural networks, a.k.a. we compute the expression
$$\eta_{(a,b,c,d)} - \lambda \cdot (\nabla c) (\eta_{(a,b,c,d)}) $$
(for some a priori chosen learning rate $\lambda$).
The gradient (which can be computed for any finite dimensional space including a space of matrices) becomes a little tricky to write down but it is essentially a function $\nabla c: \mathbb{R}^{2\times 2} \longrightarrow \mathbb{R}^{2\times 2}: \begin{bmatrix} a & b \\ c & d\end{bmatrix} \mapsto \begin{bmatrix} A & B\\ C & D\end{bmatrix}$ where $A$ represents the partial derivative of the cost function $c$ in the direction of the variable $a$ evaluated at $\eta_{(a,b,c,d)}$….
Or more explicitly
$$A = \frac{\partial c}{\partial a} (\eta_{a,b,c,d}) = \sum_{(x_i,y_i), (u_i, v_i) \in \Delta} 2\big(u_i - \sigma(ax_i + by_i)\big) \cdot -\sigma'(ax_i+by_i)\cdot x_i$$
…and so on for $B,C$ and $D$…. The important thing to note is that these are simply constants.
Now, I'm assuming that that $-\lambda \begin{bmatrix} A & B\\ C & D\end{bmatrix}$ is the error correction we'll have to feed into the new pass of the neural network down the line…. Now the question is, how does this work precisely?
Trying to work it out, the new neural network is now
$$\eta':\begin{bmatrix} x \\y \end{bmatrix}\mapsto \begin{bmatrix} \sigma(ax+by) \\ \sigma(cx+dy)\end{bmatrix} - \lambda \begin{bmatrix} A & B\\ C & D\end{bmatrix} \begin{bmatrix}\sigma(ax+by)\\ \sigma(cx+dy) \end{bmatrix}.$$
Which is the same as multiplying by $$M \cdot \sigma \cdot \eta_{a,b,c,d} $$ where $M = \mathrm{Id} -\lambda \cdot \begin{bmatrix} A & B\\ C & D\end{bmatrix} $. However this does not seem like a shallow neural network at all, in fact it turns out we have added a new layer?
I was under the impression that it was possible to just "update weights" and end up up with a neural network of the same type $\mathbb{R}^2\longrightarrow \mathbb{R}^2$ given by $\sigma \cdot \eta'_{(a,b,c,d)}$ where $\eta'_{(a,b,c,d)}$ is the new version?
Indeed, what I am suggesting is the very simple fact that starting with a neural network $\mathbb{R}^2\longrightarrow\mathbb{R}^2:\begin{bmatrix} x \\y \end{bmatrix} \mapsto \sigma \bigg( \begin{bmatrix}a & b \\ c& d \end{bmatrix} \cdot \begin{bmatrix} x \\y \end{bmatrix}\bigg) $ Following the gradient descent rule, the result is not of the same form, the gradient descent rule tells you to do something else... it doesnt simply become
$\mathbb{R}^2\longrightarrow\mathbb{R}^2:\begin{bmatrix} x \\y \end{bmatrix} \mapsto \sigma \bigg( \begin{bmatrix}a' & b' \\ c'& d' \end{bmatrix} \cdot \begin{bmatrix} x \\y \end{bmatrix}\bigg) $
where $c' = c - \lambda C$ for example
Indeed, what I am suggesting is the very simple fact that starting with a neural network $\mathbb{R}^2\longrightarrow\mathbb{R}^2:\begin{bmatrix} x \\y \end{bmatrix} \mapsto \sigma \bigg( \begin{bmatrix}a & b \\ c& d \end{bmatrix} \cdot \begin{bmatrix} x \\y \end{bmatrix}\bigg) $ Following the gradient descent rule, the result is not of the same form, the gradient descent rule tells you to do something else... it doesnt simply become
$\mathbb{R}^2\longrightarrow\mathbb{R}^2:\begin{bmatrix} x \\y \end{bmatrix} \mapsto \sigma \bigg( \begin{bmatrix}a' & b' \\ c'& d' \end{bmatrix} \cdot \begin{bmatrix} x \\y \end{bmatrix}\bigg) $
where $c' = c - \lambda C$ for example
