I also found this paper which gives a topological characterization of a classification neural network to be intuitive :
They defined data as a topological space and defined labels as closed subsets of this space. They also defined a notion of "separable data" which is topologically equivalent to "correctly classifying the data". Urysohn lemma is then used to prove that the data, considered as a topological space, can be "separated", if you can find embedded disks that are mutually disjoint and separate the labeled parts of the data (after being mapped by the neural network to the final space).
This can be seen with the disentanglement figure at the end of the paper:
which shows how the above neural network acts on the input topological space, deforms it in order to achieve a given classification task (in this case separating the two labeled linked spaces). Some hints to the general position theorem are also made there. There are also some links to "topological moves" in the paper :a neural network acts on the input topological space by a sequence of topological moves (similar to Reidemeister moves in knot theory or other moves in topology) in order to achieve the given task : which is deforming the input space to the final space, in this case the Voronoi diagram that represents the classes of the input data, such that every labeled region in the input space maps to the correct cell in the final Voronoi diagram.
To me, from this perspective, a classification neural network can be interpreted as if it is acting on the input space as a "discrete colored homotopy" : discrete because of the layers (time is the index of the layer) and colored because the class label which can be considered as "coloring" on the input space.