Neural networks are best-in-breed at solving a very specific kind of problem: computing a function $f \colon X \to Y$ given the values of $f$ on a large but finite subset $A \subseteq X$. Typically $Y$ (the space of "labels") is finite and small relative to $A$. Concrete example: $X$ is the space of images, $Y$ is a set of labels for images (e.g. "contains a car" or "is a picture of a sunset"), and $A$ is a large set of human-annotated images (the training data).

As it turns out, neural networks can learn $f$ so well that they can produce points in $f^{-1}(y)$'s near a prescribed $x \in X$ - that's why they are good at image / language generation.

This is certainly an important and broad class of problems (particularly for applications in industry), and in practice neural networks don't really have any serious rivals. But there are lots of other image processing problems for which they aren't really appropriate - compression, recovery, noise reduction, etc. I don't have a lot of expertise to draw from, but for far as I am aware the standard techniques from physics and engineering are close to the state of the art.

I'll conclude by remarking that there are some persuasive arguments that the practical effectiveness of neural networks is explained by principles in physics. There is theory which says that if $X$ is large (e.g. all images or all sequences of characters in an alphabet) then no machine learning algorithm can perform better than the most naive ones. So to understand why neural networks are so high-performing one must use something about the structure of the underlying data sources (photographs or human language) to restrict $X$. Here are some recent papers in this direction:

https://arxiv.org/abs/1608.08225

https://arxiv.org/abs/1410.3831