This is a rather open-ended question, so let me respond by offering some observations in the same spirit, focusing primarily on posets, graphs and simplicial complexes.
In a very general way, graphs and digraphs are used throughout mathematics to represent relations and other structures with metric, topological and enumerative-combinatorial properties, such as the diameter, connectedness and the number of subgraphs with prescribed properties reflecting some of the deeper or less obvious features of the represented structure. (These relations may originate from a mathematical model or be innate to a mathematical theory.) Advanced examples along these lines are provided by the order complexes of posets and Stanley-Reisner complexes of square-free ideals, where the complex both visualizes the underlying structure and brings forth its important features. In a different direction, diagrams in category theory and quivers in algebra, including advanced constructions such as the Auslander-Reiten quiver, are types of visualizations.
Here is a specific illustration already firmly within mathematical setting. The notion of Hasse diagram of a partially ordered set provides an important visualization of this structure that has been used to study its properties. For example, it is easy to see that the diamond lattice $M_3$ and the pentagon lattice $N_5$ are not distributive. In fact, a lattice is distributive if and only if it does not contain a sublattice isomorphic to $M_3$ or $N_5$. On a more rudimentary level, the fundamental concepts such as maximal and minimal elements, chains and maximal chains of a poset can be thought of as visualizations. Some research directions in group and module theory deal with the structure of groups/modules based on their properties of their subgroup/submodule lattice and one can argue that, for example, uniserial modules (where all submodules form a single chain) first became objects of interest because of this visualization aspect. Later, the success of this theory motivated investigating more complicated cases.
Finally, going beyond combinatorial and algebgraic structures, configuration spaces and moduli spaces of all kinds fit into a visualization paradigm: they "visualize" the totality of objects or "states of a system" and can be used to study generic or typical properties, statistics and often dynamics (when the configuration or moduli space arises from geometry).