An integral domain which satisfies the ascending chain condition on principal ideals (ACCP) is a UFD iff every irreducible element is prime. This is a very useful property -- in Diophantine equations and otherwise -- and is so natural that it takes some training to remember that in an arbitrary domain irreducibles need not have this stronger property.
There are other characterizations of UFDs which show that a non-UFD lacks certain nice properties. For instance, a Noetherian domain is a UFD iff every height one prime ideal is principal. Applied to the polynomial ring $k[t_1,\ldots,t_n]$, this shows that every irreducible codimension one subvariety is cut out by a single equation.
Cartier and Weil divisors coincide on a Noetherian integral scheme iff it is locally factorial. Combined with the fact that any regular local ring is factorial, this gives the equivalence between divisors and line bundles on a nonsingular variety. Turned around, this shows that there is "something wrong" with a domain which is not at least locally factorial: it has singularities.
Also, a domain satisfying (ACCP) is a UFD iff every two elements have a greatest common divisor (iff every two elements have a least common multiple). Thus in a non-UFD one cannot necessarily reduce an element of the fraction field to lowest terms.