Whenever I have seen unique factorization discussed, it is always with respect to the solution of diophantine equations; the equations are solved by splitting the equation into linear functions over a ring and then invoking unique factorization. But the discussions always give the impression that the failure of unique factorization causes all sorts of problems. Apart from the applications to diophantine equations, why is unique factorization such a desirable property?
I think a glance at any elementary number theory textbook will show you numerous topics which rely on uniqueness of factorization. The formulas for the number of divisors, the sum of the divisors, the Euler phi-function, all depend on unique factorization. Going up a level, the Euler product for the Riemann zeta function depends on unique factorization.
An integral domain which satisfies the ascending chain condition on principal ideals (ACCP) is a UFD iff every irreducible element is prime. This is a very useful property -- in Diophantine equations and otherwise -- and is so natural that it takes some training to remember that in an arbitrary domain irreducibles need not have this stronger property.
There are other characterizations of UFDs which show that a non-UFD lacks certain nice properties. For instance, a Noetherian domain is a UFD iff every height one prime ideal is principal. Applied to the polynomial ring $k[t_1,\ldots,t_n]$, this shows that every irreducible codimension one subvariety is cut out by a single equation.
Cartier and Weil divisors coincide on a Noetherian integral scheme iff it is locally factorial. Combined with the fact that any regular local ring is factorial, this gives the equivalence between divisors and line bundles on a nonsingular variety. Turned around, this shows that there is "something wrong" with a domain which is not at least locally factorial: it has singularities.
Also, a domain satisfying (ACCP) is a UFD iff every two elements have a greatest common divisor (iff every two elements have a least common multiple). Thus in a non-UFD one cannot necessarily reduce an element of the fraction field to lowest terms.