I'm going to pre-empt my esteemed colleague here and say that to some people, the purpose of diffeological spaces is as a stepping stone between manifolds and the category of sheaves on manifolds (or on cartesian spaces, it's the same thing).  So, to these people, you've stumbled on the main point: we really _ought_ to be working with sheaves all along.

The problem is that there are some ornery people who really like manifolds as they are, but sometimes have to work with things that are almost but not quite completely unlike manifolds.  For these people, the further away from true manifolds they get, the more uncomfortable they feel.  One of the biggest steps for such people is losing the underlying set.  So diffeological spaces are a category in which those people can have most of the benefits of sheaves without having to discard their comfort blanket of something that still resembles manifolds in _some_ way.

So diffeological spaces are a convenient (yes, I use the word deliberately!) half-way house whereby those who have Seen The Light can still talk to those still quivering under their comfort blankets.

To name names, people in the first category include Urs Schreiber and John Baez (indeed, I think that John makes that point somewhere on the n-Cafe).  People in the second category include me!

Indeed, I would say that diffeological spaces are closer to the One True Category of Smooth Spaces than sheaves on cartesian spaces.  Frolicher spaces seem to irretrievably have underlying sets - I and a few others have wondered from time to time if there is a way to remove that property but it seems tied up with what they are.