I'm going to pre-empt my esteemed colleague here and say that to some people, the purpose of diffeological spaces is as a stepping stone between manifolds and the category of sheaves on manifolds (or on cartesian spaces, it's the same thing). So, to these people, you've stumbled on the main point: we really ought to be working with sheaves all along.
The problem is that there are some ornery people who really like manifolds as they are, but sometimes have to work with things that are almost but not quite completely unlike manifolds. For these people, the further away from true manifolds they get, the more uncomfortable they feel. One of the biggest steps for such people is losing the underlying set. So diffeological spaces are a category in which those people can have most of the benefits of sheaves without having to discard their comfort blanket of something that still resembles manifolds in some way.
So diffeological spaces are a convenient (yes, I use the word deliberately!) half-way house whereby those who have Seen The Light can still talk to those still quivering under their comfort blankets.
To name names, people in the first category include Urs Schreiber and John Baez (indeed, I think that John makes that point somewhere on the n-Cafe). People in the second category include me!
Indeed, I would say that diffeological spaces are closer to the One True Category of Smooth Spaces than sheaves on cartesian spaces. Frolicher spaces seem to irretrievably have underlying sets - I and a few others have wondered from time to time if there is a way to remove that property but it seems tied up with what they are.
(Added in edit): I don't know why, but I didn't spot first time around the last line:
Is it that limits and colimits are more like their counterparts for manifolds?
which is odd, because that's the subject of a little theorem I've proved which can be found on the nLab:
http://ncatlab.org/nlab/show/topological+notions+of+Fr%C3%B6licher+spaces#hausdorff
Essentially, if you want to preserve those limits and colimits that already exist in the category of manifolds, then you need to work in the category of Hausdorff Frölicher spaces. When you enlarge that category (say to Frölicher spaces or to Diffeological spaces, or to sheaves) then you add in stuff "in the gaps" and create new limits or colimits that disagree with the ones that you had before (in these cases, it's almost always colimits, but if you take the "maps out" view, it will be limits). So that question isn't really a sensible one to ask of Diffeological spaces as you've already lost some colimits. I suppose you can try to do a bit of damage limitation ...