This is a bit long for a comment, so I am posting it as an answer.
With the "clarification" you made, you seem to be saying two (apparently contradictory) things: (1) Since it is easier to construct exotic smooth structures working diffeologically, one should replace the classical viewpoint on smooth manifolds with the categorical one. (2) Since there are smooth manifolds which are homeomorphic but not diffeomorphic, one should replace the classical concept of a smooth manifold with a different (unspecified) categorical one.
If you consider history of smooth manifolds, the classical notion of a smooth manifold was the result of a long chain of events, roughly, from Riemann (whose notion of a "manifold" included rainbow) to Weyl (who gave the first precise definition, in the context of Riemann surfaces). This development was driven by needs of analysis and geometry. Yes, existence of exotic spheres was a rather shocking discovery, followed by a series of exciting developments in geometric topology. However, none of this has diminished the needs coming from analysis and geometry. Incidentally (even though this is not a part of your question), it was proven by Sullivan (in 1977) that in all dimensions but 4, every topological manifold has a Lipschitz structure and this structure is unique. This result could be (potentially) used for development of analytical tools for the study of topological manifolds. In spite of efforts to develop such tools, not much has happend in this direction. On the geometric side, while some of the differential geometry could be done in the category of Lipschitz manifolds, one still does not have a complete replacement for the curvature tensor, which makes Lipschitz setting less appealing than the smooth one.
The bottom line is: Until categorical viewpoint on smoothness proves itself superior to the classical one in the context of analysis on manifolds, there does not seem to be any need to replace the classical notion of smoothness for manifolds.