First, I do it all the time and don't really see the objections. A phrase like "In [S] it was shown..." is a good alternative to "Siegel showed, [S], that ...".
Out of curiosity I did some cursory research and looked up the citation habits in Annals of Mathematics 1958. There one author (R.D. Anderson) does use "In [1] we considered..." and "an argument in [4]". On the other hand Dold sticks to phrases like "due to J.C. Moore [11]" or "defined by Eilenberg-MacLane [4]" as opposed to "defined by Eilenberg-MacLane in [4]".
So the habit is quite old. On the other hand, the list of references at the end of the paper has not been around forever. Again some research I did during some idle moments in Oberwolfach showed that a shift took place in the 1940s-50s. Before, references were handled either inline or via footnotes. Clearly these cannot be used in noun form. Instead,repeated references to the same paper were done using the rather clumsy phrase "loc. cit." or "the third paper by Siegel, cited above". Maybe the objections come from this time.
Edit: After reflecting on this issue a bit more I found that even though this style of citation is per se admissible it does have a pitfall which one should be aware of: It makes it all too easy to avoid naming an author explicitly. I consider it as very bad practice if somebody's name appears only in the references.