There's no field with one element in the literal sense, but there are constructions that work over different fields F_q and which sometimes make sense when q=1. Examples would include representation theory of GL_n(F_q) which, if I'm correct, becomes the representation theory of S_n under that limit.
In particular, indeed, vector spaces — the objects on which GL_n(F_q) act — should become sets, the objects on which S_n acts.
Though I'm not an expert on F_1, I've encountered the viewpoint you're referring to. It's been called absolute motives in the works I read, I think by Manin, Kontsevich, and some IHES people. Indeed, the think that people talk about is the mental picture of every scheme being kind a scheme defined over F_1. (The search for kontsevich absolute motives brought me up the article that I found a while ago, math/0702206)
This is also somehow related to noncommutative geometry of Connes, in fact I think F_1 is one of his favorite ideas (or is it from some other part of French math circle?)
Also, here's a blog post about F_1 with link to the introductory paper. In fact there was a whole blog about F_un, which strangely disappeared (well, at least the articles I wanted to read should be still somewhere cached at my other computer).