Update: at the bottom there's a wonderful and fresh reference.
There's no field with one element in the literal sense, but there are constructions that work over different fields F_q and which sometimes make sense when q=1. Examples would include representation theory of GL_n(F_q) which, if I'm correct, becomes the representation theory of S_n under that limit.
In particular, indeed, vector spaces — the objects on which GL_n(F_q) act — should become sets, the objects on which S_n acts.
Though I'm not an expert on F_1, I've encountered the viewpoint you're referring to. It's not hard to see why people expect F_1 to be the universal base: you usually expect that Spec F_n -> Spec F_m exists iff m divides n, so Spec F_1 should be terminal. Of course, things are not literally that simple, e.g. what Spec ZZ x Spec ZZ over Spec F_1 is?
Also, Spec ZZ should be thought of the object of dimension 3 (I think it has dualizing complex in degree 3, and primes are similar to 1-dimensional knots), so there's a pressing need for existence of some smaller scheme.
When you get stuck geometrically, it helps to lose some information and go to zeta functions. There this approach leads to some direct conjectures. The problem with zeta functions is that everything is hard. Still, search for absolute zeta and you'll see some real things being discussed.
It's been called absolute motives in the works I read, I think by Manin, Kontsevich, and some IHES people. The search for kontsevich absolute motives brought me up the article that I found a while ago, math/0702206.
This is also somehow related to noncommutative geometry of Connes (endomotives), in fact I think F_1 is one of his favorite ideas (also owes a lot to Soulé and other people of course) The two articles I found to be of special interest are 0806.2401 and 0809.2926.
Also, here's a blog post about F_1 with link to the introductory paper. In fact there was a whole blog about F_un, which strangely disappeared (well, at least the articles I wanted to read should be still somewhere cached at my other computer).
Here's a paper by Connes and Consani that seems to explain most the topics mentioned above:
Schemes over $\F_1$ and zeta functions, 0903.2024