There's no field with one element in the literal sense, but there are constructions that work over different fields F_q and which sometimes make sense when q=1. Examples would include representation theory of GL_n(F_q) which, if I'm correct, becomes the representation theory of S_n under that limit.
In particular, indeed, vector spaces — the objects on which GL_n(F_q) act — should become sets, the objects on which S_n acts.
Though I'm not an expert on F_1, I've encountered the viewpoint you're referring to. It's not hard to see why people expect F_1 to be the universal base: you usually expect that Spec F_n -> Spec F_m exists iff m divides n, so Spec F_1 should be terminal. Of course, things are not literally that simple, e.g. what Spec ZZ x Spec ZZ over Spec F_1 is?
Also, Spec ZZ should be thought of the object of dimension 3 (I think it has dualizing complex in degree 3, and primes are similar to 1-dimensional knots), so there's a pressing need for existence of some smaller scheme.
When you get stuck geometrically, it helps to lose some information and go to zeta functions. There this approach leads to some direct conjectures. The problem with zeta functions is that everything is hard. Still, search for absolute zeta and you'll see some real things being discussed.
It's been called absolute motives in the works I read, I think by Manin, Kontsevich, and some IHES people. The search for kontsevich absolute motives brought me up the article that I found a while ago, math/0702206
This is also somehow related to noncommutative geometry of Connes (endomotives), in fact I think F_1 is one of his favorite ideas (or is it from some other part of French math circle?)
Also, here's a blog post about F_1 with link to the introductory paper. In fact there was a whole blog about F_un, which strangely disappeared (well, at least the articles I wanted to read should be still somewhere cached at my other computer).