Update: at the bottom there's a wonderful and fresh reference.

----

There's no field with one element in the literal sense, but there are constructions that work over different fields $\mathbb F_q$ and which sometimes make sense when $q=1$. Examples would include representation theory of $GL_n(\mathbb F_q)$ which, if I'm correct, becomes the representation theory of $S_n$ under that limit. 


In particular, indeed, vector spaces — the objects on which $GL_n(\mathbb F_q)$ act — should become sets, the objects on which $S_n$ acts.

Though I'm not an expert on $\mathbb F_1$, I've encountered the viewpoint you're referring to. It's not hard to see why people expect $\mathbb F_1$ to be the universal base: you usually expect that  $\mathop{\text{Spec}} \mathbb F_n \to \mathop{\text{Spec}} \mathbb F_m$ exists iff $m$ divides $n$, so $\mathop{\text{Spec}} \mathbb F_1$ should be terminal. Of course, things are not literally that simple, e.g. what $\mathop{\text{Spec}} \mathbb Z \times \mathop{\text{Spec}} \mathbb Z$ over $\mathop{\text{Spec}} \mathbb F_1$ is? 

Also, $\mathop{\text{Spec}} \mathbb Z$ should be thought of the object of dimension 3 (I think it has dualizing complex in degree 3, and primes are similar to 1-dimensional knots), so there's a pressing need for existence of some smaller scheme. That smaller scheme better "have real dimension 1" so we can claim that $\mathop{\text{Spec}} \mathbb Z$ is an algebraic curve over it. One then tries to compactify it.
 
When you get stuck geometrically, it helps to lose some information and go to zeta functions. There this approach leads to some direct conjectures. The problem with zeta functions is that everything is hard. Still, search for `absolute zeta` and you'll see some real things being discussed.

It's been called `absolute motives` in the works I read, I think by Manin, Kontsevich, and some IHES people. The search for `kontsevich absolute motives` brought me up the article that I found a while ago, [math/0702206][1].

This is also somehow related to noncommutative geometry of Connes (`endomotives`), in fact I think $\mathbb F_1$ is one of his favorite ideas (also owes a lot to Soulé and other people of course) The two articles I found to be of special interest are [0806.2401][2] and [0809.2926][3].

Also, here's a [blog post about $\mathbb F_1$][4] with link to the introductory paper. In fact there was a whole blog about $F_{un}$, which disappeared (articles, unfortunately,  also disappeared from my RSS reader).

-------

Here's a paper by Connes and Consani that seems to explain most the topics mentioned above: 
> <b> <i>Schemes over F<sub>1</sub> and zeta functions</i>, [0903.2024][5] </b>


  [1]: http://arxiv.org/abs/math/0702206
  [2]: http://arxiv.org/abs/0806.2401
  [3]: http://arxiv.org/abs/0809.2926
  [4]: http://noncommutativegeometry.blogspot.com/2008/06/fun-day-two.html
  [5]: http://arxiv.org/abs/0903.2024