Let G be an algebraic group acting on a scheme X. Then f: X --> Y is a categorical quotient if it is constant on G-orbits and any other G-invariant morphism factors through it in a unique fashion. We say f is a 'good' categorical quotient if:

1) f is a surjective open submersion (i.e. the topology on Y is induced from X).

2) for any open U ⊂ Y, the induced map from functions on U to G-invariant functions on f^-1(U) is an isomorphism.

Does anyone know an example of a 'bad' categorical quotient (by which I mean...well...a not good one).

-
I don't think you mean open immersion. –  David Zureick-Brown Oct 16 '09 at 21:06
Correct; it's sub- not im- –  Harold Williams Oct 16 '09 at 22:52

Note that if f: X→Y is a categorical quotient in the category of schemes which is stable under base change by open immersions, then the second condition (ie. OY→(f* OX)G is an isomorphism) is automatically satisfied.

In the paper "Examples and counterexamples for existence of categorical quotients" by A'Campo-Neuen and Hausen, there is an example of a categorical quotient f: X→A1 such that f-1(A1 - 0)→A1 - 0 is not a categorical quotient. I haven't checked but I believe this should also give an example where condition (2) fails.

I don't know of example of a categorical quotient where condition (1) fails.

-
It seems to me that you are trying to get a useful notion of quotient for group operations. Points in such a quotient $Y$ correspond to invariant closed subsets of $X$. So, if you have only a closed orbit in $X$ or if you have only one open orbit the good quotient you described is a point and there is not much useful information on $X$ that you can see there.
Once you acknowledged this, you can try to fix this. The direction choosed my Mumford is to look for invariant principal open subsets: by discarding the zero locus of some $G$-invariant homogeneous form, it is possible to construct a quotient.