Let $F: (Sch)^{o}\to (Set)$ be a functor that admits a coarse moduli $Y$ (a scheme). We can consider $Y$ as a representable functor $h_{Y}: (Sch)^{o}\to (Set)$. Is there a direct way to produce $h_Y$ from $F$ (maybe by taking sheafification of $F$ in some topology or something like that)?
The same question for a DeligneMumford stack $F: (Sch)^{o}\to (Groupoids)$ and its coarse moduli space (a scheme or an algebraic space). Is it produced as taking $\pi_0$ of $F$ and then sheafification or something like that?
It is not true that one can obtain the coarse moduli space of a functor or stack $F$ by sheafifying $F$ or $\pi_0(F)$, so probably the answer to both of your questions is NO.
I'll take the following definition of a coarse moduli space of a functor or stack $F: (Sch)^{op} \rightarrow \mathrm{Gpoid}$: It's an algebraic space $X$ together with a natural transformation $F \rightarrow X$ such that
$X$ is universal for maps from $F$ to algebraic spaces (a bit stronger than being universal to schemes), and
For any algebraically closed field $k$, the natural map $F(k) \rightarrow X(k)$ is a bijection.
For this definition (which I think is the standard one), the coarse moduli space is typically NOT obtained by sheafifying $\pi_0(F)$. For example, take $F$ to be the DeligneMumford stack $[\mathbb{A}^1/(\mathbb{Z}/2)]$, for $\mathbb{A}^1 = \mathrm{Spec \ }\mathbb{C}[x]$ and $\mathbb{Z}/2$ acting in the obvious way by $x \mapsto x$. Then the coarse moduli space of $F$ is given by $X = \mathbb{A}^1$, and the map $F \rightarrow X$ is induced by the map $x \mapsto x^2$ from $\mathbb{A}^1$ to $\mathbb{A}^1$. But $\mathbb{A}^1$ is not isomorphic to the sheafification of $\pi_0(F)$ for the etale topology (and probably not for any subcanonical topology, although I'm not sure how to prove this at the moment).
To see why, let $Y$ be the etale sheafification of $\pi_0(F)$, and let $T_0 F$, $T_0 X$, and $T_0 Y$ be the tangent spaces to these functors at 0 (by which I mean the set of lifts of $0 \in F(\mathbb{C})$ to $F(\mathbb{C}[\epsilon])$, where $\epsilon^2 = 0$). If $X$ were given by the etale sheafification of $\pi_0(F)$, there would be a natural map $Y \rightarrow X$ (induced from the projection $F \rightarrow X$) which would be an isomorphism on tangent spaces. But one sees that the map $T_0 F \rightarrow T_0 X$ is the zero map since this is true for the map $x \mapsto x^2: \mathbb{A}^1 \rightarrow \mathbb{A}^1$. On the other hand, the map $T_0 X \rightarrow T_0 Y$ is nonzero, because $\mathbb{C}[\epsilon]$ has no nontrivial etale covers and so $T_0 Y = T_0(\pi_0(F))$, which is isomorphic to $\mathbb{C}/(\mathbb{Z}/2)$ with the projection map $T_0 F \rightarrow T_0 Y$ identified with the quotient map $\mathbb{C} \rightarrow \mathbb{C}/(\mathbb{Z}/2)$.
If you look at the proof of the existence of coarse moduli spaces, you'll see that it's done by first considering the case of quotienting an affine scheme by a finite flat groupoid, where the coarse moduli space is given by Spec of the invariants. One then carefully glues together the coarse moduli spaces on open patches. This construction makes me suspect that there is no direct (i.e., global) construction of the coarse moduli space, at least not one that is easy to work with.

$\begingroup$ Ah, I see. So, basically all I was missing was condition 2.), except, to allow more coarse moduli spaces to exist, the colimit should be taken in algebraic spaces, makes sense. You're right that there shouldn't be a functor producing coarse moduli spaces then, but, there is a formula, which gives the right answer when the colimit exists, and 2) is satified, but you're completely right, in that it is not easy to work with. However, the idea behind it, is what allows you to compute it adhocly: break down the colimit into a small one you can compute. $\endgroup$ – David Carchedi Mar 27 '13 at 10:54

$\begingroup$ (P.S. +1 for illuminating me on the correct established conditions) $\endgroup$ – David Carchedi Mar 27 '13 at 10:55
I could be wrong, but I am going to be brave and assume that coarse moduli spaces are defined in the analogous way as for topological stacks. If this turns out being incorrect, I will remove this answer:
Let $F:Sch^o \to Gpd$ be your functor. $F$ is canonically a colimit of representable sheaves:
Let $\pi_F:\int F \to Sch$ be the fibered category representing $F.$ Then $F$ is the colimit of $y \circ \pi_F,$ where $y:Sch \to St\left(Sch\right)$ is the Yoneda embedding into stacks.
If the category of schemes (edit algebraic spaces) were cocomplete, you could define the coarse moduli space functorially, by sending each $F$ to the colimit of $\pi_F$
This would be the pointwise the left Kan extension of the identity functor with respect to the Yoneda embedding there are some size issues here, so you should really replace the identity functor with the inclusion into a larger Grothendieck universe of schemes, but anyway...) (edit) inclusion of schemes into (a larger Grothendieck universe of) algebraic spaces. However, since algebraic spaces are not cocomplete, this cannot be made functorial in general.
Anyway, I believe, that "$F$ has a coarse moduli space", is the same thing as "$\pi_F$ has a colimit" (and that this colimit is the coarse moduli space) (edit actually the composition of $\pi_F$ with the inclusion into algebraic spaces)
It's worth mentioning, that often times this colimit can be replaced by a much smaller one, e.g., if $F$ is a stack coming from a groupoid object in schemes (or algebraic spaces) ($G_1 \rightrightarrows G_0$) (Hopf algebroid), then $F$ can be written as a finite colimit involving only $G_1 \times_{G_0} G_1,$ $G_1$ and $G_0,$ and the resulting coarse moduli space, if it exists, would be in fact just the coequalizer of $G_1 \rightrightarrows G_0$ in schemes. (edit) algebraic spaces.
(edit) At any rate, apparently, for this coarse moduli space $X = \varinjlim \pi_F$ to be well behaved, in addition to being this colimit, one wants the canonical map $F \to X$ to induce a bijection on $k$points for all algebraically closed fields.