Let $G$ be a group and let $H$ be a subgroup. If $H$ is normal in $G$, then $G/H$ has a group structure. But in general, can there be a groupoid structure on $G/H$(left cosets or right cosets) that generalizes the normal case?
4 Answers
One answer to your question is that there is always the notion of an "action groupoid", although this does not reproduce the group structure on $G/H$ when $H$ is normal.
Let $G$ be a group acting on a set $X$. (There are generalizations when both $X,G$ are groupoids.) Then the action groupoid $X//G$ is the groupoid with objects $X$, and morphisms $X\times G$. More precisely, if $x,y \in X$, then $\hom(x,y) = \{ g\in G \text{ s.t. } gx = y\}$. The groupoid axioms are essentially obvious.
For example, let $X = G/H$ be the set of left $H$cosets. Then the action groupoid is very simple: it is connected (any object is isomorphic to any other), with $\text{aut}(e) = H$, where $e = eH$ is the trivial left $H$coset. And $\hom(e,g) = gH$, where $g = gH$ is a coset. So as a discrete groupoid, this action groupoid is equivalent to $\{\text{pt}\}//H$, also sometimes called the "classifying groupoid" $\mathcal B H$, because the geometric realization of the nerve of this groupoid is the usual classifying space of $H$.
In shorthand, we have the following equation: $$ (G/H) //G \cong 1//H$$ where $\cong$ denotes equivalence of groupoids. (Actually, since I'm talking about left actions, I should probably write $G \backslash \backslash X$ for the action groupoid, and so the equation really should be $G \backslash \backslash (G/H) = H \backslash \backslash 1$, but typing "/" is much faster than typing "\backslash", so I won't use the better notation.)
But you are probably asking a different question. Recall that when $H$ is normal, then $G/H$ has a group structure, which is to say there is a groupoid with one object and whose morphisms are elements of $G/H$. Of course, as you know, if $H$ is not normal, then $G/H$ does not have a natural group structure, because in general $g_1Hg_2H$ is not a left $H$coset.
You can try to do the following. Any set is naturally a groupoid with only trivial morphisms, and then the set $G/H$ is equivalent to the groupoid $G//H$, where $H$ acts on the set $G$ by translation = right multiplication. (This is because $H$ acts on $G$ freely.) But $G$ is actually more than a set: it is a group. So let's think about it as a "groupal groupoid" or "2group", i.e. a 2groupoid with only one object; in this case, it will also only have identity 2morphisms.
Then I guess you should try to form the "action 2group" or something, by adding 2morphisms for the translations by $H$. But I think that if you do, you no longer have a groupal groupoid: I think that if $H$ is not normal, then the group multiplication is not a functor from the action groupoid $G//H$.
The other only thing I can think of is to define $K = \text{Norm}_GH$, the normalizer, and then $K/H$ is a group that embeds in $G/H$, so let the objects of your groupoid be cosets of $K$ and the morphisms given by $K/H$?
So, long story short: in the way that I think you are hoping, no, $G/H$ is not naturally a groupoid.

$\begingroup$ The second part of your comments is what I am looking for. I was thinking of forming a groupoid with objects given by conjugacy class of the subgroup $H$. But it is not clear how to define the morphisms. I think your idea of forming a 2group is quite interesting. But I don't quite understand it. So i understand that $G$ can be seen as a 2groupoid. But why are we expecting that the "action 2group" you were mentioning has a groupal groupoid structure? $\endgroup$– JunwuTuMar 11, 2010 at 1:57

$\begingroup$ A 2group is the same as a group object in groupoids. $\endgroup$ Mar 11, 2010 at 4:38
When $G$ is finite, for arbitrary $H$ you can consider the quotient $G/H$ as an association scheme in the sense of [Zieschang, PaulHermann. An algebraic approach to association schemes. Lecture Notes in Mathematics, 1628. SpringerVerlag, Berlin, 1996. xii+189 pp. ISBN: 3540614001 MR1439253].
The notion of association scheme generalizes (at least...) that of group, that of distance regular graph, and that of (some types of?) building. When $H$ is normal, then the association scheme $G/H$ is the same thing as the association scheme associated to the group $G/H$, so in a sense, for $H$ nonnormal considering the association scheme $G/H$ is a quite natural.
(In the infinite case, you can probably do more or less the same... I don't recall having seen infinite association schemes, though)
(Of course, this does not answer your question... but as Theo pointed in an earlier answer, I do not think $G/H$ is a groupoid in any sensible way in general, so the bit of structure I am mentioning might be a useful consolation prize!)
Any set on which a group acts can be given a natural groupoid structure, called the action groupoid. Since $G/H$ admits a left $G$action, the action groupoid construction applies. You can find the definition in the wikipedia page de rigueur, under Group actions.
Another way of putting this is that there is an equivalences of categories between actions of a group (or groupoid) $G$ and covering morphisms of the group (or groupoid) $G$. See for example
Higgins, P.J. Notes on categories and groupoids, Mathematical Studies, Volume~32. Van Nostrand Reinhold Co. London (1971); Reprints in Theory and Applications of Categories, No. 7 (2005) pp 1195.
This links the notion of covering morphisms of groupoids nicely with the theory of covering spaces. There one finds that for "nice" spaces $X$ the fundamental groupoid $\pi_1$ gives an equivalence of categories from covering maps of $X$ to covering morphisms of $\pi_1 X$. Thus a map of spaces is modelled by a morphism of groupoids, and this is convenient. For an exposition in these terms, see my book "Topology and groupoids", or earlier editions (1968, 1988).