If $G$ is a topological group and $G_{{e}}$ is the identity component, the it is well known that $G_{{e}}$ is a normal subgroup of $G$ and the quotient group $G/G_{{e}}$ is totally disconnected. What can we say about the subgroup $H$ (need not be normal) of $G$ such that coset space is totally disconnected? Has it been studied for some topological groups such that the coset space is totally disconnected?

$\begingroup$ How do you want to form the quotient when $H$ is not normal? $\endgroup$– Lee MosherJan 6 '18 at 16:22

$\begingroup$ I mean coset space. I will edit. $\endgroup$– SteveJan 6 '18 at 16:41

$\begingroup$ Unlike what you suggest, the question is already interesting when $H$ is normal (see Uri's abelian example). Of course, we can restrict to the case when $G$ is totally disconnected. $\endgroup$– YCorJan 7 '18 at 23:27
Claim 1: For any topological group $G$ and a subgroup $H$, if $G/H$ is totally disconnected (td) then $H<G$ is a closed subgroup containing $G^o$ , the identity connected component of $G$.
Let me denote by $\pi:G\to G/H$ the map given by $\pi(g)=gH$. If $H$ is not closed then $\pi(\bar{H})$ is a connected set which is not a singleton. If $H$ does not contain $G^o$ then $\pi(G^o)$ is a connected set which is not a singleton. In both cases we get that $G/H$ is not td.
For locally compact groups this is an "if and only if".
Claim 2: If $G$ is locally compact and $H<G$ is a closed subgroup containing $G^o$ then $G/H$ is td.
We may identify $G/H$ with $(G/G^o)/(H/G^o)$ thus WLOG, $G=G/G^o$ and we get that $G$ is td locally compact and $H<G$ is a closed subgroup. In this case, by van Danzig theorem, for every $x\notin H$ we can find a compact open subgroup $K<G$ such that $Kx\cap H=\emptyset$, thus $\pi(K)$ is a compact open neighborhood of $eH$ not containing $xH$.
Let me note that there exist td groups (necessarily not locally compact) which have nontrivial connected quotients.
Example: Consider the space $\ell^1$ consisting of real absolutely summable sequences as an additive group and let $G$ be its subgroup consisting of rational sequences. Endow $G$ with the subspace topology. Note that the map $G\to \mathbb{R}$, $x\mapsto \sum x_n$ is a continuous and open surjective homomorphism, thus $\mathbb{R}\simeq G/H$ where $H$ is the subgroup consisting of rational absolutely summable sequences with sum 0. Thus $G/H$ is connected. However, $G$ is totally disconnected as every two points in $G$ could be separated by disjoint open sets, using preimages of rays under the application of functionals in $\ell^\infty\simeq (\ell^1)^*$.
The answer relies crucially on the underlying topology of $G$.
For example if $G$ is a $p$adic group (e.g. $G = \mathrm{GL}_n(\mathbb Q_p)$), then you have infinitely many open compact subgroups $H$ of infinite index (for example $H = 1 + p^N \mathrm{Mat}_n (\mathbb Z_p)$), and due to openness the coset space $G / H$ is totally disconnected.
If instead your $G$ is a real (or complex) Lie group, then any open subgroup $H$ has the same Lie algebra, and thus will be finite index.
I think if you do not further specify some more properties on $G$, it will be hard to give a more precise answer.

2$\begingroup$ Isn't any discrete group a zero dimensional Lie group? Because those surely can have open subgroups of infinite index. $\endgroup$ Jan 6 '18 at 17:30

$\begingroup$ You are correct, I should have said classical Lie group, or positivedimensional Lie group. $\endgroup$ Jan 6 '18 at 18:40

$\begingroup$ Classical should be Ok, positivedimensional is still too weak (just take the direct product of your favorite Lie group with a discrete group). $\endgroup$ Jan 6 '18 at 18:44

3$\begingroup$ For a locally compact group, the question has an immediate answer: $G/H$ is totally disconnected iff $H$ contains $G^0$, as mentioned by Uri. No specific discussion is needed for special classes of locally compact group such as $p$adic Lie groups. $\endgroup$– YCorJan 7 '18 at 23:29