Let $\mathfrak{g}$ be a semisimple finite dimensional Lie algebra over $\mathbb C$. Is it true that $$ \dim \mathfrak h \ge \mathop{rank} \mathfrak g $$ for any maximal abelian subalgebra $\mathfrak h \subset \mathfrak g$ ? Here by maximal I mean that $\mathfrak h$ is not properly contained in any other abelian subalgebra of $\mathfrak g$.
I think the answer to my question is no. A search at Mathscinet revealed the following:
Gerstenhaber conjectured in [Ann. of Math. (2) 73 (1961), 324348; MR0132079 (24 #A1926)] that a maximal commutative subalgebra of the associative ring of $n\times n$ matrices $M_n(k)$ always has dimension $\ge n$, and proved this is the case for subalgebras generated by two elements.
The conjecture was disproved by Courter in [Duke Math. J. 32 (1965), 225232; MR0174583 (30 #4784)], by constructing a maximal commutative subalgebra of $M_{14}(k)$ having dimension $13$. Tensor powers give further examples of dimension $13^m$ in $M_{14^m}(k)$.
Gustafson [J. Algebra 42 (1976), no. 2, 557563; MR0422341 (54 #10331)] proved the lower bound $\ge n^{2/3}$ for arbitrary maximal commutative subalgebras of $M_n(k)$.
More recently, Laffey [Linear Algebra Appl. 71 (1985), 199212; MR0813045 (87a:15025)] improved the lower bound to $(2n)^{2/3}$ and exhibited infinitely many maximal commutative subalgebras of $M_n(k)$ having dimension $3n^{2/3}  4$.
I haven't looked at the papers mentioned above yet but they seem to imply that there are maximal abelian Lie subalgebras of $\mathfrak{gl}(n,k)$ ( or $\mathfrak{sl}(n,k)$ ) having dimension smaller than $n1$.
Indeed, start with a maximal commutative subalgebra $H \subset M_n(k)$. Since, it is commutative the Lie bracket $[A,B]=ABBA$ is trivial over it. Therefore it gives rise to an abelian subalgebra $\mathfrak h \subset \mathfrak{gl}(n,k)$ which must be maximal. Otherwise the associative subalgebra generated by any abelian Lie subalgebra containing it would be a commutative subalgebra of $M_n(k)$ strictly containg $H$.
I finally took a look at the papers mentioned above. Courter's example is the subalgebra of $M_{14}(k)$ generated by the Identity and elements of the form $$ \left( \begin{array}{cccccccccccccc} 0&0&a&0&b&0&c&e&0&f&0&g&m&n\\ 0&0&0&a&0&b&d&0&e&0&f&h&p&q\\ & & & & & & & & & & & &c&0\\ & & & & & & & & & & & &d&0\\ & & & & & & & & & & & &0&c\\ & & & & & & & & & & & &0&d\\ & & & & & & & & & & & &a&b\\ & & & & & & & & & & & &g&0\\ & & & & & &0 & & & & & &h&0\\ & & & & & & & & & & & &0&g\\ & & & & & & & & & & & &0&h\\ & & & & & & & & & & & &e&f\\ & & & & & & & & & & & &0&0\\ & & & & & & & & & & & &0&0 \end{array} \right) $$ where $a,b,c,d,e,f,g,h,m,n,p,q \in k$ are arbitrary. These very same elements generate a maximal abelian subalgebra of $\mathfrak{sl}(14,k)$ of dimension $12$ as I, or rather my computer, checked.

$\begingroup$ Interesting. Well, since these results have all been refereed and I am just some guy on the internet, I hope I can find my mistake! It will have to wait til tomorrow, though. $\endgroup$ – Sheikraisinrollbank Dec 12 '10 at 20:56

$\begingroup$ I'm skeptical about the sketch above, notably the final statement. Details? Associative and Lie subalgebras of the matrix algebra need to be compared more carefully. I still haven't pinned down a literature reference for the original question, but have some recollection that the rank is known to be least dimension of a maximal abelian Lie subalgebra. It may be useful to look at some of the older papers by Zassenhaus et al. In any case, the question has certainly come up in the past. $\endgroup$ – Jim Humphreys Dec 12 '10 at 23:25

$\begingroup$ I added backticks (the symbol to the left of the 1, on an american QWERTY keyboard) around your TeX, and removed some spurious curly braces, in order to get it to display. $\endgroup$ – David E Speyer Dec 15 '10 at 17:37

$\begingroup$ @jvp: See my expanded answer concerning the problem I have with your approach. Study of the full matrix algebra, or general linear Lie algebra, isn't directly relevant for the study of semisimple Lie algebras. $\endgroup$ – Jim Humphreys Dec 15 '10 at 18:04

1$\begingroup$ @jim I don't see what your worry can be. The space jvp presents is easily seen to be closed under multiplication. So it is both a subalgebra and a subLiealgebra of $sl_{12}$. If you want algebras to have units, then the span of it and the identity is a subalgebra of $gl_{12}$. I have not checked the computation that this is abelian and maximal, but if that computation is right, it should be right both in the sense of Lie algebras and associative algebras. $\endgroup$ – David E Speyer Dec 15 '10 at 20:48
[Edited again. This argument has a fatal flaw (and the conclusion is probably false; see jvp's answer). Lessons: random maximal abelian subalgebras might be pretty weird, selfnormalizing is a much nicer condition; check mathscinet more carefully; don't abuse Jordan normal form, it might bite back.]
There is a mistake below: it is not true that the kernel of a generic linear combination of commuting nilpotent matrices is the intersection of the kernels. For a counterexample, take x to be a 4 by 4 matrix which is the direct sum of two 2 by 2 Jordan blocks, and take y to have the same block diagonal entries, and a 2 by 2 identity matrix in its upper right hand corner. Then the kernel of $ax+by$ is only contained in the kernel of x when b=0. I sincerely hope I have never assigned this "fact" as a homework problem...
The rest is preserved for your amusement.
Here goes again:
The answer is yes. The proof rests on 2 facts:
(1) in a semisimple Lie algebra of rank $n$, the minimal dimension of the centralizer $C(x)$ of an element $x$ is $n$, and
(2) in any Lie algebra (say over $\mathbb{C}$, though surely this works in greater generality) every maximal abelian subalgebra is the centralizer of one of its elements.
For (1):
The set $$C=\{(x,y) \in \mathfrak{g} \times \mathfrak{g} \  \ [x,y]=0\}$$ is a subvariety of $\mathfrak{g} \times \mathfrak{g}$ whose fiber over $x \in \mathfrak{g}$ is the centralizer $C(x)$ of $x$ in $\mathfrak{g}$. For generic (regular semisimple) $x$ the dimension of $C(x)$ is $n=\text{rank}(\mathfrak{g})$. It follows that the dimension of $C(x)$ is always at least $n$.
For (2):
Let $\mathfrak{h}$ be a maximal abelian subalgebra. If $x_1,...,x_m$ are a basis of $\mathfrak{h}$ and $z \in \bigcap_{i=1}^m C(x_i)$ then $[z,\mathfrak{h}]=0$ and hence $z \in \mathfrak{h}$ by maximality of $\mathfrak{h}$. It follows that $$\mathfrak{h}=\bigcap_{i=1}^m C(x_i).$$
Now we claim that the intersection $\bigcap_{i=1}^m C(x_i)$ is the centralizer of a generic linear combination $\sum_{i=1}^m a_i x_i$. It suffices to show that there exist $a_1,\dots,a_m \in \mathbb{C}$ such that $$\bigcap_{i=1}^m C(x_i)=C(\sum_{i=1}^m a_i x_i).$$ By induction on $m$ we may assume $m=2$. Since $C(x)=\mathrm{ker}(\mathrm{ad}(x))$ we have reduced to the following linear algebra problem: let $x$ and $y$ be commuting matrices with complex entries. Then for generic $a,b \in \mathbb{C}$ the kernel of $ax+by$ is the intersection of the kernels of $x$ and $y$.
Let $x=x_s+x_n$ be the decomposition of $x$ into commuting semisimple and nilpotent parts, and likewise write $y=y_s+y_n$. Then since $x$ and $y$ commute $ax+by=a(x_s+y_s)+b(x_n+y_n)$ is the decomposition of $ax+by$ into commuting nilpotent and semisimple parts. Furthermore $\mathrm{ker}(x)=\mathrm{ker}(x_s) \cap \mathrm{ker}(x_n)$, so we may assume that $x$ and $y$ are either both semisimple or both nilpotent.
If $x$ and $y$ are both semisimple (and commuting) then it's easy to see that the kernel of $ax+by$ is the intersection of the kernels of $x$ and $y$ for generic $a,b \in \mathbb{C}$.
In the remaining case $x$ and $y$ are both nilpotent, and since they commute we may choose a basis such that $x$ is in Jordan form and $y$ is (strictly) upper triangular. It follows that as long as $a+bc \neq 0$ for all superdiagonal entries $c$ of $y$, we have $\mathrm{ker}(ax+by) \subseteq \mathrm{ker}(x)$. Symmetrically, $\mathrm{ker}(ax+by) \subseteq \mathrm{ker}(y)$ for generic $a,b \in \mathbb{C}$, done.
[Remark: I'm pretty sure I've assigned the proof of this last fact (that the kernel of a generic linear combination $ax+by$ is the intersection of the kernels if $x$ and $y$ commute) as a homework problem at some point, which is why I was originally so cavalier. But I have to admit I can't find it written down somewhere so maybe I'm imagining things. Can anyone provide a reference?]

1$\begingroup$ The intersection of centralisers does not have to be abelian. Thus it is not clear why it is equal to $h$. $\endgroup$ – Bugs Bunny Dec 9 '10 at 22:55

1$\begingroup$ Bugs: whether or not it is abelian is irrelevant: let $z$ be in this intersection. Then $[z,\mathfrak{h}]=0$ so $\mathfrak{h}+\mathbb{C} z$ is abelian, and hence $z \in \mathfrak{h}$ by maximality. $\endgroup$ – Sheikraisinrollbank Dec 9 '10 at 23:34

1$\begingroup$ Sheik: i do not see why the intersection of centralizers is the centralizer of a linear combination. $\endgroup$ – Jorge Vitório Pereira Dec 10 '10 at 0:50

1$\begingroup$ This argument doesn't work. Let $\mathfrak{g}$ be
$\mathfrak{sl}_{2n}$
and let $\mathfrak{h}$ be the $n^2$ dimensional subspace of matrices which are $0$ except in the upper right $n \times n$ submatrix. So $\mathfrak{h}$ is abelian. For any $x \in \mathfrak{h}$, we have $x^2=0$, so the Jordan form of $x$ is a direct sum of$\left( \begin{smallmatrix} 0 & 1 \\ 0 & 0 \end{smallmatrix} \right)$'s
and $(0)$'s. As the reader can work out, the centralizer of such a matrix always has dimension at least $2n^21$. $\endgroup$ – David E Speyer Dec 10 '10 at 13:39 
$\begingroup$ David: urgh, thanks for pointing that out. I was just about to get to 4 by 4 examples. $\endgroup$ – Sheikraisinrollbank Dec 10 '10 at 15:19
Some of my previous comments were unfortunately too casual and unfocused, but I still suspect that the answer to the original question is yes. At the same time, I can't document precisely enough what is known about the exceptional simple Lie algebras over $\mathbb{C}$. I'm not aware of any unified Lietheoretic arguments, in any case.
For the classical simple types (including the trace zero matrices?), the older papers are sometimes hard to read but have many concrete details. The main tool used is linear algebra, inspired in part by a long tradition in the study of (associative) matrix algebras going back to Schur, Kravchuk, and Mal'cev. In particular, there is a series of papers by Patera, Winternitz, and Zassenhaus on classical real or complex Lie algebras, some published in the journal Linear Algebra and its Applications which is in our university library but not available online to us. One article I can access online is typical:
MR713527 (84i:17006) 17B05 (22E10), Patera, J. (3MTRLR); Winternitz, P. (3MTRLR); Zassenhaus, H. (1OHS), Maximal abelian subalgebras of real and complex symplectic Lie algebras. J. Math. Phys. 24 (1983), no. 8, 1973–1985.
Their emphasis is on classifying the maximal ones up to conjugacy under the adjoint group, with emphasis on those like the Cartan subalgebras and the contrasting ones consisting of nilpotent matrices. Dimension considerations seem secondary. But as far as I can see, over $\mathbb{C}$ they always find that the minimum dimension of a maximal abelian subalgebra is equal to the rank, while the maximum dimension tends to be larger.
Concerning jvp's own references to work of Gerstenhaber and others on the full matrix algebra, it has to be kept in mind that the simple linear Lie algebras in question aren't usually closed under ordinary matrix multiplication. This makes comparisons of commutative subalgebras and abelian Lie algebras tricky, I think. However, I haven't gone far enough into the literature to feel confident about exactly what is true. It would help to have a modern survey of the entire problem area for both matrix algebras and semisimple Lie algebras.
ADDED: Though I haven't yet seen the Courter paper or his precomputer proof method, the description here and jvp's reported computer verification suggests that for type A this Lie subalgebra of the special linear algebra consisting of nilpotent upper triangular matrices is maximal and of dimension less than the rank. On the other hand, the published results for classical simple Lie algebras suggest that the rank is a lower bound in those types. I'm uncertain how much is known about the exceptional types, so I guess I'd emphasize the last line of my earlier answer and the related question whether a Lietheoretic understanding of the problem exists (beyond linear algebra). But I'm happy to drop my original guess that the rank is always the minimum dimension if Courter's proof says otherwise.

$\begingroup$ @Jim. Thanks for the comments. Most of the arguments I've found in the literature are commutative algebra together with clever linear algebra. I agree with you that a modern survey of the problem area would very welcome. I am even tempted to try my hand at it, but I doubt I will have time in a near future. $\endgroup$ – Jorge Vitório Pereira Dec 17 '10 at 1:23
$G_2$
suggests how natural examples arise with dimension$>2$
: If$\alpha$
/$\beta$
are the short/long simple roots, root vectors belonging to$\alpha, 3\alpha+\beta, 3\alpha+2\beta$
span a 3dimensional abelian ideal in a Borel subalgebra. (The highest root always gives a vector in the center of the nilradical.) $\endgroup$ – Jim Humphreys Dec 9 '10 at 23:06