Let $F$ be a field of characteristic $p > 0$. Let $\mathfrak{g}$ be a linear Lie algebra, that is $\mathfrak{g}\subset M_n(F)$ for some natural number $n$. Does there exist a condition involving $n$ and $p$ such that $\mathfrak{g}$ is semisimple if and only if its Killing form is nondegenerate?
3 Answers
The short answer is no. In prime characteristic, the Killing form sometimes behaves badly even for simple Lie algebras. If "semisimple" means that the solvable radical is zero, there is no way to obtain the classical equivalences with nondegeneracy of the Killing form and with the direct sum decomposition into simples. Moreover, the simple Lie algebras have only recently been classified when $p=5$ (PremetStrade), while for $p=2,3$ little is known and for $p>5$ the classification takes an enormous amount of work (by BlockWilson and others). Conditions on the dimension of a faithful representation or on the prime are not enough to sort out the concept of semisimplicity. Still, a lot is known. For example, Seligman and others explored in the 1960s the class of modular Lie algebras for which the Killing form is nondegenerate.
ADDED: Much more could be said along these lines, but for older results see the book Modular Lie Algebras by G.B. Seligman (Springer, 1967). Modular Lie algebras have been of much less importance overall than linear algebraic groups, whose Lie algebras are "restricted" (have a nice $p$th power operation) but still don't reflect precisely the group structure or representation theory. Moreover, the representations of simple or more generally semisimple Lie algebras in prime characteristic are poorly understood even for those arising from Lie algebras of semisimple algebraic groups. So working with a fixed $p$ and fixed $n$ will usually not be illuminating.

$\begingroup$ Thanks a lot for your reply Jim. Although it was bit disheartening to know that very little is known about this problem. $\endgroup$ Nov 21, 2010 at 19:17

2$\begingroup$ In some ways a lot is actually known, but your original formulation doesn't get at that. While enormous progress has been made on finding all simple Lie algebras (starting with the restricted ones), far less is known about their linear representations. The notion of semisimple in prime characteristic still looks too opaque to be explored systematically, using the range of tools now available. $\endgroup$ Nov 22, 2010 at 23:03
If $g\subset M_n(F)$ and $n\le p2$ then $g$ is semisimple if and only if the Killing form of g is nondegenerate. This statement is clear (and empty) if $p\in\{2,3\}$, while for $p>3$ one can use various results of the theory of modular Lie algebras. First one may assume that $g$ is semisimple as the converse statement is an easy exercise. One may also assume that $F$ is algebraically closed. Using Block's theorem one the structure of semisimple Lie algebras we then observe that $g$ is sandwiched between $\bigoplus_i S_i$ and $\bigoplus_i{\rm Der}(S_i)$ where the $S_i$'s are simple Lie algebras. As $n\le p2$ the Classification Theorem implies that each $S_i$ is isomorphic to the Lie algebra of a simple algebraic $F$group. Looking at the ranks of these groups more closely (and again using the inequality $n\le p2$) we deduce that each $S_i$ has a nondegenerate Killing form and ${\rm Der}(S_i)=\mathrm{ad}\, S_i$ (this is discussed in Seligman's book quoted earlier). Hence the Killing form of $g\cong\bigoplus_i S_i$ is nondegenerate as well. The assumption on $p$ cannot be relaxed as for $p>3$ the Witt algebra $g={\rm Der}\,\mathcal{O}$ where $\mathcal{O}=F[X]/(X^p)$ is simple, has the zero Killing form, and acts irreducibly on the vector space $\mathcal{O}/F1$ of dimension $p1$.
This started out as a comment on Jim's answer, but it got too long (EDIT : It looks like Jim edited his answer to say some of this while I wrote this up).
First, I recommend Seligman's book "Modular Lie Algebras" for a nice account of what happens to Lie algebras in positive char if you assume that the Killing form is nondegenerate.
It's maybe also worth giving an example to show what can happen (there might be easier ones, but this came up in a paper I wrote a while ago and gave me no end of headaches). Consider $\mathfrak{sp}_{2g}(\mathbb{Z}/p\mathbb{Z})$. It is not hard to show that this is simple for $p > 2$. For instance, an argument is contained in the paper
N. Jacobson, Classes of restricted Lie algebras of characteristic p. I, Amer. J. Math. 63 (1941), 481–515.
By the way, this is false for $p=2$. Anyway, one can calculate the Killing form, and it turns out to be degenerate exactly when $p$ divides $g+1$. One interesting observation, however, is that we are really looking at the wrong bilinear form. Elements of $\mathfrak{sp}_{2g}(\mathbb{Z}/p\mathbb{Z})$ are matrices, so we can define a blinear form
$$(A,B) = \text{Tr}(AB).$$
This is NOT the Killing form, as we are not looking at the adjoint representation. One can then show that this bilinear form is nondegenerate for all $p>2$.

2$\begingroup$ In characteristic 0, this form would just be off by a factor from the Killing form, right? Something like 4(g+1), maybe. The factor of 4 would also explain why it always fails for p=2. $\endgroup$ Nov 21, 2010 at 16:49

2$\begingroup$ Thanks a lot for your comments Andy, these are very interesting for me. In fact the remark that you mentioned about bilinear form $(A,B) = Tr(AB)$ for $sp_{2g}(Z/pZ)$, is also true for $sl_n(Z/pZ)$, i.e. trace form is non degenerate on $sl_n(Z/pZ)$ for all $p >2$. I was just wondering if you have any other comments about this specific bilinear on other linear semisimple lie algebras (i.e. is it nondegenerate or something)? For my purposes this form may be good as well. $\endgroup$ Nov 21, 2010 at 19:12

2$\begingroup$ Let's take semisimple to mean that the solvable radical is zero. Then the usual proof in characteristic $0$ goes through in characteristic $p$ to show that if a Lie algebra $L$ has a nondegenerate bilinear form with appropriate invariance properties (similar to the Killing form), then $L$ is semisimple. The converse, alas, is false, but in many examples you can find good bilinear forms like the one I mentioned above. Seligman's book has good information on this (though since it is old I expect that more is known). $\endgroup$ Nov 21, 2010 at 20:09

2$\begingroup$ (continued) I don't have Seligman's book handy, but in the paper I mentioned in my answer I refer to page 47 of it, so that is probably a good place to start looking. Be warned, however, that semisimplicity doesn't work in char p like it does in char 0; for instance, semisimple Lie algebras in char p are not necessarily direct sums of simple Lie algebras. $\endgroup$ Nov 21, 2010 at 20:13

3$\begingroup$ @Jim: The question you mentioned seems to be solved by S.R. Garibaldi and A.A. Premet, Vanishing of trace forms in low characteristics, Algebra and Number Theory 3 (2009), No.5, 543566, arXiv:0712.3764 $\endgroup$ Jan 4, 2011 at 15:27