This is interesting but gets complicated, so maybe it's useful to supplement Vladimir's answer with more details.

1. Smale is applying a fairly elementary but not quite standard algebraic fact about Lie algebras to show that a certain real Lie algebra is nilpotent (and therefore is the Lie algebra of a nilpotent Lie group, by the usual correspondence). As he notes in his 1967 paper, Armand Borel suggested this exercise from the 1960 Chapter 1 in Bourbaki's treatise; Borel was himself active in Bourbaki, of course.

2. Chapter 4 of Bourbaki is devoted to finite dimensional nilpotent Lie algebras over an arbitrary field. This includes Engel's Theorem: a Lie algebra is nilpotent iff every operator ad $x$ is nilpotent. Since the kernel of the adjoint representation is abelian, it's usually easy to reduce questions to a *;inear" Lie algebra. As occurs elsewhere in Bourbaki, this chapter is augmented by a varied list of exercises; some are marked with the special symbol for "challenging", for instance exer. 21 here (and the earlier exer. 11 to which it refers). These exercises are often quite interesting, but also frustrating when they provide no context or source.

3. Exercise 21 concerns a given Lie algebra $\mathfrak{g}$ having an automorphism $\sigma$. In part (a) it's observed that the bracket of two generalized eigenvectors is again a generalized eigenvector (for the product of the two eigenvalues), as noted by Vladimir. For the proof it's harmless to extend the base field to an algebraic closure. This is a bit tricky, so there is a helpful observation:

$(\sigma - \lambda \mu) [xy] = [(\sigma -\lambda)x, \sigma y] + [\lambda x, (\sigma-\mu)y]$.

4. Then part (b) of the exercise, cited by Smale, assumes that no eigenvalue of $\sigma$ in an algebraic closure of the field is a root of unity. From this it follows that $\mathfrak{g}$ is nilpotent. Again it's harmless to pass to an algebraic closure. Vladimir outlines the method of proof, which in Bourbaki refers back to Exercise 11. Actually, the needed discussion is done in II.2 of Jacobson's 1962 book Lie Algebras. While Jacobson's style differs a lot from Bourbaki's, he tends to include all the details for these relatively elementary steps.

Concerning the remarks by Yves, it should be emphasized that his proposed approach to the question requires at least that the field have characteristic 0 (not a problem for Smale, but a further complication here). Arguing in terms of semisimple Lie algebras and nondegeneracy of the Killing form (to ensure that derivations are inner) doesn't generalize at all well to other fields.

This is interesting but gets complicated, so maybe it's useful to supplement Vladimir's answer with more details.

1. Smale is applying a fairly elementary but not quite standard algebraic fact about Lie algebras to show that a certain real Lie algebra is nilpotent (and therefore is the Lie algebra of a nilpotent Lie group, by the usual correspondence). As he notes in his 1967 paper, Armand Borel suggested this exercise from the 1960 Chapter 1 in Bourbaki's treatise; Borel was himself active in Bourbaki, of course.

2. Chapter 4 of Bourbaki is devoted to finite dimensional nilpotent Lie algebras over an arbitrary field. This includes Engel's Theorem: a Lie algebra is nilpotent iff every operator ad $x$ is nilpotent. Since the kernel of the adjoint representation is abelian, it's usually easy to reduce questions to a *;inear" Lie algebra. As occurs elsewhere in Bourbaki, this chapter is augmented by a varied list of exercises; some are marked with the special symbol for "challenging", for instance exer. 21 here (and the earlier exer. 11 to which it refers). These exercises are often quite interesting, but also frustrating when they provide no context or source.

3. Exercise 21 concerns a given Lie algebra $\mathfrak{g}$ having an automorphism $\sigma$. In part (a) it's observed that the bracket of two generalized eigenvectors is again a generalized eigenvector (for the product of the two eigenvalues), as noted by Vladimir. For the proof it's harmless to extend the base field to an algebraic closure. Starting the inductive proof is a bit tricky, so Bourbaki makes a helpful observation:

$(\sigma - \lambda \mu) [xy] = [(\sigma -\lambda)x, \sigma y] + [\lambda x, (\sigma-\mu)y]$.

4. Then part (b) of the exercise, cited by Smale, assumes that no eigenvalue of $\sigma$ in an algebraic closure of the field is a root of unity. From this it follows that $\mathfrak{g}$ is nilpotent. Again it's harmless to pass to an algebraic closure. Vladimir outlines the method of proof, which in Bourbaki refers back to Exercise 11. Actually, the needed discussion is done in II.2 of Jacobson's 1962 book Lie Algebras. While Jacobson's style differs a lot from Bourbaki's, he tends to include all the details for these relatively elementary steps.

Concerning the remarks by Yves, it should be emphasized that his proposed approach to the question requires at least that the field have characteristic 0 (not a problem for Smale, but a further complication here). Arguing in terms of semisimple Lie algebras and nondegeneracy of the Killing form (to ensure that derivations are inner) doesn't generalize at all well to other fields.