For affine (= linear) algebraic groups and their Lie algebras over an algebraically closed field of characteristic 0, there are fairly elementary discussions of the connection between their actions (including finite dimensional representations): see the three books titled Linear Algebraic Groups by Borel, Springer, and me. While the style of these treatments varies, the basic result is that the Lie group theory carries over well to the algebraic group setting in characteristic 0. For example, invariants of the group correspond naturally to invariants of the Lie algebra. In my book, this is all discussed in Sections 13-14. In prime characteristic everything gets more subtle, at which point Jantzen's more advanced book is helpful but requires more scheme language.

If an algebraic group in characteristic 0 isn't affine, older theorems of Barsotti and Chevalley show how to reduce most questions to the cases of affine groups and abelian varieties. (Brian Conrad provided a more modern proof of Chevalley's theorem. See his homepage here.) For the latter, Lie algebra ideas don't seem to contribute anything extra to the question asked here.

For affine (= linear) algebraic groups and their Lie algebras over an algebraically closed field of characteristic 0, there are fairly elementary discussions of the connection between their actions (including finite dimensional representations): see the three books titled Linear Algebraic Groups by Borel, Springer, and me. While the style of these treatments varies, the basic result is that the Lie group theory carries over well to the algebraic group setting in characteristic 0. For example, invariants of the group correspond naturally to invariants of the Lie algebra. In my book, this is all discussed in Sections 13-14. In prime characteristic everything gets more subtle, at which point Jantzen's more advanced book is helpful but requires more scheme language.

If an algebraic group in characteristic 0 isn't affine, older theorems of Barsotti and Chevalley show how to reduce most questions to the cases of affine groups and abelian varieties. (Brian Conrad provided a more modern proof of Chevalley's theorem. See his homepage here.) For the latter, Lie algebra ideas don't seem to contribute anything extra to the question asked here.

ADDED: Maybe it's helpful to comment further on the original question as well as the other answers.

1. If there is a definite reason to consider the notion of "Lie algebra of a non-affine algebraic group" over an algebraically closed field of characteristic 0, it's essential to define the notion explicitly or at least give a reference. In his 1950s work, Chevalley imitated successfully the correspondence between Lie groups and Lie algebras when the Lie group is replaced by a linear algebraic group in characteristic 0. This is essentially the material covered in the three textbooks I mentioned above, which provide clear answers to the questions raised in the affine/linear case. As noted, the Lie algebra by itself is too weak a tool in prime characteristic.

2. The paper by Brian Conrad updates the older language used for Chevalley's theorem on general algebraic groups. But note that Conrad always requires the groups to be connected, including the affine closed normal subgroup in the theorem. And there seems to be no point in working with Lie algebras in this general setting.

For affine (= linear) algebraic groups and their Lie algebras over an algebraically closed field of characteristic 0, there are fairly elementary discussions of the connection between their actions (including finite dimensional representations): see the three books titled Linear Algebraic Groups by Borel, Springer, and me. While the style of these treatments varies, the basic result is that the Lie group theory carries over well to the algebraic group setting in characteristic 0. For example, invariants of the group correspond naturally to invariants of the Lie algebra. In my book, this is all discussed in Sections 13-14. In prime characteristic everything gets more subtle, at which point Jantzen's more advanced book is helpful but requires more scheme language.

If an algebraic group in characteristic 0 isn't affine, older theorems of Barsotti and Chevalley show how to reduce most questions to the cases of affine groups and abelian varieties. (Brian Conrad provided a more modern proof of Chevalley's theorem.) For the latter, Lie algebra ideas don't seem to contribute anything extra to the question asked here.

For affine (= linear) algebraic groups and their Lie algebras over an algebraically closed field of characteristic 0, there are fairly elementary discussions of the connection between their actions (including finite dimensional representations): see the three books titled Linear Algebraic Groups by Borel, Springer, and me. While the style of these treatments varies, the basic result is that the Lie group theory carries over well to the algebraic group setting in characteristic 0. For example, invariants of the group correspond naturally to invariants of the Lie algebra. In my book, this is all discussed in Sections 13-14. In prime characteristic everything gets more subtle, at which point Jantzen's more advanced book is helpful but requires more scheme language.

If an algebraic group in characteristic 0 isn't affine, older theorems of Barsotti and Chevalley show how to reduce most questions to the cases of affine groups and abelian varieties. (Brian Conrad provided a more modern proof of Chevalley's theorem. See his homepage here.) For the latter, Lie algebra ideas don't seem to contribute anything extra to the question asked here.