Forgive me if this question is too elementary however, I haven't found an answer. If $\mathfrak{g}$ is a Lie algebra one can define its Lie algebra cohomology: the definition is quite similar to the way in which de Rham cohomology is defined. For such theory we can consider coefficients in arbitrary $\mathfrak{g}$-module$ M$. If $\mathfrak{h} \subset \mathfrak{g}$ is a Lie subalgebra one can define the *relative* theory using subcomplex consisting from cochains $\varphi$ satisfying $i_X \varphi=0$ and $i_X d\varphi =0$ for every $X \in \mathfrak{h}$. This indeed defines subcomplex and the cohomology of this complex is the relative cohomology of $\mathfrak{g}$ (relative to $\mathfrak{h}$).

How one can define cohomology of Lie algebra $\mathfrak{g}$ with respect to some

group$H$?

Obviously I'm aware that $H$ must have something to do with $\mathfrak{g}$. I found one definition but I'm not sure whether I should cite it here since I'm not sure whether it makes sense (if I should, let me know in the comments and I will edit my post).

I would be very grateful if someone who is familiar with this topic could shed some light on it.

EDIT: The only definition which I saw is the following: we assume that $H$ is a Lie group such that $\mathfrak{h}:=Lie(H)$ acts on $\mathfrak{g}$ and on $M$ (the coefficient module) such that the differential of the action on $\mathfrak{g}$ is $ad_{\mathfrak{g}} \mathfrak{h}$. We consider the complex: $$C^{\bullet}(\mathfrak{g},H;M):=\{ \varphi \in Hom_H(\Lambda^{\bullet}\mathfrak{g},M): i_X \varphi =0 \ \ \forall_{X \in \mathfrak{h}}\}$$ and its cohomology. Maybe I should add that the particular case in which I'm interested is the case when $\mathfrak{g}$ is Lie algebra of all formal vector fields on $\mathbb{R}^n$ and $H=SO(n)$. More context in which I'm interested can be found in this post.

SECOND EDIT: As pointed out in the comments relevant definition can be found in Fuchs' book ,,Cohomology of infinite dimensional Lie algebras''. However still some issues are not quite clear for me:

1. Consider once again the relative modulo Lie-algebra version of cohomology. Relative cochains are antisymmetric and vanish if the first argument is taken from $\mathfrak{h}$: therefore every such cochain may be viewed as element in $Hom(\Lambda^{\bullet}(\mathfrak{g}/\mathfrak{h}),M)$. However authors claim that such cochain may be viewed as element in $Hom_{\mathfrak{h}}(\Lambda^{\bullet}(\mathfrak{g}/\mathfrak{h}),M)$.

Why every such cochain is $\mathfrak{h}$-module map?

The action of $\mathfrak{h}$ should be understood as follows: $\mathfrak{g}/\mathfrak{h}$ is $\mathfrak{h}$-module by $Y \cdot (X +\mathfrak{h}):=[Y,X]+\mathfrak{h}$ and we require our cochain to be $\mathfrak{h}$-linear in each variable.

2. Now I will quote the assumptions given in mentioned above book: assume that $\mathfrak{h}$ is a Lie algebra of some finite dimensional Lie group $H$ and the action of $\mathfrak{h}$ on $\mathfrak{g}$ and $M$ are the differentials of certain representations of $H$, the representation of $H$ in $\mathfrak{g}$ being the extension of adjoint representation of $H$ in $\mathfrak{h}$. Then setting $C^q(\mathfrak{g},H;M):=Hom_H(\Lambda^{\bullet}(\mathfrak{g}/\mathfrak{h}),M)$.

Here I'm running into several difficulties: first of all I suspect that $M$ must be equipped with some manifold structure since we would like to view the action $\mathfrak{H}$ on $M$ as the differential of some representation (I understand it as follows: we have some representation $\pi:H \to Aut(M)$ and we take it differential $d_e\pi:T_eH \to T_e(Aut(M))$ which can be identified with the map $\mathfrak{h} \to End(M)$). When it comes to the action on $\mathfrak{g}$ I'm not sure but as far as I understand it, this is just fancy way of saying that $\mathfrak{h}$ acts on $\mathfrak{g}$ in the standard way, by Lie bracket *provided* we somehow identify $\mathfrak{h}$ as a Lie subalgebra of $\mathfrak{g}$. Without such identification the sentence ,,the representation of $H$ in $\mathfrak{g}$ being the extension of adjoint representation of $H$ in $\mathfrak{h}$'' is not clear for me, since we would like to extend from $\mathfrak{h}$ to $\mathfrak{g}$.

I would also like to ensure myself how to understand $\mathfrak{g} / \mathfrak{h}$ in this context (if $\mathfrak{h}$ only *acts* on $\mathfrak{g}$): my guess would be that we identify $X_1,X_2 \in \mathfrak{g}$ if there is some $Y \in \mathfrak{h}, X \in \mathfrak{g}$ such that $X_1-X_2=Y \cdot X$. Is this corect? (I'm afraid that this is not transitive)
Finally, we have defined our cochains as elements in $Hom_H(...)$ so they should respect the action of $H$. Now we have assumed that we have representation of $H$ in $M$ and in $\mathfrak{g}$ so the only thing which needs some explanation is whether the action of $H$ on $\mathfrak{g} /\mathfrak{h}$ is well defined. I don't see how to check this since I'm not quite sure how to understand the action of $H$ on $\mathfrak{g}$, as explained above. So to summarize, here are all technicalities which I would like to adress:

a) Do we have to assume that $M$ has some manifold structure?

b) How do we understand the action of $\mathfrak{h}$ on $\mathfrak{g}$?

c) How to understand $\mathfrak{g} / \mathfrak{h}$ in our context?

d) How $H$ acts on $\mathfrak{g} / \mathfrak{h}$?

Forgive me that this question expanded so much: it is quite frustrating when you try to learn one definition and encounter so many difficulties. I'm sure that someone familiar with this stuff would be able to put this in the more understandable form.

not"define" these cohomologies in terms of the standard complexes or variants thereof, but as derived functors of fixed-vector or co-fixed-vector functors. From that viewpoint, structure beyond the usual can easily be accommodated from a mildly categorical viewpoint... Is this a helpful direction of discussion for you? $\endgroup$ – paul garrett Dec 22 '17 at 23:32group) theory via derived functors I will also be happy with it. $\endgroup$ – truebaran Dec 22 '17 at 23:39