SECOND EDIT: As pointed out in the comments relevant definition can be found in Fuchs' book ,,Cohomology of infinite dimensional Lie algebras''. However still some issues are not quite clear for me:

1. Consider once again the relative modulo Lie-algebra version of cohomology. Relative cochains are antisymmetric and vanish if the first argument is taken from $\mathfrak{h}$: therefore every such cochain may be viewed as element in $Hom(\Lambda^{\bullet}(\mathfrak{g}/\mathfrak{h}),M)$. However authors claim that such cochain may be viewed as element in $Hom_{\mathfrak{h}}(\Lambda^{\bullet}(\mathfrak{g}/\mathfrak{h}),M)$.

Why every such cochain is $\mathfrak{h}$-module map?

The action of $\mathfrak{h}$ should be understood as follows: $\mathfrak{g}/\mathfrak{h}$ is $\mathfrak{h}$-module by $Y \cdot (X +\mathfrak{h}):=[Y,X]+\mathfrak{h}$ and we require our cochain to be $\mathfrak{h}$-linear in each variable.
2. Now I will quote the assumptions given in mentioned above book: assume that $\mathfrak{h}$ is a Lie algebra of some finite dimensional Lie group $H$ and the action of $\mathfrak{h}$ on $\mathfrak{g}$ and $M$ are the differentials of certain representations of $H$, the representation of $H$ in $\mathfrak{g}$ being the extension of adjoint representation of $H$ in $\mathfrak{h}$. Then setting $C^q(\mathfrak{g},H;M):=Hom_H(\Lambda^{\bullet}(\mathfrak{g}/\mathfrak{h}),M)$.

Here I'm running into several difficulties: first of all I suspect that $M$ must be equipped with some manifold structure since we would like to view the action $\mathfrak{H}$ on $M$ as the differential of some representation (I understand it as follows: we have some representation $\pi:H \to Aut(M)$ and we take it differential $d_e\pi:T_eH \to T_e(Aut(M))$ which can be identified with the map $\mathfrak{h} \to End(M)$). When it comes to the action on $\mathfrak{g}$ I'm not sure but as far as I understand it, this is just fancy way of saying that $\mathfrak{h}$ acts on $\mathfrak{g}$ in the standard way, by Lie bracket provided we somehow identify $\mathfrak{h}$ as a Lie subalgebra of $\mathfrak{g}$. Without such identification the sentence ,,the representation of $H$ in $\mathfrak{g}$ being the extension of adjoint representation of $H$ in $\mathfrak{h}$'' is not clear for me, since we would like to extend from $\mathfrak{h}$ to $\mathfrak{g}$.
I would also like to ensure myself how to understand $\mathfrak{g} / \mathfrak{h}$ in this context (if $\mathfrak{h}$ only acts on $\mathfrak{g}$): my guess would be that we identify $X_1,X_2 \in \mathfrak{g}$ if there is some $Y \in \mathfrak{h}, X \in \mathfrak{g}$ such that $X_1-X_2=Y \cdot X$. Is this corect? (I'm afraid that this is not transitive) Finally, we have defined our cochains as elements in $Hom_H(...)$ so they should respect the action of $H$. Now we have assumed that we have representation of $H$ in $M$ and in $\mathfrak{g}$ so the only thing which needs some explanation is whether the action of $H$ on $\mathfrak{g} /\mathfrak{h}$ is well defined. I don't see how to check this since I'm not quite sure how to understand the action of $H$ on $\mathfrak{g}$, as explained above. So to summarize, here are all technicalities which I would like to adress:

a) Do we have to assume that $M$ has some manifold structure?
b) How do we understand the action of $\mathfrak{h}$ on $\mathfrak{g}$?
c) How to understand $\mathfrak{g} / \mathfrak{h}$ in our context?
d) How $H$ acts on $\mathfrak{g} / \mathfrak{h}$?

Forgive me that this question expanded so much: it is quite frustrating when you try to learn one definition and encounter so many difficulties. I'm sure that someone familiar with this stuff would be able to put this in the more understandable form.

Forgive me if this question is too elementary however, I haven't found an answer. If $\mathfrak{g}$ is a Lie algebra one can define its Lie algebra cohomology: the definition is quite similar to the way in which de Rham cohomology is defined. For such theory we can consider coefficients in arbitrary $\mathfrak{g}$-module$ M$. If $\mathfrak{h} \subset \mathfrak{g}$ is a Lie subalgebra one can define the relative theory using subcomplex consisting from cochains $\varphi$ satisfying $i_X \varphi=0$ and $i_X d\varphi =0$ for every $X \in \mathfrak{h}$. This indeed defines subcomplex and the cohomology of this complex is the relative cohomology of $\mathfrak{g}$ (relative to $\mathfrak{h}$).

How one can define cohomology of Lie algebra $\mathfrak{g}$ with respect to some group $H$?

Obviously I'm aware that $H$ must have something to do with $\mathfrak{g}$. I found one definition but I'm not sure whether I should cite it here since I'm not sure whether it makes sense (if I should, let me know in the comments and I will edit my post).

I would be very grateful if someone who is familiar with this topic could shed some light on it.

Forgive me if this question is too elementary however, I haven't found an answer. If $\mathfrak{g}$ is a Lie algebra one can define its Lie algebra cohomology: the definition is quite similar to the way in which de Rham cohomology is defined. For such theory we can consider coefficients in arbitrary $\mathfrak{g}$-module$ M$. If $\mathfrak{h} \subset \mathfrak{g}$ is a Lie subalgebra one can define the relative theory using subcomplex consisting from cochains $\varphi$ satisfying $i_X \varphi=0$ and $i_X d\varphi =0$ for every $X \in \mathfrak{h}$. This indeed defines subcomplex and the cohomology of this complex is the relative cohomology of $\mathfrak{g}$ (relative to $\mathfrak{h}$).

How one can define cohomology of Lie algebra $\mathfrak{g}$ with respect to some group $H$?

Obviously I'm aware that $H$ must have something to do with $\mathfrak{g}$. I found one definition but I'm not sure whether I should cite it here since I'm not sure whether it makes sense (if I should, let me know in the comments and I will edit my post).

I would be very grateful if someone who is familiar with this topic could shed some light on it.

EDIT: The only definition which I saw is the following: we assume that $H$ is a Lie group such that $\mathfrak{h}:=Lie(H)$ acts on $\mathfrak{g}$ and on $M$ (the coefficient module) such that the differential of the action on $\mathfrak{g}$ is $ad_{\mathfrak{g}} \mathfrak{h}$. We consider the complex: $$C^{\bullet}(\mathfrak{g},H;M):=\{ \varphi \in Hom_H(\Lambda^{\bullet}\mathfrak{g},M): i_X \varphi =0 \ \ \forall_{X \in \mathfrak{h}}\}$$ and its cohomology. Maybe I should add that the particular case in which I'm interested is the case when $\mathfrak{g}$ is Lie algebra of all formal vector fields on $\mathbb{R}^n$ and $H=SO(n)$. More context in which I'm interested can be found in this post.