Is there a general method to calculate the cohomology groups of homogeneous spaces ($G/H$), such as $\frac{U(4)}{U(2)\times U(2)}$, $\frac{U(5)}{U(2)\times U(3)}$, $U(4)/U(2)$, etc. If yes, could you give the references for the method? Can this be achieved by computer algebra? Thanks.
In general textbooks, there are only examples for special cases, such as $S^{n-1}=SO(n)/SO(n-1)$, and the method cannot be generalized for all homogeneous spaces.
There is a general theorem to the effect that often $H^*(G/H)$ is at least additively isomorphic to
$$Tor_{H^{\ast}(G)}(R,H^*(BH)),$$
regraded by total degree, where all cohomology is taken with coefficients in a commutative ring $R$. This holds if the cohomology of $BG$ is a polynomial algebra and $H$ is a compact Lie group such that the cohomology of $BH$ is a polynomial algebra on even degree generators. (Of course, if the characteristic of $R$ is not $2$, the generators of $H^*(BG)$ will also lie in even degrees).
$Tor$ is reasonably computable, once the map
$$H^{\ast}(BG) \longrightarrow H^{\ast}(BH)$$
has been computed, but that map or some avatar of it must be computed with any method.
The theorem does not even need groups, suitable $H$-spaces will do.
References (both posted on my web page)
V.K.A.M. Gugenheim and J.P. May. On the theory and applications of differential torsion products. Memoirs Amer. Math. Soc. No. 142, 1974.
F. Neumann and J.P. May. On the cohomology of generalized homogeneous spaces. Proc. Amer. Math. Soc. 130 (2002), 267-270.
The only general method I know of is the following (we assume $G$ and $H$ compact and connected).
Let $g$ and $h$ be the Lie algebras of $G$ and $H$ respectively.and let $A$ be a $G$-module. Recall that the standard cochain complex $C^*(g,A)$ is defined as follows: $C^n(g,A)=Hom(\Lambda^n g,M)$ with the differential of $a\in C^n(g,A)$ given by
$$da(x_1,\ldots, x_{n+1})=\sum_{1\leq k<l\leq n+1}(-1)^{k+l+1}a([x_k,x_l],x_1,\ldots,\hat x_k,\ldots,\hat x_l,\ldots,x_{n+1})$$ $$+\sum_{m=1}^{n+1}(-1)^m x_m\cdot a(x_1,\ldots,\hat x_m,\ldots, x_{n+1}).$$
The relative cochain complex $C^*(g,h,A)$ is the subcomplex of $C^*(g,A)$ formed by all $a$ such that when $x_1\in h$ both $a$ and $da$ kill any $(x_1,\ldots, x_n)$, resp. $(x_1,\ldots, x_{n+1})$.
The cohomology of $C^*(g,h,A)$ is isomorphic to the real cohomology of $G/H$ when $A$ is the trivial 1-dimensional module.
This is programmable, but perhaps not very illuminating. Much more can be said when $H$ contains a maximal torus. In that case $G/H$ decomposes into Schubert cells of even dimension, so the integral cohomology is torsion free; one can compute the ranks of the cohomology groups and also the cup product.
I think that the best way to compute this cohomology is the following. The homogeneous spaces you are looking at are all of the form $X=G/M$ where $G$ is compact and $M$ is a connected subgroup of $G$ Let $T_M$ be a maximal torus of $M$, embedded into a maximal torus $T_G$ of $G$ and let $\mathfrak t_M$ $\mathfrak t_G$ be the corresponding (complexified) Lie algebras. Let us first look at the equivariant cohomology $H^*_G(X)$ (say, with $\mathbb C$-coefficients). It is obvious that it is the same as $H^*_M(pt)$ (here $pt$ denote "the point") which is known to be $Sym(\mathfrak t_M^*)^{W_M}$; here $Sym$ means "symmetric algebra", and $W_M$ means the Weyl group of $M$. By abstract nonsense it is clear that $H^*(X)$
is just $H^*_G(X)\underset{H^*_G(pt)}\otimes {\mathbb C}$, where $\otimes$ in principle means "derived tensor product". If $M$ and $G$ have the same rank, then you can show that $H^*_G(X)$ is always free over $H^*_G(pt)=Sym(\mathfrak t^*)^{W_G}$ (here I denote $\mathfrak t=\mathfrak t_M=\mathfrak t_G$) hence you finally get
that $H^*(X)$
is equal to $Sym(\mathfrak t^*)^{W_M}\underset{Sym(\mathfrak t^*)^{W_G}}\otimes{\mathbb C}$.
In the general case, you need to compute the above derived tensor product, which in every specific case is usually easy to do.
If $G$ is a compact connected Lie group and $H$ is a closed connected subgroup then $H^\ast(G/H; \mathbb R)$ is isomorphic, as a graded algebra, to $H^\ast(\mathfrak g, \mathfrak h; \mathbb R)$ (relative Lie algebra cohomology). The latter is linear algebraically defined and therefore amenable to computation. For example, you can use Maple to compute the cohomology of various homogeneous spaces.
A search for "homogeneous space" and "(relative) Lie algebra cohomology" will turn up lots of hits on google.
