Let $x, y\in R^n$ and $x, y$ are nonzero, it is well known $\frac{x^Ty}{\parallel x\parallel_2\parallel y\parallel_2}(\parallel x\parallel_2+\parallel y\parallel_2)\le \parallel x+y\parallel_2$. How to extend this to complex vectors? $arcos\frac{x^Ty}{\parallel x\parallel_2\parallel y\parallel_2} $ is the angle between $x$ and $y$. What is the appropriate definition of the angle between two complex vectors? I know $\frac{\mid x^*y\mid}{\parallel x\parallel_2\parallel y\parallel_2}(\parallel x\parallel_2+\parallel y\parallel_2)\le \parallel x+y\parallel_2$ does not hold generally.
Let $x,y$ be two nonzero complex vectors, let $\hat x=x/\x\$ and $\hat y=y/\y\$, and consider the parabola $$\phi(t)=\t\hat x+(1t)\hat y\^2=1+2(t^2t)(1\Re(\hat x \overline{\hat y})). $$ You easily check that $\phi(t)\ge\phi(1/2)$ for all $t$. This gives the inequality $$ \t\hat x+(1t)\hat y\\ge \sqrt{\frac{1+\Re(\hat x \overline{\hat y})}2} $$ for all real $t$. This is stronger than your inequality, which can be obtained by choosing $$t=\frac{\x\}{\x\+\y\}$$ at the left hand side, and noticing that $$ \sqrt{\frac{1+\sigma}2}\ge\sigma $$ for all real $\sigma\le1$ at the right hand side. So yes, the correct extension is using $\Re(x \cdot\overline{y})$ instead of $x\cdot y$. 


Let $V$ be a Euclidean vector space (in particular $V$ can be a Hermitean vector space considered as a real vector space). According to J.H.C. Whitehead (Manifolds with transversal fields in Euclidean space, Ann Math 73, 154212) the angle between vector subspaces $V_1$ and $V_2$ of $V$ can be defined as the Hausdorff distance (see e.g. http://en.wikipedia.org/wiki/Hausdorff_distance) between their intersections with the unit sphere. More explicitly, it is $$max(max_{x\in V_1\cap S_0}\angle(x,V_2),max_{x\in V_2\cap S_0}\angle(x,V_1))$$ where $S_0$ is the unit sphere and $\angle(x,W)$ is the angle between $x$ and its orthogonal projection to $W$ (if $x$ is orthogonal to $W$, the angle is set to be $\frac{\pi}{2}$). 


Probably use inner product Re(x* y), where x* is the conjugate transpose of x ... 


Actually, the case of complex vector spaces is rather a particular case than an extension, with respect to the case of real vector spaces. Recall that, as a vector space over $\mathbb{R}$, your $\mathbb{C}^n$ is isomorphic to $\mathbb{R}^{2n}$, and that, in terms of the Hermitian form of the former, the standard scalar product of the latter writes $\Re(x\cdot \bar y)$. Generally speaking, the appropriate definition of topological/uniform/metric notions for complex vector spaces is just the same for real vector spaces, seeing the complex vector spaces as real vector spaces by restriction of scalars. So the angle of vectors in $\mathbb{C}^n$ is just the angle in $\mathbb{R}^{2n}$. Sometimes, in the complex version, one also requires some kind of algebraic compatibility with the complex structure (e.g., the definition of a norm for complex VS). 


Please refer K. Scharnhorst, “Angles in complex vector spaces,” Acta Applicandae Math., vol. 69, pp. 95–103, Nov. 2001. and appendix in V. G. Reju, S. N. Koh and I. Y. Soon, “Underdetermined Convolutive Blind Source Separation via TimeFrequency Masking,” IEEE Transactions on Audio, Speech and Language Processing, Vol. 18, NO. 1, Jan. 2010, pp. 101–116. 

