Control a linear system to the kernal space of the output matrix. Consider the following controllable and observable linear system
             $$\dot x=Ax+Bu, y=Cx,$$
where $x\in \mathbb{R}^n, u\in \mathbb{R}^m, y\in \mathbb{R}^p$.
The observability of $(A,C)$ tells us that Cx(t)=0 implies x(t)=0 for any trajectory of the system. Considering this, I have the following questions:


*

*Why not design a controller to steer the state to $\{x|Cx=0\}$ instead of $\{0\}$?

*If this idea is ok, how to design such kind of controllers?


I posted it here and hope to get some answers or references.
 A: In linear geometric control theory, the problem you posed is called the output stabilization problem (OSP). See Wonham's book on linear geometric control.
The approach to solving OSP, conceptually, is as follows.
Consider the LTI system: $\dot x = A x + B u, y=Cx$. We wish to design a state feedback such that for all initial conditions, the solution $x(t)$ has the property that $y(t) = C x(t) \to 0$ as $t \to \infty$.
Intuition: The idea is to try to asymptotically stabilize the subspace $\ker C$, rather than the equilibrium $x=0$. However, one can show that a necessary condition for a subspace to be asymptotically stable for the closed-loop system is that the subspace be an invariant set for the closed-loop system. So we can only hope to stabilize a subspace of $\ker C$ which can be made invariant via feedback. In fact, the best thing we can do is to stabilize the largest subspace of $\ker C$ which can be made invariant via feedback. In geometric control theory, this is called the maximal $(A,B)$-invariant subspace contained in $\ker C$. In nonlinear circles, this is called the zero dynamics manifold.
Solution:
1) Find the maximal $(A,B)$-invariant subspace contained in $\ker C$. Let's call it $\cal V$.
2) By definition of $(A,B)$-invariant subspace, there exists a gain $K$ such that $\cal V$ is $A+BK$-invariant, i.e., $(A+BK)\cal V \subset \cal V$.
3) Set $u= Kx+v$. We obtain a new control system $\dot x = \hat A x + Bv, y=Cx$, where $\hat A :=A+BK$ and, by construction, $\ker C$ is $\hat A$-invariant.
4) Decompose the system with respect to the invariant subspace: find a coordinate transformation $(z_1,z_2) = T x$ such that 
(a) In $(z_1,z_2)$-coordinates the subspace $\cal V$ is the $z_1$ plane, $\{(z_1,z_2):z_2=0\}$.
(b) The matrix $T \hat A T^{-1}$ has the upper triangular structure
$$
T \hat A T^{-1} = \left[\matrix{A_{11} & A_{12} \\ 0  & A_{22}}\right].
$$
5) Then, letting $\operatorname{col}(B_1,B_2) = TB$, we have the control system
$$
\matrix{\dot z_1 = A_{11} z_1 + A_{12} z_2+ B_1 v \\ \dot z_2 = A_{22} z_2 + B_2 v}$$
and the control objective is to stabilize the subspace $\{z2=0\}$, i.e., the equilibrium of the $z_2$-subsystem. This is a familiar equilibrium stabilization problem. If the pair $(A_{22},B_2)$ is stabilisable, then there exists a feedback $v = K_2 z_2$ that stabilizes the subspace $\{z_2=0\}$, which in original coordinates corresponds to stabilizing $\cal V$.
6) The final feedback solving OSP is 
$$
u=Kx + [\matrix{0 & K_2}] T^{-1}x.$$
Comments:
The steps provided above are conceptual. The actual computations you have to conduct to perform each step are standard and you can find them in Wonham' book. A simpler presentation is given in "Control Theory for linear systems" by Trentelman and others.
To check whether the pair $(A_{22},B_2)$ is, say, controllable, you don't need to perform the steps above. You check it directly using the following 
Theorem: The pair $(A_{22},B_2)$ in step 5 is controllable if and only if
$$
{\cal V} + \operatorname{image}([\matrix{B & AB & \cdots & A^{n-1} B}]) = \Re^n. 
$$
An analogous result for stabilizability is given in Wonham.
