In linear geometric control theory, the problem you posed is called the *output stabilization problem (OSP)*. See Wonham's book on linear geometric control.

The approach to solving OSP, conceptually, is as follows.

Consider the LTI system: $\dot x = A x + B u, y=Cx$. We wish to design a state feedback such that for all initial conditions, the solution $x(t)$ has the property that $y(t) = C x(t) \to 0$ as $t \to \infty$.

**Intuition:** The idea is to try to asymptotically stabilize the subspace $\ker C$, rather than the equilibrium $x=0$. However, one can show that a necessary condition for a subspace to be asymptotically stable for the closed-loop system is that the subspace be an invariant set for the closed-loop system. So we can only hope to stabilize a subspace of $\ker C$ which can be made invariant via feedback. In fact, the best thing we can do is to stabilize the *largest* subspace of $\ker C$ which can be made invariant via feedback. In geometric control theory, this is called the maximal $(A,B)$-invariant subspace contained in $\ker C$. In nonlinear circles, this is called the *zero dynamics manifold*.

**Solution**:

1) Find the maximal $(A,B)$-invariant subspace contained in $\ker C$. Let's call it $\cal V$.

2) By definition of $(A,B)$-invariant subspace, there exists a gain $K$ such that $\cal V$ is $A+BK$-invariant, i.e., $(A+BK)\cal V \subset \cal V$.

3) Set $u= Kx+v$. We obtain a new control system $\dot x = \hat A x + Bv, y=Cx$, where $\hat A :=A+BK$ and, by construction, $\ker C$ is $\hat A$-invariant.

4) Decompose the system with respect to the invariant subspace: find a coordinate transformation $(z_1,z_2) = T x$ such that

(a) In $(z_1,z_2)$-coordinates the subspace $\cal V$ is the $z_1$ plane, $\{(z_1,z_2):z_2=0\}$.

(b) The matrix $T \hat A T^{-1}$ has the upper triangular structure
$$
T \hat A T^{-1} = \left[\matrix{A_{11} & A_{12} \\ 0 & A_{22}}\right].
$$

5) Then, letting $\operatorname{col}(B_1,B_2) = TB$, we have the control system
$$
\matrix{\dot z_1 = A_{11} z_1 + A_{12} z_2+ B_1 v \\ \dot z_2 = A_{22} z_2 + B_2 v}$$

and the control objective is to stabilize the subspace $\{z2=0\}$, i.e., the equilibrium of the $z_2$-subsystem. This is a familiar equilibrium stabilization problem. If the pair $(A_{22},B_2)$ is stabilisable, then there exists a feedback $v = K_2 z_2$ that stabilizes the subspace $\{z_2=0\}$, which in original coordinates corresponds to stabilizing $\cal V$.

6) The final feedback solving OSP is
$$
u=Kx + [\matrix{0 & K_2}] T^{-1}x.$$

**Comments:**

The steps provided above are conceptual. The actual computations you have to conduct to perform each step are standard and you can find them in Wonham' book. A simpler presentation is given in "Control Theory for linear systems" by Trentelman and others.

To check whether the pair $(A_{22},B_2)$ is, say, controllable, you don't need to perform the steps above. You check it directly using the following

**Theorem:** The pair $(A_{22},B_2)$ in step 5 is controllable if and only if

$$
{\cal V} + \operatorname{image}([\matrix{B & AB & \cdots & A^{n-1} B}]) = \Re^n.
$$

An analogous result for stabilizability is given in Wonham.

notimply "$Cx = 0 \;\Rightarrow \; x = 0$". – gerw Apr 17 '13 at 19:08