I presume by "taking the derivative" you mean that you want to know how this expression $\Omega(X)=a^TM(X)a$, with $M(X)=(I+\alpha X^TA^TAX)^{-1}$, changes when you change $X$ by a small amount $\delta X$. So just make a series expansion,

$$\Omega(X+\delta X)=a^TM(X)\sum_{n=0}^\infty(-1)^n\{(\alpha \delta X^TA^TAX+\alpha X^TA^TA\delta X)M(X)\}^na^T$$

To first order you find

$$\Omega(X+\delta X)=a^TM(X)a-\alpha a^TM(X)(\delta X^TA^TAX+X^TA^TA\delta X)M(X)a$$

If you wish, you can restrict yourselves to scalar perturbations, $\delta X=\epsilon I$, then

$$\frac{d\Omega}{d\epsilon}=-\alpha a^TM(X)(A^TAX+X^TA^TA)M(X)a$$