There are several different definitions of relative entropy, and some of them are not equivalent. Following is the definition we will use in this question.

Let $M$ be a closed manifold and $\mathcal{P}$ the set of Borel probability measures on $M$. Given a reference measure $\omega\in \mathcal{P}$ (usually the normalized Lebsgue measure), the relative entropy of a measure $\mu\ll\omega$ is defined as:

$$E(\mu|\omega)=\int_M\log\phi_\mu \ d\mu,$$

where $\displaystyle \phi_\mu=\frac{d\mu}{d\omega}$ is the Radon-Nykodim derivative (since we assume $\mu\ll\omega$). The integration is well defined since $\mu(\lbrace\phi_\mu=0\rbrace)=0$. For example $E(\mu|\omega)\ge0$ since

\begin{align*} E(\mu|\omega)&=\int\log\phi_\mu d\mu=\int_{\phi_\mu>0}-\log\frac{d\omega}{d\mu} d\mu \\\&\ge-\log\int_{\phi_\mu>0}\frac{d\omega}{d\mu} d\mu =-\log\omega(\lbrace\phi_\mu=0\rbrace)\ge0. \end{align*}

In particular $E(\mu|\omega)=0$ implies $\mu=\omega$.

Moreover, this function is convex: for all $\mu,\nu\ll\omega$, $$E(p\mu+q\nu|\omega)\le p\cdot E(\mu|\omega)+q\cdot E(\nu|\omega).$$

As noticed by Pablo (thank you!), the above claim is indeed a direct corollary of the convexity of $h(x)=x\log x$.

A more interesting statement I want to know if that, if $\mu_n\ll\omega$ such that $\mu_n\to\mu\not$$\ll\omega$, will we always have $E(\mu_n|\omega)\to+\infty$? Thank you!

Here $\mu_n\to\mu$ in the sense that $\mu_n(f)\to\mu(f)$ for all continuous functions $f$.

The paper given by Ashok below provides an equivalent definition: $\displaystyle E(\mu|\omega)=\sup_{\alpha}\sum_{A\in\alpha}\mu(A)\log\frac{\mu(A)}{\omega(A)}$, where the supremum is taken over all finite, Borel partitions $\alpha$ with $\omega(A)>0$.

In particular if $\mu\not$$\ll\omega$, we can take open sets $A_k$ with $\mu(A_k)\ge2\delta$ and $\omega(A_k)\to 0$. So $E(\mu|\omega)=+\infty$.

Now let's make a better choice of $A_k$'s such that $\mu(\partial A_k)=0$. Then if $\mu_n\to\mu$, $\mu_n(A_k)\to\mu(A_k)$ for all $k$. Hence we can pick $n_k$ such that $\mu_{n}(A_k)\ge\delta$ for all $n\ge n_k$. So for all $n\ge n_k$, we have $$E(\mu_{n}|\omega)\ge\mu_{n}(A_k)\log\frac{\mu_{n}(A_k)}{\omega(A_k)} \ge\delta\cdot\log\frac{\delta}{\omega(A_k)}\to\infty.$$