**Some thoughts on this antiderivative:**

Attacking $\log(\cos x)$ using integration by parts, we find:

$$\int \log(\cos x) = x\log(\cos x) + \int x \tan x dx$$

So the question has now become: how do we find an antiderivative for log(cos x)?

Next, we observe that

$$\cos x = \frac{1}{2}(e^{ix} + e^{-ix}) = \frac{1}{2}e^{ix}(1 + e^{-2ix})$$

Taking the log of this, we end up with:

$$-\log 2 + ix + \log(1 + e^{-2ix})$$

Recall that we can write

$$\log(1 + y) = \sum_{k = 1}^{\infty} \frac{(-1)^{k+1}y^{k}}{k}$$

We can now apply this with $y = e^{-2ix}$ as above and integrate term by term.

Putting all these pieces together will give you a (nasty) way to integrate $x\tan x$.

As far as showing it's not integrable in elementary functions, I suspect your best bet would be an appeal to a theorem of Liouville. See, for example, this link. (Sorry I can't be of more help here!)

All that said, perhaps you could ask your students some form of the following: show

$$\int x\tan^{2}x dx = x\tan x - \frac{x^2}{2} + \log(\cos x) + C$$

(You can find this latter, more tractable problem and its solution written out in nice detail here.)

rigorouslywhich functions have elementary antiderivatives? – Yemon Choi Oct 2 '12 at 4:54