The Schwartz kernel theorem seams relevant here. You might recall from your signal processing books that in the linear but non-time-invariant case we still get the output as a convolution of a kernel with the input, but a different kernel for each point in time. The kernel theorem makes this rigorous as I recall.
Once you have that theorem it is probably easy to get the statement you want. Indeed the kernel can hardly be time-varying if the system is to be time-invariant.