Assume that $(H,\langle\cdot,\cdot\rangle)$ is a real separable Hilbert space equipped with a Gaussian measure $\mu$ with a mean $m$ and a covariance operator $C$. Let $x\in H$ be a fixed vector. What are the values of the following integrals: $$\int_H\frac{\langle x,h\rangle}{\h\}\mu(\mbox{d}h)$$ and $$\int_H\frac{\langle x,h\rangle^2}{\h\^2}\mu(\mbox{d}h),$$ can we express these integrals in terms of just $x$ and an inner product or norm?

4$\begingroup$ Do you know the result if $H$ is finite dimensional? $\endgroup$ – András Bátkai Jun 24 '16 at 18:22
First integral: no, this expression is not polynomial in $x$ due to the absolute value sign in the numerator, so it cannot be written as an inner product or norm of $x$. It certainly can be expressed in terms of an inner product (for instance, in the form you already have), and I argue below that any finite number of terms in its series expansion (in $C$) should be obtainable in closed form.
Second integral: this expression is a seminorm. Whether it is a norm depends on $C$. Evaluating it is also not easy to my knowledge, but should be easier than the first integral.
I cannot speak to the infinite dimensional case.
Your first integrand is not differentiable in $h$. I'd suggest integrating separately over the halfspaces separated by the plane $\langle x, h\rangle=0$. In each half space, I'd perform the Taylor expansion around the $m$, then carry out the Gaussian integrals over the dimensions orthogonal to $x$ via Isserlis's theorem. The final integration parallel to $x$ should leave you with a series of expressions in closed form in terms of incomplete Gamma functions.
My inner engineer notes that if $m$ is large compared to the rooteigenvalues of $C$, then
$$ \int_H\frac{\left\langle x,h\rangle\right}{\h\}\mu(\mbox{d}h)\approx \mathrm{sgn}(\langle x,m\rangle) \left\langle x, \int_H\frac{h}{\h\}\mu(\mbox{d}h) \right\rangle \\ = \mathrm{sgn}(\langle x,m\rangle) \left\langle x, \int_H\frac{m+\xi}{\m+\xi\}\nu(\mbox{d}\xi) \right\rangle, $$ where $\nu$ is a Gaussian measure over $H$ with mean $0$ and covariance operator $C$.
$$ = \mathrm{sgn}(\langle x,m\rangle) \left\langle x, \int_H(m+\xi)\left(\langle m,m\rangle + 2 \mbox{Re}\langle m,\xi\rangle + \langle \xi, \xi\rangle\right)^{1/2}\nu(\mbox{d}\xi) \right\rangle \\ \approx \frac{\mathrm{sgn}(\langle x,m\rangle)}{\m\} \left\langle x, \int_H(m+\xi)\left(1  \frac{\langle m,\xi\rangle}{\m\^2}  \frac12\frac{\langle \xi, \xi\rangle}{\m\^2}+\frac32\frac{\langle m,\xi\rangle^2}{\m\^4}\right)\nu(\mbox{d}\xi) \right\rangle \\ = \frac{\mathrm{sgn}(\langle x,m\rangle)}{\m\} \left[\langle x, m\rangle\left(1  \frac{\mbox{Tr}C}{2\m\^2} + \frac{3\langle m, C m\rangle}{2\m\^4}\right)  \frac{\langle x,Cm\rangle}{\m\^2}\right] \\ = \left\langle x, \hat m\rangle\right\left(1  \frac12\mbox{Tr}\hat C + \frac32\langle \hat m, \hat C \hat m\rangle\right)  \mathrm{sgn}(\langle x,\hat m\rangle) \langle x,\hat C\hat m\rangle, \\ \text{where }\hat C = C/\m\^2, \hat m = m/\m\. $$
For your second integral,
$$ \int_H \frac{\left\langle x,h\rangle\right^2}{\h\^2}\mu(\mathrm{d}h) = \left\langle x\right \left(\int_H \frac{\lefth\right\rangle\left\langle h\right}{\h\^2}\mu(\mathrm{d}h)\right) \leftx\right\rangle $$
The integral is quadratic in $x$. If $C$ is positive definite, then the operator in parentheses is a norm, since the integrand is a (positive semidefinite) orthogonal projector onto the subspace parallel to $h$, and the integration measure is nonzero over a fulldimensional region of $H$. In general, it looks like it is a seminorm, e.g. in the case where the kernel of $C$ includes vectors orthogonal to $m$.
The matrix elements of this operator could, in general, be evaluated by Taylor expanding around $m$, then using Isserlis' theorem to write the resulting Gaussian integrals over monomials in terms of traces of products of $C$ and $m$.
So I think the answer to your question is mostly "yes, in certain limiting cases".