I may be missing an obvious example, but here goes...

Let $X$ denote a complex manifold of dimension $n$. If $X$ is Kähler, then the induced metric on any complex submanifold is also Kähler. If instead $X$ is non-Kähler, we can at least say that any coordinate neighborhood $(U,\varphi)$ inherits a Kähler metric from the induced Euclidean metric on an open subset of $\mathbb{C}^n$. If $X$ is already equipped with a metric, note that this metric on $U$ may not be compatible with it. However, I am unsure of what can be said about the metric structure of $U$ if instead we allow something a little more exotic. For example, what if we take $U$ to be the complement of an analytic set? This is the situation in which I am most interested.

My question is:

Is there an example of a non-Kähler manifold, $X$, and a Zariski open subset $U\subset X$, such that $U$ admits a Kähler metric?

Added: Francesco has provided a class of examples in dimensions $\geq 3$ and BS has provided an example in dimension 2 as a comment to Benoît's answer. I am going to write up BS's example, with a few minor additions, as an answer. In case he decides to write his own answer, I'll delete mine and encourage everyone to vote his up.