I am not aware of such results in full generality, but I know that working without the properness assumtpion was in part the main motivation for Grothendieck to write SGA 2.
Let me focus on a related question (but not exactly the same). Let $\pi : X \rightarrow Y$ be a flat morphism of finite type with $Y$ a smooth variety over a field. Given a coherent sheaf $F$ on $X$ flay over $Y$, you would like to know if $h^i(X_t, F_t)$ might be upper semi-continous in some cases.
Essentially, what you have to answer are the following question:
Is the sheaf $R^i \pi_* F$ coherent?
If the answer to this question is yes, then I believe the semi-continuity of $h^i(X_t, F_t)$ is true under the flatness hypothesis (this becomes a linear algebra question if I remember correctly).
As Jason Starr mentionned, the well-known result is that the answer to this question is yes, if you assume $\pi$ proper.
If you want to drop the properness assumption, you have to add some other hypotheses for the coherence of $R^i \pi_* F$ to hold. (though I don't have a counter-example, I am pretty sure the cohrence does not hold without any assumption). In fact, many interesting results in SGA 2 address the coherence issue if you withdraw the properness hypothesis.
The price you have to pay is to add a depth hypothesis. In fact you will "compactify" $\pi$ from $\tilde{X}$ to $Y$. But you don't assume that $F$ comes from a coherent sheaf on $\tilde{X}$. What you want to know is when the sheaf $R^i j_* F$ is coherent (where $j : X \rightarrow \tilde{X}$ is the open immersion).
You have to make some assumptions on the depth of $F_x$ for $x \in X$.
Corollary $2.3$ and Theorem $3.1$ of expose $VIII$ in SGA 2 tell you what depth hypothesis you have to add to get some coherence results