I am not aware of such results in full generality, but I know that working without the properness assumtpion was in part the main motivation for Grothendieck to write SGA 2.

Let me focus on a related question (but not exactly the same). Let $\pi : X \rightarrow Y$ be a flat morphism of finite type with $Y$ a smooth variety over a field. Given a coherent sheaf $F$ on $X$ flay over $Y$, you would like to know if $h^i(X_t, F_t)$ might be upper semi-continous in some cases.

Essentially, what you have to answer are the following two questions:

i) Is there a base change isomorphism $R^i \pi_* F \otimes \mathcal{O}_t = H^i(X_t, F_t)$?

ii) Is the sheaf $R^i \pi_* F$ coherent?

If the answer to this two questions is yes, then the semi-continuity of $h^i(X_t, F_t)$ is obvious (this becomes a linear algebra question).

If you assume that $Y$ is affine, I think this is an easy result to prove that base change formula holds true : see Hartshorne III corollary 9.4.

As Jason Starr mentionned, the well-known result is that the answer to ii) is yes, if you assume $\pi$ proper.

If you want to drop the properness assumption, you have to add some other hypotheses for the coherence of $R^i \pi_* F$ to hold. (though I don't have a counter-example, I am pretty sure the cohrence does not hold without any assumption). In fact, many interesting results in SGA 2 address the coherence issue if you withdraw the properness hypothesis.

The price you have to pay is to add a depth hypothesis. In fact you will "compactify" $\pi$ from $\tilde{X}$ to $Y$. But you don't assume that $F$ comes from a coherent sheaf on $\tilde{X}$. What you want to know is when the sheaf $R^i j_* F$ is coherent (where $j : X \rightarrow \tilde{X}$ is the open immersion).

You have to make some assumptions on the depth of $F_x$ for $x \in X$.

Corollary $2.3$ and Theorem $3.1$ of expose $VIII$ in SGA 2 tell you what depth hypothesis you have to add to get a positive answer to question ii).

I am not aware of such results in full generality, but I know that working without the properness assumtpion was in part the main motivation for Grothendieck to write SGA 2.

Let me focus on a related question (but not exactly the same). Let $\pi : X \rightarrow Y$ be a flat morphism of finite type with $Y$ a smooth variety over a field. Given a coherent sheaf $F$ on $X$ flay over $Y$, you would like to know if $h^i(X_t, F_t)$ might be upper semi-continous in some cases.

Essentially, what you have to answer are the following question:

Is the sheaf $R^i \pi_* F$ coherent?

If the answer to this question is yes, then I believe the semi-continuity of $h^i(X_t, F_t)$ is true under the flatness hypothesis (this becomes a linear algebra question if I remember correctly).

As Jason Starr mentionned, the well-known result is that the answer to this question is yes, if you assume $\pi$ proper.

If you want to drop the properness assumption, you have to add some other hypotheses for the coherence of $R^i \pi_* F$ to hold. (though I don't have a counter-example, I am pretty sure the cohrence does not hold without any assumption). In fact, many interesting results in SGA 2 address the coherence issue if you withdraw the properness hypothesis.

The price you have to pay is to add a depth hypothesis. In fact you will "compactify" $\pi$ from $\tilde{X}$ to $Y$. But you don't assume that $F$ comes from a coherent sheaf on $\tilde{X}$. What you want to know is when the sheaf $R^i j_* F$ is coherent (where $j : X \rightarrow \tilde{X}$ is the open immersion).

You have to make some assumptions on the depth of $F_x$ for $x \in X$.

Corollary $2.3$ and Theorem $3.1$ of expose $VIII$ in SGA 2 tell you what depth hypothesis you have to add to get some coherence results