So, the latest formulation is much better, but there are still some problems with this setup. You can push-forward cycles via morphisms, but rational maps are trickier.

First of all if the inverse of $f$ contracts a divisor, then $f_*$ does not (necessarily) respect linear equivalence. Here is an example. Let $X$ be your favorite smooth projective variety and $\pi:Y\to X$ the blow-up of a (closed) point $x\in X$. Let $f:X\dashrightarrow Y$ be the inverse of $\pi$ (as a rational map). Now let $\mathfrak d$ be a very ample linear system on $X$. If $D\in\mathfrak d$ is such that $x\not\in X$, then $f_*D\sim\pi^*D$ and if $D'\in\mathfrak d$ is such that $x\in D'$ and, for simplicity, the multiplicity of $D'$ at $x$ is $1$ (this happens for instance if $X=\mathbb P^n$ and $\mathfrak d$ is the hyperplane class), then $f_*D'=\pi^*D'-E \sim \pi^*D-E\sim f_*D-E$.

Therefore there is a short exact sequence of sheaves on $Y$: $$ 0\to \mathscr O_Y(f_*D')\to \mathscr O_Y(f_*D) \to \mathscr O_E \to 0. $$ Taking global sections one obtains another short exact sequence: $$ 0\to H^0(Y,\mathscr O_Y(f_*D'))\to H^0(Y,\mathscr O_Y(f_*D)) \to H^0(E,\mathscr O_E) \to 0. $$ Here the non-obvious exactness follows from the fact that $f_*D\cap E=\emptyset$ and hence the restriction of the corresponding section to $E$ is a non-zero global section of $\mathscr O_E$.

This shows that (using the OP's notation): $$ h^0(f_*D')= h^0(f_*D)-1. $$ On the other hand, from the projection formula it follows that $$ h^0(D)=h^0(f_*D) $$ and hence $$ h^0(D')=h^0(f_*D')+1 $$

OK, so we see that we better assume that the inverse of $f$ does not contract divisors. I assume you meant this to be part of the assumption that $f$ is a small modification, but you also seemed to be asking without that. Also, the above should warn you that using notation like $h^0(f_*D)$ is dangerous. The usage of $h^0$ suggests using the linear equivalence class of $D$, but $f_*D$ is not well-defined for that. Of course, this can still be done right, you just have to make sure to emphasize that $D$ is an explicit divisor, not a divisor class. (So for instance this is another reason why your original $D\in\mathrm{Pic}\\, X$ was bad).

Assuming that $f^{-1}$ does not contract divisors, there is of course a problem if $f$ contracts a divisor. Just take $D''=f_*D'$ from the above example. Clearly, $\pi_*D''=D'$ and we already saw that $$ h^0(D'')=h^0(\pi_*D'')-1. $$

So, we're left with the case when neither $f$ nor $f^{-1}$ contract any divisors. In this case it is indeed true what you want. Here is why:

The assumption means that in this case we have open sets $U\subseteq X$ and $V\subseteq Y$ such that $\mathrm{codim}_X(X\setminus U)\geq 2$, $\mathrm{codim}_Y(Y\setminus V)\geq 2$, and $f:U\to V$ is an isomorphism. In this case clearly we have that $$ H^0(X,\mathscr O_X(D)) = H^0(U,\mathscr O_U(D|_U)) = H^0(V,\mathscr O_V(f(D|_U))) = H^0(Y,\mathscr O_Y(f_*D)). $$ The middle equality is obvious the other two follows from the fact that $X$ and $Y$ are $S_2$. This is sometimes called the Hartog property. See this MO answer for more.

Note that this does not need $X$ and $Y$ to be smooth, only $S_2$. In order to deal with divisors you're probably better off assuming that they are normal. For some musings about that see this MO answer.

Also, to be fair, you asked for a relation in general, not equality so I assume you are aware of some of the above. I think that in general the relationship between $h^0(D)$ and $h^0(f_*D)$ will be very complicated and has to do with how $D$ relates to the exceptional divisor(s).

So, the latest formulation is much better, but there are still some problems with this setup. You can push-forward cycles via morphisms, but rational maps are trickier.

First of all if the inverse of $f$ contracts a divisor, then $f_*$ does not (necessarily) respect linear equivalence. Here is an example. Let $X$ be your favorite smooth projective variety and $\pi:Y\to X$ the blow-up of a (closed) point $x\in X$. Let $f:X\dashrightarrow Y$ be the inverse of $\pi$ (as a rational map). Now let $\mathfrak d$ be a very ample linear system on $X$. If $D\in\mathfrak d$ is such that $x\not\in X$, then $f_*D\sim\pi^*D$ and if $D'\in\mathfrak d$ is such that $x\in D'$ and, for simplicity, the multiplicity of $D'$ at $x$ is $1$ (this happens for instance if $X=\mathbb P^n$ and $\mathfrak d$ is the hyperplane class), then $f_*D'=\pi^*D'-E \sim \pi^*D-E\sim f_*D-E$.

Therefore there is a short exact sequence of sheaves on $Y$: $$ 0\to \mathscr O_Y(f_*D')\to \mathscr O_Y(f_*D) \to \mathscr O_E \to 0. $$ Taking global sections one obtains another short exact sequence: $$ 0\to H^0(Y,\mathscr O_Y(f_*D'))\to H^0(Y,\mathscr O_Y(f_*D)) \to H^0(E,\mathscr O_E) \to 0. $$ Here the non-obvious exactness follows from the fact that $f_*D\cap E=\emptyset$ and hence the restriction of the corresponding section to $E$ is a non-zero global section of $\mathscr O_E$.

This shows that (using the OP's notation): $$ h^0(f_*D')= h^0(f_*D)-1. $$ On the other hand, from the projection formula it follows that $$ h^0(D)=h^0(f_*D) $$ and hence $$ h^0(D')=h^0(f_*D')+1 $$

OK, so we see that we better assume that the inverse of $f$ does not contract divisors. I assume you meant this to be part of the assumption that $f$ is a small modification, but you also seemed to be asking without that. Also, the above should warn you that using notation like $h^0(f_*D)$ is dangerous. The usage of $h^0$ suggests using the linear equivalence class of $D$, but $f_*D$ is not well-defined for that. Of course, this can still be done right, you just have to make sure to emphasize that $D$ is an explicit divisor, not a divisor class. (So for instance this is another reason why your original $D\in\mathrm{Pic}\\, X$ was bad).

Assuming that $f^{-1}$ does not contract divisors, there is of course a problem if $f$ contracts a divisor. Just take $D''=f_*D'$ from the above example. Clearly, $\pi_*D''=D'$ and we already saw that $$ h^0(D'')=h^0(\pi_*D'')-1. $$

So, we're left with the case when neither $f$ nor $f^{-1}$ contract any divisors. In this case it is indeed true what you want. Here is why:

The assumption means that in this case we have open sets $U\subseteq X$ and $V\subseteq Y$ such that $\mathrm{codim}_X(X\setminus U)\geq 2$, $\mathrm{codim}_Y(Y\setminus V)\geq 2$, and $f:U\to V$ is an isomorphism. In this case clearly we have that $$ H^0(X,\mathscr O_X(D)) = H^0(U,\mathscr O_U(D|_U)) = H^0(V,\mathscr O_V(f(D|_U))) = H^0(Y,\mathscr O_Y(f_*D)). $$ The middle equality is obvious the other two follows from the fact that $X$ and $Y$ are $S_2$. This is sometimes called the Hartog property. See this MO answer for more.

Note that this does not need $X$ and $Y$ to be smooth, only $S_2$. In order to deal with divisors you're probably better off assuming that they are normal. For some musings about that see this MO answer.

Also, to be fair, you asked for a relation in general, not equality so I assume you are aware of some of the above. I think that in general the relationship between $h^0(D)$ and $h^0(f_*D)$ will be very complicated and has to do with how $D$ relates to the exceptional divisor(s).