I'm not sure what it is that you read in Hartshorne that suggested that $(f_*\mathcal O_{\mathrm{Spec} B})_{\mathfrak p}$ is equal to $(\mathcal O_{\mathrm{Spec} B})_{\mathfrak q}$, since this is not true.

My suggestion is that you consider two illustrative cases:

1. Let $A = k$ (a field) and $B = k\times k$, with $A \to B$ the diagonal morphism. In this case Spec $A$ is a single point, and so there is only stalk to consider.

2. Let $A = k[t]$ (again, $k$ is a field) and $B = k[t,t^{-1}]$, with $A \to B$ being the inclusion. In this case, the map Spec $B \to $ Spec $A$ coicides with the identity at all point of Spec $A$ other than the point $t = 0$, so the interesting case is the stalk of the pushforward at $t = 0$ (this is a case with empty fibre).

In each case you can compute the stalk you asked about directly from the definition, and I recommend that you try to do so.

I'm not sure what it is that you read in Hartshorne that suggested that $(f_*\mathcal O_{\mathrm{Spec} B})_{\mathfrak p}$ is equal to $(\mathcal O_{\mathrm{Spec} B})_{\mathfrak q}$, since this is not true.

My suggestion is that you consider two illustrative cases:

1. Let $A = k$ (a field) and $B = k\times k$, with $A \to B$ the diagonal morphism. In this case Spec $A$ is a single point, and so there is only stalk to consider.

2. Let $A = k[t]$ (again, $k$ is a field) and $B = k[t,t^{-1}]$, with $A \to B$ being the inclusion. In this case, the map Spec $B \to $ Spec $A$ coicides with the identity at all point of Spec $A$ other than the point $t = 0$, so the interesting case is the stalk of the pushforward at $t = 0$ (this is a case with empty fibre).

In each case you can compute the stalk you asked about directly from the definition, and I recommend that you try to do so.

Added: If $f: X \to Y$ and $\mathcal F$ is a sheaf on $X$, then for any $x \in X$ there is a canonical map of stalks $(f_*\mathcal F)_{f(x)} \to \mathcal F_x,$ given as follows: if $V$ is a n.h. of $f(x)$, then $f^{-1}(V)$ is a n.h. of $x$, and by definition $f_*\mathcal F(V) = \mathcal F(f^{-1}(V)).$ If $V$ runs over all n.h.s of $f(x)$, then $f^{-1}(V)$ will range over some (but typically not all) n.h.s of $x$, and so there will be an induced map $(f_*\mathcal F)_{f(x)} \to \mathcal F_x$, but this will typically not be an isomorphism (exactly because $f^{-1}(V)$ typically doesn't range over all n.h.s of $x$, but just certain ones). In the case of a morphism $f:X \to Y$ of ringed spaces, the given map $\mathcal O_Y \to f_*\mathcal O_X$ then induces maps of stalks $(\mathcal O_Y)_{f(x)} \to (f_*\mathcal O_X)_{f(x)}$ (by functoriality of the construction of stalks) and $(f_*\mathcal O_X)_{f(x)} \to (\mathcal O_X)_x$ (via the above construction). Their composite is the morphism $(\mathcal O_Y)_{f(x)} \to (\mathcal O_X)_x$ that Hartshorne uses when he makes the definition of a morphism of locally ringed spaces.