I'm not sure what it is that you read in Hartshorne that suggested  that $(f_*\mathcal O_{\mathrm{Spec} B})_{\mathfrak p}$ is equal to $(\mathcal O_{\mathrm{Spec} B})_{\mathfrak q}$, since this is not true.

My suggestion is that you consider two illustrative cases: 

1. Let $A = k$ (a field) and $B = k\times k$, with $A \to B$ the diagonal morphism.  In this
case Spec $A$ is a single point, and so there is only stalk to consider.

2. Let $A = k[t]$ (again, $k$ is a field) and $B = k[t,t^{-1}]$, with $A \to B$ being the inclusion.  In this case, the map Spec $B \to $ Spec $A$ coicides with the identity
at all point of Spec $A$ other than the point $t = 0$, so the interesting case is the
stalk of the pushforward at $t = 0$ (this is a case with empty fibre).

In each case you can compute the stalk you asked about directly from the definition,
and I recommend that you try to do so.