Edit. Surely $f$ has at least two critical points if $M$ is closed, so we cannot have $|df|_g\ne 0$ everywhere in that case. This arguments fails if we assume that each component of $M$ has a nonempty boundary. To the contrary, now one can assume that $f$ is generic (after a $C^1$-small perturbation, which would not destroy $df(X)|_\gamma>0$ - this definitely works if $\gamma$ is proper, e.g., connects two points on $\partial M$) and hence has only finite many critical points. Because $\dim M=3>2$, we can connect these critical points with the boundary by a family of disjoint paths that do not intersect $\gamma$. Then using an isotopy of $M\cup_{\partial M}\mathrm\partial M\times[0,\varepsilon)$, we can pull out each critical point of $f$ and find the desired function if we started with a function satisfying $df(X)|_\gamma>0$.
Together with Ben's answer, this solves your problem when $\gamma$ is a proper embedding. I am not sure if that condition is necessary. However, whenever you have two sequences $s_i$, $t_i$ with distinct limit points in $\overline{\mathbb R}=\mathbb R\cup\{\pm\infty\}$, such that $\gamma(s_i)$ and $\gamma(t_i)$ converge to the same point in $M$, then such a function $f$ cannot exist.