If $\mathcal{R}\subset J^rX$ is closed, then there's a smooth function $f:J^rX\to\mathbb R$ with $\mathcal{R}=f^{-1}(0)$. So you can construct a differential operator $H:J^kX\to M\times \mathbb{R}$ by $H(\theta):=(\pi_X(\pi^r_0(\theta)),f(\theta))$ and the equation $\mathcal{R}$ will be given by $H(j^r\phi)=0$.
So there is no big difference between the two definitions. If you are only interested in the "space" of solutions of the differential equation, then i'd say that the set $\mathcal{R}$ is enough, or put differently, you could choose the differential operator which suits you best to represent the equation.
Edit In response to Willies comment: Here's a counterexample to what you are asking for: recall that there's no submersion from $\mathbb{RP}^2$ to something, which has $\mathbb{RP}^1\subset \mathbb{RP}^2$ as a fiber. So take $M=\mathbb{R}$, $X=M\times \mathbb{RP}^2$ and $\mathcal{R}=M\times \mathbb{RP}^1\subset X$. Then there's no fiber bundle $Y\to M$ allowing a submersion $H:X\to Y$ with $\mathcal{R}=H^{-1}(\psi)$ for any $\psi\in \Gamma(Y)$. This is probably a silly example since the PDE is of order zero, but I'm sure one can come up with examples in higher order.
Anyway: if what you are interested in is an intrinsic notion of overdeterminedness of a PDE you might want to take a look at Bryant and Griffiths Characteristic Cohomology of Differential systems. Roughly the codimension of the characteristic variety serves as such a measure. And the characteristic variety can be defined completely without referring to an operator describing the equation. As Deane says, much of this can be found in the book Exterior differential systems. There are also the books by Vinogradov, Krasil'shchik and Lychagin.