It's deeper than theta cycles, I think.

I am going to assume that $N$ is prime to $p$ -- you don't say this in your question but most of my answer assumes this in a very serious way.

If $f$ is ordinary then the space of oldforms attached to $f$ at level $Np$ has dimension 2 and contains one ordinary and one non-ordinary form. Hida theory tells you that if there are no forms congruent to $f$ in weight $k$ and level $Np$ then the space of forms of weight $k+(p-1)$ and level $Np$ will also then contain one ordinary form congruent to $f$ (and only one). This form must be $p$-old, because newforms have big slope at big weight, and there's your explanation for the one form.

The non-ordinary case is deeper. Results of Fontaine/Edixhoven (see Edixhoven's Inventiones article on the weights in Serre's conjectures) tell you that the Galois representation attached to $f$ is irreducible even when restricted to a decomposition group at $p$ and furthermore on inertia it's $\omega_2^{k-1}$ plus its conjugate, with $\omega_2$ a niveau 2 character (see Serre's paper on Serre's conjecture, Duke 1987). The eigenform produced by "Deligne-Serre" (really just the going down theorem for finite flat $\mathbf{Z}_p$-algebras) has to have some slope. But the $p$-adic Langlands program is now sufficiently developed for $GL(2,\mathbf{Q}_p)$ that we can say a lot about this slope. Indeed, a theorem of Breuil [a convenient reference for this is Berger's recent Seminaire Bourbaki talk: http://perso.ens-lyon.fr/laurent.berger/articles/article17.pdf Theoreme 5.2.1 part 3] shows that in almost all cases (I guess $k$ has to be strictly greater than 2) the slope of the weight $k+p-1$ form is forced to be strictly between 0 and 1. But this means that the slope is not an integer and hence the Deligne-Serre form has at least one other Galois conjugate and often precisely one by the general principle that if you only look at a few examples, then things will probably only be as complicated as they have to be. The Galois conjugate will give rise to a mod $p$ representation isomorphic to a conjugate of the mod $p$ representation attached to $f$, but your assumption about the coefficient field being the rationals mean that the mod $p$ representation attached to $f$ is defined over $GL(2,\mathbf{F}_p)$ and hence hitting with Galois doesn't move anything. Hence one is forced to get at least two, and probably in many cases exactly two, non-ordinary forms giving rise to the Galois representation in weight $k+p-1$.