Do there exist positive integers $m$, $n$, $p$, $q$ such that $m>1$, $p\neq q$, $p$ and $q$ divide $mn^2 - 1$, and $mn$ divides $p - q$?
It seems numerically up to $n \leq 10^6$ that for $m=3$ or $m=4$, such a triplet $(p,q,n)$ satisfying the conditions does not exist. The case of $m=2$ was answered negatively at MathSE.
I will show below that there are no solutions. I will use the ideas of Denis Shatrov from the MSE page along with Keith Conrad's notes about the Pell equation. I will also use some ideas from comments by Thomas Browning. (Meanwhile, Thomas Browning found a more direct proof.)
Let $(m,n,p,q)$ be a tuple satisfying the conditions. Without loss of generality, $p$ and $q$ are coprime, and $p<q$. Hence $q=p+mnt$ for some integer $t>0$. Now $pq$ divides $mn^2-1$, hence for some integer $s>0$ we have $$mn^2-1=sp(p+mnt).$$ Equivalently, $$mn^2-s p^2=1+mnpst.\tag{1}$$ In particular, $mn^2>mnpst$, whence $n>pst$. Now we consider the positive integers $$d:=m(ms^2t^2+4s),\qquad r:=m(2n-pst),\qquad u:=mst^2+2.\tag{2}$$ It is straightforward that $d\neq\square$ and $d\equiv 0,1\pmod{4}$, hence $d$ is a positive discriminant. Let us denote by $\mathcal{O}$ the ring of integers of the real quadratic field $\mathbb{Q}(\sqrt{d})$. Observing that $$r^2-dp^2=4m\qquad\text{and}\qquad u^2-dt^2=4,$$ we infer that $\xi:=(r+p\sqrt{d})/2$ and $\epsilon:=(u+t\sqrt{d})/2$ are totally positive elements of $\mathcal{O}$ of norms $m$ and $1$, respectively. Multiplying $\xi$ by a suitable integral power of $\epsilon$, we obtain a totally positive element $(x+y\sqrt{d})/2\in\mathcal{O}$ of norm $m$ such that $$\sqrt{m/\epsilon}\leq\frac{x-y\sqrt{d}}{2}\leq\frac{x+y\sqrt{d}}{2}\leq\sqrt{m\epsilon}.$$ It follows that $$0\leq y<\sqrt{\frac{m\epsilon}{d}}<\sqrt{\frac{mu}{d}}=\sqrt{\frac{mst^2+2}{ms^2t^2+4s}}<1.$$ As $y$ is an integer, it must be zero, and hence $m=\square$. Denoting $k:=\sqrt{m}$, which is an integer exceeding $1$, the original equation $(\ast)$ becomes $$(kn)^2-s p^2=1+(kn)ps(kt).$$ This is the same equation as $(1)$, but with $(1,kn,kt)$ in place of $(m,n,t)$. So we are left with showing that $(1)$ has no positive integer solution $(m,n,p,s,t)$ with $m=1$ and $\gcd(n,t)>1$.
Let $(m,n,p,s,t)$ be a positive integer solution of $(1)$ with $m=1$. We need to show that $\gcd(n,t)=1$. Using the notations of the previous paragraph, we conclude that $(x+y\sqrt{d})/2=1$. That is, $\xi$ is an integral power of $\epsilon$. In other words, either $(r+p\sqrt{d})/2$ or $(r-p\sqrt{d})/2$ can be written as $\epsilon^j$ with an integer $j\geq 0$. Note that $r>2$ by $n>pst$, whence $j>0$. In particular, by the binomial theorem, $$2^{j-1}r=\sum_{i=0}^{\lfloor j/2\rfloor}\binom{j}{2i}u^{j-2i}t^{2i}d^i.\tag{3}$$ Assume first that $d\equiv 1\pmod{4}$. Then $t$ and $u$ are odd by $(2)$, hence it follows from $(2)$ and $(3)$ that $$\gcd(n,t)=\gcd(2n,t)=\gcd(r,t)=\gcd(2^{j-1}r,t)=\gcd(u^j,t)=\gcd(2^j,t)=1.$$ Assume now that $d\equiv 0\pmod{4}$. Then $st$ and $r$ and $u$ are even by $(2)$. We claim that $ps$ is also even. Indeed, if $s$ is odd, then $t$ is even, hence $\epsilon\in\mathbb{Z}[\sqrt{d}]$, and therefore $\epsilon^j\in\mathbb{Z}[\sqrt{d}]$, and $p$ is even. Hence both $r$ and $ps$ are even, and then it follows from $(2)$ and $(3)$ that $$\gcd(n,t)=\gcd(r/2,t)=\gcd((u/2)^j,t)=1.$$ So in all cases we have that $\gcd(n,t)=1$, and the proof is complete.
I'll first give a quick summary of this solution, and then fill in the details of each step below.
In fact, the arc of the hyperbola between $(a^{-1/2},-a^{1/2}b)$ and $(a^{-1/2},0)$ is a fundamental domain for the action of $T$ on the $x>0$ branch of the hyperbola, but we will not need to prove this.
By dividing $p$ and $q$ by their greatest common divisor, we may assume that $p$ and $q$ are coprime. We may also assume that $p>q$. Now write $p=q+kmn$ and $mn^2-1=pqr=(q+kmn)qr$ for positive integers $k$ and $r$. Then we must show that the equation $$mn^2-kmnqr-q^2r=1$$ has no positive integer solutions with $m>1$. After relabeling variables, it suffices to show that if $a>1$, $b$, and $c$ are positive integers, then there are no integral points on the $x>0$ branch of the hyperbola $$ax^2-abcxy-cy^2=1.$$ Of course, there will also be no integral points on the $x<0$ branch since the two branches are related by $(x,y)\leftrightarrow(-x,-y)$.
The quadratic form $ax^2-abcxy-cy^2$ is invariant under the linear transformation $$T:(x,y)\mapsto((ab^2c+1)x+bcy,abx+y)$$ in $SL(2,\mathbb{Z})$. In particular, $T$ acts on the hyperbola, and maps integral points to integral points. To check whether $T$ swaps the two branches or not, we can observe that $T(a^{-1/2},-a^{1/2}b)=(a^{-1/2},0)$. So by continuity, $T$ acts on the $x>0$ branch of the hyperbola.
Let $x_{min}=\sqrt{4c/(a^2b^2c^2+4ac)}$ be the minimal $x$-coordinate on the $x>0$ branch of the hyperbola. Then the action of $T$ on the $x>0$ branch of the hyperbola will always increase the $y$-coordinate by at least $abx_{min}$. In particular, every point on the $x>0$ branch of the hyperbola can be mapped to a point satisfying $y\leq0$ and $abx+y\geq0$. But then $ax^2-1=cy(abx+y)\leq0$, so $x\leq a^{-1/2}$.
If $a>1$, then there are no integral points satisfying $0<x\leq a^{-1/2}$, so there are no integral points on the $x>0$ branch of the hyperbola.
In contrast to the two great answers already given, I thought it would be interesting to try an approach using Pell's equation and continued fractions only.
Before I do so I'd like to note that Theorem 2 of the paper Stolt, Bengt, On a Diophantine equation of the second degree, Ark. Mat. 3, 381-390 (1957). ZBL0077.05002. completely solves the problem and actually proves the stronger condition that the minimum positive value of the form over the integers is $a$.
I found this in the paper Matthews, Keith R.; Robertson, John P.; Srinivasan, Anitha, On fundamental solutions of binary quadratic form equations, Acta Arith. 169, No. 3, 291-299 (2015). ZBL1418.11050. which refines Stolt's bounds to give a complete characterisation of the fundamental solutions.
Very Briefly the theorem states that if $(u,v)$ is a fundamental solution to $Au^2 + Buv + cv^2= -N$ with $A$ and $N$ positive integers and $D=B^2-4AC$ a positive integer not a perfect square then if $(x_1,y_1)$ is a fundamental solution to $x_1^2-Dy_1^2=4$ we have the inequality $$0<v\leq \sqrt{\frac{AN(x_1+2)}{D}}.$$ To apply this to the equation $ax^2-abcxy-cy^2=N$ we multiply by $-1$ and exchange $x$ and $y$ to obtain $cx^2+abcxy-ay^2=-N$. Note also that $x=2+ab^2c$, $y=b$ is a fundamental solution to $x^2-Dy^2=4$ where in this case $D=(abc)^2+4ac$.
For a fundamental solution $x=u$, $y=v$ the inequality then becomes $$0<v\leq \sqrt{\frac{cN(x+2)}{D}}=\sqrt{\frac{cN(4+ab^2c)}{(abc)^2+4ac}}=\sqrt{\frac{N}{a}}$$ This implies that $N\geq a$.
My own solution:
As in Thomas Browning's answer, we will show that if $a>1$, $b$, and $c$ are positive integers, then $$ax^2-abcxy-cy^2=1\tag{1}$$ has no integer solutions.
The plan is to show that integer solutions to (1) give solutions to Pell's equation such as ${x'}^2-Dy^2=4a$. For most cases this gives convergents to the continued fraction for $\sqrt{D}$ where $D=a^2b^2c^2+4ac$. The convergents $x/y$ can explicitly computed, and hence the possible values of $x^2-Dy^2$. Comparing these values with the RHS of the Pell Equation gives contradictions except for 3 exceptional equations which are handled via congruence obstructions.
We first note that that $(a,c)=1$ is a necessary condition for integer solutions to exist.
From (1) we obtain by multiplying by 4a and -4c respectively and completing the square: $${x'}^2-Dy^2=4a\tag{2}$$. $${y'}^2-Dx^2=-4c\tag{3}$$.
Where $x'=2ax-abcy$ and $y'=2cy+abcx$.
Now for an odd prime $p$, $p|y$ and $p|x'$ implies $p|a$ from (2) and then from (1) $p|1$, so $(y,x')$ is a power of $2$. Also if $2|y$ then from (1) $ax^2$ must be odd and hence $ax$ also. Thus if $4|y$ then $x'/2=ax-abcy/2$ is odd. Thus
$$(y,x')=1 \textrm{ or } 2.$$
Similarly using (3) we can show that $$(x,y')=1 \textrm{ or } 2.$$
Theorem 5.1 from Keith Conrad's notes states that if positive integers $x$ and $y$ satisfy $x^2 − Dy^2 = n$ with $|n| <\sqrt{D}$ then $x/y$ is a convergent to the continued fraction of √D.
Also the particular form of $D$ actually allow us to compute the continued fraction for $\sqrt{ D}$ explicitly and in addition, the possible values of $X^2-DY^2$ when $X/Y$ is a convergent. These are $S=\{1, 4, -4ac, -ac, 1 - ac - abc, 1 - ac + abc\}$. See lemma 1 below for details.
If $(y',x)=2$ then $X^2-DY^2=-c$, has an integer solution given by $X=y'/2$ and $Y=x/2$ with $(X,Y)=1$.
In this case, $a>1$ implies $|-c|\leq|abc|<\sqrt{D}$ and therefore $X/Y$ is a convergent to $\sqrt{D}$. Since $(X,Y)=1$ this is a contradiction since, if $a>1$, $-c$ is not in the list of possible values $S$ of $x^2-Dy^2$ where $x/y$ is a convergent to $\sqrt{D}$.
Similarly if $(x',y)=2$ then $X^2-DY^2=a$, has an integer solution given by $Y=y/2$ and $X=x'/2$ with $(X,Y)=1$. $|a|\leq |abc|<\sqrt{D}$ thus $a$ must lie in $S$ and this is only possible if $a=4$, again assuming $a>1$.
We now assume that $(y',x)=(y,x')=1$. It is possible to show that $|4a|<\sqrt{D}$ and $|-4c|<\sqrt{D}$ holds for all positive $a,b,c$ if $a>1$ and $(a,c)=1$, except for those $(a,b,c)\in\{(2,1,1),(2,1,3),(3,1,2)\}$. (See Lemma 2 below for details). Aside from these equations, we can then argue as before that if $a>1$ the only possibility for $4a$ or $-4c$ to be in $S$ is when $a=4$.
Now consider the case, $a=4$. For equation (3) we have already shown that $(x,y')=2$ leads to a contradiction. If $a=4$ then $|-4c|\leq|4bc|=|abc|<\sqrt{D}$. Now $(x,y')=1$ and $-4c$ only lies in $S$ if $-4c=-ac\in S$ but since $abc=4bc$ is even we can exclude the value $-ac$ from $S$ as this is absent if $abc$ is even. (see Lemma 1 below)
Finally we deal with the remaining equations: $2x^2-2xy-y^2=1$, $3x^2-6xy-2y^2=1$ and $2x^2-6xy-3y^2=1$ which can be shown to have no solutions by reducing modulo 3, 5 and 3 respectively.
$\blacksquare$
Lemma 1
If $D=a^2b^2c^2+4ac$ then $\sqrt{D}$ has the following continued fraction decomposition and set of periodic values of $X^2-DY^2$:
Proof Sketch
Use the standard method for computing the continued fraction of a quadratic irrational.
For example if $b$ is even:
$\sqrt{D}=abc+\sqrt{D}-abc$.
$1/(\sqrt{D}-abc)=(\sqrt{D}+abc)/4ac=b/2+(\sqrt{D}-abc)/4ac$.
$4ac/(\sqrt{D}-abc)=\sqrt{D}+abc$ etc...
$\blacksquare$
Lemma 2
If $(a,c)=1$ and $D=a^2b^2c^2+4ac$ then either $|4a|<\sqrt{D} \tag{4}$ or $|-4c|<\sqrt{D} \tag{5}$ holds for any positive $a,b,c$ with $a>1$ except for the following exceptions, $(a,b,c)\in\{(2,1,1),(2,1,3),(3,1,2)\}$
Proof
If $4c\leq abc$ then (5) is satisfied. Since $a\geq2$, if $b\geq 2$ (5) holds. So only $b=1$ is excluded. Also if $a\geq4$ $ab\geq4$ and so (5) again holds. Therefore we may now assume that $b=1$ and $2\leq a\leq3$ are the only possible exceptions. If $4a\leq abc=ac$ then (4) is satisfied. Therefore when $c\geq 4$ (4) holds. In total this yields $2\leq a\leq3$, $b=1$ and $1\leq c\leq3$ as possible values satisfying neither (4) nor (5). However we can use the condition (a,c)=1. to reduce this set of $(a,b,c)$'s to $\{(2,1,1),(2,1,3),(3,1,1),(3,1,2)\}$. $(3,1,1)$ satisfies (5) since $12^2<3^2+4\times3$ therefore we are left with $\{(2,1,1),(2,1,3),(3,1,2)\}$
$\blacksquare$
