N0. It is NOT the case that it is true in general. It is true for all cases with both $m \le 5$ and all prime $p n \lt 100$but may always be false . It is also true for $m=6$ and $n = 31$. But actually it is false ( which requires for $p\ge 31.$)
At m=6$) over $\mathbb{Z}$ and also over $\mathbb{Z}/{n}\mathbb{Z}$ for any rate$n \ge 36$, a specific example is the prime or composite.
The sets $$A=\{0, 1, 4, 10, 12, 17 \} \text{ and } B=\{ 0, 1, 8, 11, 13, 17\}$$ for $n= 37.$ Here are Sidon sets with $A-A=B-B$ is everything in $A \subset \mathbb{Z}/{37}\mathbb{Z}$ except \mathbb{Z}$. So the only restriction on $\{14, 15, 18, 19n$ is that they are Sidon sets. Since the elements of $A-A=B-B$ range from $-17$ to $17$, they are Sidon sets with $A-A=B-B$ in $\mathbb{Z}/{n}\mathbb{Z}$ for any modulus, 22prime or not, 23\}.$ starting with $n=35.$ It can does not work for $n=32,33,34$. It does also work for $n=31.$ However for $n=31$ we have $B=11A.$ Also, for $n=35$ we have $B=17A-11.$ These are the only cases where an affine map takes $A$ to $B$.
I found this example looking in $\mathbb{Z}/{n}\mathbb{Z}$ but it turns out to be checked that known.
I think it could be decided if there are no any exceptions for $m,b$ with m=4,5$ (other then the special case $B=\{ma+b \mid m=4$ and $n=13$) however the method seems tedious. Here is a \in A \}.$
There sketch: We may be assume that $A=\{0,1,a,b\}.$ If $B-B=A-A$ then $s-r=1$ for some construction here but I did not find one yet$r,s\in B.$ We can translate to have $r=0,s=1$ so that $B=\{0,1,w,x\}.$ Further more, $w,x$ are elements of $A-A.$ Three of the 13 elements ($-1,0,1$) are ruled out, leaving $\binom{10}{2}=55$ possible cases. Many of these (like $w=-x$, $w=1-x$ and $w=x-1$) are immediately seen to be impossible. The ratios between elements rest come in pairs $A$ \{0,1,w,x\}$ and $\{0,1,1-w,1-x\}$ which are all squares reflections of each other.
If we try $B=\{0,1,1-a,a-b\}$ then $A-A$ and $B-B$ overlap in $9$ elements leaving $\{b,-b,1-b,b-1\}$ unmatched in $A-A$ and $\{1-2a+b,-1+2a-b,a-b-1,1-a+b\}$ in $B-B$ If $b=1-2a+b$ then $n=2a-1$ and $B$ is seen to be a translated reflection of A. We can't have $b=1-a+b$ because $a \subset ne 1.$ The other two possibilities for $b$ also fail.
But if we try $B=\{0,1,-a,b-1\}$ there are $6$ unmatched expressions in each of $A-A$ and $B-B$ and running through the possible matchups of pairs one case is $b=-a-1,a-1=1-b-a$ which comes out to $a=3,b=-4$ which leaves the remaining elements as $\pm2,\pm 4,\pm 7$ for $A-A$ and $\pm2, \mathbb{Z}/{37}\mathbb{Z}$ but pm4, \pm6$ for $B-B$. So this sub-sub-case is possible for, but only partially the case for, $B.$n=13$ where, it can be checked, $B$ is a translated diolation of $A$.
