EDIT: New exact counterexample immediatley below:
At the request of the OP, I am providing an exact counterexample with all matrix elements being integer. As with my other counterexample,s $A$ and $B$ are both full rank, and as with my second counterexample, all $Y_i's$ are identity matrices.
$m = n = N = 2$.
>> disp(P1)
1 0
0 1
>> disp(P2)
1 0
0 2
>> disp(A)
60 42
44 38
>> disp(60*38-42*44)
432
>> disp(B)
22 19
-20 -14
>> disp(22*(-14)-(-20)*19)
72
>> Delta1 = A'*P1*B+B'*P1*A
Delta1 =
880 688
688 532
>> Delta2 = A'*P2*B+B'*P2*A
Delta2 =
-880 -688
-688 -532
+++++++++++++++++++++++++++++++++++++++++++
I am editing this post to add another counterewxample which shows counterexample exists even if the $Y_i$ are restricted to identity matrices. My original counterexample still stands, and is at the end (I also added singular values of Y1 and Y2, to demonstrate full rank). So as to make these counterexamples reproducible, I use exactly the matrices, as displayed.