Your first calculation for $3\times3$ matrices applies in full generality: If your matrix writes blockwise $[1 \quad e^T ; e \quad M]$, with $e^T=(1,\ldots,1)$, and if $M$ is non-singular (implied by your assumption), then the property that $\det A=0$ is equivalent to $e^T\hat M e=\det M$, that is $e^TM^{-1}e=1$, or to $\det(J-M)=0$, with $J=ee^T$ the matrix with $1$s everywhere. Just use the Sherman-Morrison formula $\det(B+xy^T)=(\det B)(1+y^tB^{-1}x)$.
I have no time to continue now, but I keep it in mind.