To (Q1): Yes, $F$ being invertible means in this context exactly that $F$ admits a polynomial compositional inverse. If you are working over a general commutative ring $k$, then invertibility of $J_F$ is equivalent to invertibility of $\det J_F$ in $k[x_1,\dots,x_n]$. In particular, if $k$ is not reduced, then the determinant need not be a constant. Vice versa, even if the determinant is a non-zero constant, this does not guarantee invertibility of $J_F$, when $k$ is general (it has to be in the group of units of $k$).
To (Q2): Let me state a crucial observation that will hopefully clarify the matter (I believe it is also stated somewhere in BCW's paper.)
Lemma: Let $S$ be any $\mathbb{N}_0$-graded (noncommutative) ring and let $s \in S$ with $s_0 = 0$ (i.e. $s$ has vanishing degree 0 part). Then $1 + s \in U(S)$ if and only if $s$ is nilpotent, where $U(S)$ denotes the group of units of $S$.
(Proof: easy exercise.)
You can now take $S := \operatorname{M}_n(k[x_1,\dots,x_n]) = \operatorname{M}_n(k)[x_1,\dots,x_n]$, graded in the obvious way, for any commutative ring $k$. Thus, $J_F$ is invertible (as a matrix with polynomial entries) if and only if $J_N$ is nilpotent (as a matrix with polynomial entries).
Let me finally mention that the equivalence between the Jacobian Conjecture and BCW's reduction is a stable one: to prove JC, one has to prove JC for all $F$ of the form $F = I + N$ with $\deg(N)=3$, i.e. in all dimensions simultaneously. In other words, in order to prove JC in dimension $d$, it does not suffice to prove it only for $F = I + N$ with $F: k^d \to k^d$ of the form $F = I + N$.
To address your Edit, yes, BCW's reduction theorem applies to any commutative ring $k$ of characteristic 0, in particular to $k = \mathbb{R}$. Let me know if you have any further questions.