Of course, one might say that both $Z(G)$ and $[G,G]$, in a sense, "measure" the non-commutativity of $G$. But they are not very good "quantitative" measures.

I think what you are aiming at is a notion introduced by Turán and Erdős (*Some problems of a statistical group theory IV*, Acta Math. Acad. of Sci. Hung. **19** (1968), 413-435), the "probability that two elements of $G$ commute":
$$P(G) = \frac{\left|\{ (x,y)\in G\times G\mid
xy=yx\}\right|}{|G|^2}.$$
In fact, $P(G) = k/|G|$, where $k$ is the number of conjugacy classes of $G$. Gustafson proved that if $G$ is nonabelian then $P(G)\leq 5/8$, and extended the notion to compact groups using Haar measure (W. Gustafson, *What is the probability that two group elements commute?* American Math. Monthly **80** (1973) 1031-1034). MacHale proved that certain values cannot occur: if $P(G)\gt \frac{1}{2}$, then $P(G) = \frac{1}{2} + \left(\frac{1}{2}\right)^{2s+1}$; and $P(G)$ cannot satisfy $\frac{7}{16} \lt P(G) \lt \frac{1}{2}$. Joseph proved that if $G$ is not commutative and $p$ is the smallest prime that divides $|G|$, then $P(G)\leq \frac{p^2+p-1}{p^3}$ (K.S. Joseph, *Commutativity in non-abelian groups*, PhD thesis, 1969, UCLA). There's been some other work on this.

In the case of $S_3$. $|G|=6$, and the set of pairs $(x,y)$ with $xy=yx$ is, as you note, $18$, so the probability that two elements commutes is precisely your "50% nonabelian".

Your second notion seems to be that of looking at $G/[G,G]$, which is the "largest" quotient of $G$ which is abelian.

**Added:** Since I edited to fix the accent on Erdős, I'll take the opportunity to add some references:

- Desmond MacHale,
*How commutative can a non-commutative group be?*, Math. Gazette **58** (1974), 299-202.
- David J. Rusin,
*What is the probability that two elements of a finite group commute?*, Pacific J. Math **82** (1979), no. 1, 237-247.
- Robert Guralnick and Geoff Robinson,
*On the commuting probability in finite groups*, J. Algebra **300** (2006), no. 2, 509-528, MR 2228209 (2007g:60011); *Addendum*, J. Algebra **319** (2008), no. 4, 1822.