One can have $|2A|$ as small as $2|A|$. Take $A = H \cup \{g\}$ where $H$ is a subgroup, $g \notin H$ and$g \neq -g$. Then $|A+A| = 2|A| + O(1)$ while $g+H$ and $H - g$ all have a unique representative in $A-A$.
On the other hand, I can show if the number of uniquely representable elements of $A-A$ is at least $|A|$ and $|A+A| \leq (7/3)|A|$, then there is a subgroup, $H$, of size at most $(3/2)|A|$ such that the unique representatives lie in a coset of $H$. I'll adopt notation from Tao and Vu (mostly Chapter 2). Let
$$U : = \{x \in G : r_{A-A}(x) =1\}.$$
Let $g, h\in U$. By the Bonferroni inequalities, we have
$$ |A+A| \geq |A| + |A+g| + |A+h| - |A\cap(A+g)| - |A\cap(A+h)| - |(A+g) \cap (A+h)|.$$ As $g,h \in U$, we have $|A\cap(A+g)| = |A \cap (A+h)| = 1$ and so
$$|A+A| \geq 3|A| - 2 - r_{A-A}(g-h).$$
Thus if $r_{A-A}(g-h) \leq (1-\epsilon)|A|$
we have $$|A+A| \geq (2+\epsilon)|A| - 2.$$
So we suppose for all $g,h\in U$,
$$\tag{1}\label{1} r_{A-A}(g-h) \geq (1-\epsilon)|A|.$$
Note that \eqref{1} implies that $$U-U \subset {\rm Sym}_{1-\epsilon}(A).$$ Markov implies $$|{\rm Sym}_{1-\epsilon}(A)| \leq \frac{|A|}{1-\epsilon},$$ and so by assumption $$|U-U| \leq \frac{|A|}{1-\epsilon} \leq \frac{|U|}{1-\epsilon}.$$ Suppose now that $(1-\epsilon)^{-1} \leq 3/2$ (i.e. $\epsilon \leq 1/3$). Then by baby Freiman (see Theorem 1.5.2Theorem 1.5.2), we have that $$U \subset H + t,$$ for some $H \leq G$, with $|H| \leq (3/2)|A|$ and $t \in G$.