The ambiguity in the the added homotopy sphere is captured by a suitable inertia group $I(X)$, which in this case is the group of homotopy spheres $\Sigma$ such that $\Sigma$ bounds a parallelizable manifold, and the standard homeomorphism $\Sigma\# X\to X$ is homotopic to the diffeomorphism.
There is a related inertial group $\bar I(V)$ that consists of homotopy spheres $\Sigma$ such that $\Sigma\# X$ and $X$ are diffeomorphic. For example, if $X=S^3\times CP^2$, then $\bar I(X)=\Theta_7$, the group of all homotopy $7$-spheres.
The group $bP_d$ of homotopy spheres bounding parallelizable $d$-dimensional manifolds is cyclic of known order.
More precisely, the order is known except when $d=126$, the only remaining case of the Kervaire invariant problem, see here. Thus to understand the ambiguity in the the added homotopy sphere one needs to compute the index of $I(X)$ in $bP_{d}$ where $\dim(X)=d-1$.
Suppose $d$ is divisible by $4$ and $d\ge 8$, and $X$ is a closed oriented manifold of dimension $d-1$. Then $bP_{d}$ has large order given in terms of Bernoulli numbers. For such $d$ a theorem of L.Taylor says that the index of $I(X)$ in $bP_{d}$ is $\ge 2$. This is actually sharp, e.g., the index is $2$ for $X=S^3\times CP^{2m}$. The index is $4$ if $X=S^7\times CP^2$. On the other hand, Browder showed that
$I(X)$ is trivial when $d$ is not divisible by $8$, the group $H^1(X;\mathbb Z_2)$ is zero, and $X$ stably parallelizable; thus in this case one can ``determine the diffeomorphism type of the added homotopy sphere".
More details and references to the above claims can be found here
on p.8.
