If I understand your question correctly, the story is as follows. The Lie group $M$ acts on itself by (left, say) translation, and this action is transitive and faithful. From this, we can trivialize the tangent bundle to $M$, by identifying ${\rm T}_mM$ with ${\rm T}_eM$ via the differential of the "left-multiply by $m$" map. In this way, a choice of $\Delta \in {\rm T}_eM$ determines (and indeed is the same as) a right-invariant vector field $\Delta \in \Gamma({\rm T}M)$. It is complete, in the sense that even when $M$ is not compact, this vector field has arbitrary-length flow. The map $x \mapsto x\exp(\Delta)$ is "flow by time $1$" for this vector field; in particular $\exp(\Delta)$ is defined by flowing for time $1$ starting at $e$.

Fix $x \in M$. As $\Delta$ varies through ${\rm T}_eM$, the value of $x \exp(\Delta)$ smoothly varies over $M$. For small $\Delta$, the map $\Delta \mapsto x\exp(\Delta)$ is a local diffeomorphism (the derivative of this map as $\Delta = 0$ is the left-translation map ${\rm T}_e M \to {\rm T}_x M$, and so is an isomorphism). So when $y$ is close to $x$, there is a unique $\Delta$ close to $0$ with $y = x\exp \Delta$.

Note that there might be other solutions $\Delta$ to $y = x\exp \Delta$, but they necessarily have $\Delta$ "large". Note also that when $x,y$ are not close together, then there is no guarantee that a solution exists at all (even if $M$ is connected). (When $M$ is compact connected, there is necessarily a solution, but not in the noncompact case.) The standard example is $M = {\rm SL}(2,\mathbb C)$, the group of complex $(2\times 2)$-matrices with determinant $1$. The Lie algebra ${\rm T}_e M$ can be identified with the vector space $\mathfrak{sl}(2,\mathbb C)$ of complex $(2\times 2)$-matrices with trace $0$, and then the exponential map is the matrix exponential. Every traceless $(2\times 2)$ matrix with nonzero eigenvalues has distinct eigenvalues, and hence is diagonalizable; therefore its exponential is also diagonalizable. In particular, there does not exist a *traceless* $2\times 2$ matrix $\Delta$ with
$$ \exp(\Delta) = \begin{pmatrix} -1 & 1 \\ 0 & -1 \end{pmatrix} $$
as this matrix is not diagonalizable, but its eigenvalues are not $1$ so it is not the exponential of a matrix with eigenvalues $0$.

I don't know anything about iterative optimization. Any reasonable book on Lie theory, even if it only discusses matrix groups, should contain a good explanation of exponentials.