Let $M$ be a non-compact matrix Lie group and $T_e M$ its lie algebra.
Consider a point $x \in M $ and $ \triangle \in T_e M$.
To move from $x$ to a point $y \in M$ along $\triangle$, below group operation seems to be commonly used in iterative optimization on Lie groups.
$y = x \exp(\triangle)$
Is this straight forward? How can this operation be explained?
The answer of Theo basically says it all what the exponential is concerned, but I maybe can shed some light regarding the optimization perspective.
Let $M$ be your Lie group and suppose it is a subgroup of $\textrm{GL}_n$. Now, if you want to solve \[ \min f(x) \quad \textrm{s.t. $x \in M$} \] by an iterative method, one usually uses a so-called smooth retraction map \[ R_x: T_x M \to M. \] to replace your current approximation $x$ to $x_+ := R_x(\Delta)$. Think for example of doing a line-search $R_x(-t \textrm{ grad}_x f)$ for $t>0$ where $\textrm{ grad}_x f$ is the Riemannian gradient of $f$.
(The retraction map has to fulfill some properties, in order for this to work, likes smoothness and being a first-order approximation of the geodesics (see below).)
Due to the left (or right) action of a Lie group on itself by multiplication, the exponential mapping at the identity $\exp$ can be transported to get a retraction at $x$ as $x\exp$. As Theo already explained, $\exp$ does not need to be a global diffeomorphism, but that is not needed for optimization, since we perform updates locally.
Another typical choice for $R_x$ are the geodesics in $x$. For some metrics, the $\exp$ coincides with the geodesics (for instance, bi-invariant metrics), but not always. One can also use cheaper alternatives for $R_x$ if your are only concerned with optimization. A nice reference for this retraction-based optimization on manifolds (and so, also Lie groups) is http://press.princeton.edu/titles/8586.html .
If I understand your question correctly, the story is as follows. The Lie group $M$ acts on itself by (left, say) translation, and this action is transitive and faithful. From this, we can trivialize the tangent bundle to $M$, by identifying ${\rm T}_mM$ with ${\rm T}_eM$ via the differential of the "left-multiply by $m$" map. In this way, a choice of $\Delta \in {\rm T}_eM$ determines (and indeed is the same as) a right-invariant vector field $\Delta \in \Gamma({\rm T}M)$. It is complete, in the sense that even when $M$ is not compact, this vector field has arbitrary-length flow. The map $x \mapsto x\exp(\Delta)$ is "flow by time $1$" for this vector field; in particular $\exp(\Delta)$ is defined by flowing for time $1$ starting at $e$.
Fix $x \in M$. As $\Delta$ varies through ${\rm T}_eM$, the value of $x \exp(\Delta)$ smoothly varies over $M$. For small $\Delta$, the map $\Delta \mapsto x\exp(\Delta)$ is a local diffeomorphism (the derivative of this map as $\Delta = 0$ is the left-translation map ${\rm T}_e M \to {\rm T}_x M$, and so is an isomorphism). So when $y$ is close to $x$, there is a unique $\Delta$ close to $0$ with $y = x\exp \Delta$.
Note that there might be other solutions $\Delta$ to $y = x\exp \Delta$, but they necessarily have $\Delta$ "large". Note also that when $x,y$ are not close together, then there is no guarantee that a solution exists at all (even if $M$ is connected). (When $M$ is compact connected, there is necessarily a solution, but not in the noncompact case.) The standard example is $M = {\rm SL}(2,\mathbb C)$, the group of complex $(2\times 2)$-matrices with determinant $1$. The Lie algebra ${\rm T}_e M$ can be identified with the vector space $\mathfrak{sl}(2,\mathbb C)$ of complex $(2\times 2)$-matrices with trace $0$, and then the exponential map is the matrix exponential. Every traceless $(2\times 2)$ matrix with nonzero eigenvalues has distinct eigenvalues, and hence is diagonalizable; therefore its exponential is also diagonalizable. In particular, there does not exist a traceless $2\times 2$ matrix $\Delta$ with $$ \exp(\Delta) = \begin{pmatrix} -1 & 1 \\ 0 & -1 \end{pmatrix} $$ as this matrix is not diagonalizable, but its eigenvalues are not $1$ so it is not the exponential of a matrix with eigenvalues $0$.
I don't know anything about iterative optimization. Any reasonable book on Lie theory, even if it only discusses matrix groups, should contain a good explanation of exponentials.
