For the action of the upper unipotent group, which we shall denote by $G$ here, one can continue in the following way: Let $g_r = \pmatrix{1 & r \\ 0 & 1}\in G$. Consider a $2\times 2$ real matrix $M=\pmatrix{a & b \\ c & d }$. Then $g_rMg_r^{-1} = \pmatrix{a+rc & b-r(a-d) - r^2c \\ c & d-rc}$. Let us write $x=a+d$ and
$y = a-d$. Then $x$ and $c$ are invariants under the action of $G$, and the action of $g_r$ on the coordinate functions $y$ and $b$ is given by $g_r\cdot y = y+2rc$ and $g_r\cdot b = b -ry - r^2c$. A direct calculation shows that the polynomial $bc+4y^2$ is also $G$-invariant (it is a linear combination, in fact, of the trace and determinant polynomials).
I claim that $\mathbb{R}[a,b,c,d]^G = \mathbb{R}[bc+4y^2,x,c]$.
To see why this is true, consider the localization $\mathbb{R}[a,b,c,d]_c$.
Since the action of $G$ on $c$ is trivial, we have an action of $G$ on this ring. We can then write $\mathbb{R}[a,b,c,d]_c = \mathbb{R}[x,y,c,b+4y^2/c]_c$.
In this presentation the action of $G$ on 3 of the generators is trivial, and one can show by an easy induction that if $G$ acts trivially on $p(x,y,c,b+4y^2/c)$ then no nontrivial power of $y$ appear in $p$.
We get that $\mathbb{R}[a,b,c,d]_c^G = \mathbb{R}[x,c,b+4y^2/c]_c$.
The ring of invariants in $\mathbb{R}[a,b,c,d]$ will just be the intersection of the ring of invariants in $\mathbb{R}[a,b,c,d]_c$ with $\mathbb{R}[a,b,c,d]$, and one can easily show that this ring is exactly $\mathbb{R}[c,x,bc+4y^2] = \mathbb{R}[c,tr,det]$.
About the embedding of $GL(1,\mathbb{C})$ in $GL(2,\mathbb{R})$:
We denote the group by $G$ again. We need to find the invariants in $\mathbb{R}[a,b,c,d]$. We will find the invariants in the extension of scalars: $\mathbb{C}[a,b,c,d]$. The advatage is that now the action of $x+iy\in \mathbb{C}^{\times}$ is given by one dimensional representations: on $a+d$ and $c-b$ the action is trivial, while on $(c+b) + (d-a)i$ and $(c+b) + (a-d)i$ the action is given by $(x-iy)^2/(x^2+y^2)$ and $(x+iy)^2/(x^2+y^2)$.
Since these two weights are of infinite order, and they are the inverses of each other, we get easily that the ring of invariants is now spanned by $a+d, b+c$ and $((c+b) + (d-a)i)((c+b) - (d-a)i) = (c+b)^2 + (a-d)^2$. Another set of generators will then indeed be $a+d,c-b, a^2+b^2+c^2+d^2$.