If you take, say, set of real points of the group-scheme $O(n)$, i.e., $O(n, {\mathbb R})$, then you recover the usual orthogonal (real Lie) group, which you know as $O(n)$. Same applies to $SL(n)$, etc. There is one case when this does not work well, namely when you deal with character varieties. For instance, take $\pi$, say, the free group on two generators, and try to form the quotient $Q=Hom(\pi, SU(2))/SU(2)$. The standard way to do this is to consider the corresponding character variety (or, rather, affine scheme) $X$ and take its set of real points. However, the result will contain both equivalence classes of representations of $\pi$ to $SU(2)$ (as you expected), but also equivalence classes of representations to $SL(2, {\mathbb R})$! The easiest way to see this is to realize that the coordinate ring of $X$ is generated by traces of the elements $A, B, C=AB$ of $\pi$ (where $A, B$ are the free generators). To get the set of real points, you need to use points with real traces, so you end up with the elements of both real Lie groups $SU(2)$ and $SL(2, {\mathbb R})$. This is rather annoying, but one can learn to live with this problem. Namely, in order to isolate $Q=Hom(\pi, SU(2))/SU(2)$ inside $X({\mathbb R})$, you impose also some inequalities, so $Q$ becomes a real semi-algebraic subset. Same problem appears if you consider $Hom(\pi, SL(2, {\mathbb R}))$: Character variety will give you unitary representations as well. The standard way to deal with this problem (in Teichmuller theory) is to consider not all representations to $SL(2, {\mathbb R})$, but only discrete and faithful ones, so that the commutator $[A,B]$ maps to elements of the fixed trace.
Then you can form the (topological) quotient by $SL(2, {\mathbb R})$ by taking slice, i.e., restricting to representations $\rho$ so that the (attractive, repulsive) fixed points of $\rho(A)$ are $0, \infty$ and the attractive fixed point of $\rho(B)$ is $1$.
Addendum: More generally, in all "interesting" case I know, the desired quotient can be constructed without algebraic geometry. Suppose you are interested in $R:=Hom(\pi, O(n,1))$ and the group $\pi$ is "nonelementary", i.e., is not virtually abelian. Then, in $R$ consider the open subset $R'$ consisting of representations $\rho$ so that $\rho(\pi)$ does not fix a point in $S^{n-1}$. For instance, $R'$ will contain all discrete and faithful representations. Then you can just take the "naive" quotient $Q=R'/O(n,1)$ instead of the (set of real points of) character variety $X({\mathbb R})$. Then $Q$ will embed in $X({\mathbb R})$.
Concerning existence of a slice: The same argument I described works for representations to $SL(2, {\mathbb C})$. However, if you consider representations to $O(n,1), n\ge 4$, there is no (in general) global slice. However, locally, it does exist, see e.g.
Slice Theorem for the general information about slices for group actions.
Recommended reading: D. Johnson and J. Millson, Deformation spaces, associated to compact hyperbolic manifolds, in "Discrete Groups in Geometry and Analysis" (Papers in honor of G. D. Mostow on his sixtieth birthday), 1984. I read this paper as a graduate student and still find it useful.
By the way, here is where algebraic viewpoint is definitely superior to the Lie theoretic. Suppose that you are interested in understanding local structure of the analytic variety $Hom(\pi, G)$, where $G$ is a real Lie group, i.e., what singularity it has at a representation $\rho: \pi\to G$. In general it is a rather difficult problem as singularities could be "arbitrarily complicated." See M.Kapovich, J.Millson, "On representation varieties of Artin groups, projective arrangements and the fundamental groups of smooth complex-algebraic varieties", Math. Publications of IHES, vol. 88 (1999), p. 5-95, one of the main results there is that singularities of character varieties could be arbitrary (defined over ${\mathbb Q}$).
So, assume that $G=\underline{G}({\mathbb R})$, where $\underline{G}$ is an algebraic group (scheme) over reals. Let $A$ be an Artin local ${\mathbb R}$-ring with the projection to the residue field $\nu: A\to {\mathbb R}$. Then the set of $A$-points $G_A:=\underline{G}(A)$ is a certain nilponent extension of the Lie group $G$ with the quotient $\nu_G: G_A\to G$ induced by $\nu$. Then, instead of analyzing the scheme $Hom(\pi, \underline{G})$ at $\rho$, you consider the collection of real-algebraic sets $Hom_{\rho}(\pi, G_A)\cong Hom_{\rho}(\pi, \underline{G})(A)$, consisting or representations $\tilde\rho: \pi\to G_A$ which project to $\rho$ under $\nu_G$. The point is that the collection of real-algebraic sets $Hom_{\rho}(\pi, G_A)$ "knows everything" (and even more!) about the singularity of $Hom(\pi, G)$. For instance, to recover the (Zariski) tangent space $T_\rho Hom(\pi, G)$, you just take $A$ to be the "dual numbers", which is the quotient ${\mathbb R}[t]/(t^2)$. Then
$$
T_\rho Hom(\pi, G)\cong Hom_{\rho}(\pi, G_A)$$
for this choice of $A$.
This staff is explained in the paper
W.Goldman, J.Millson, The Deformation Theory of Representations of Fundamental Groups of Compact Kahler Manifolds, Publ. Math. I.H.E.S.; 67 (1988).
