Suppose that the algebraic group $G$ over $k$ acts on the vector space $V$, i.e. that
there is map of algebraic groups $G \to GL(V).$ Passing to Lie algebras (= Zariski tangent
space to the identity) is a functor, and so we get a map on Lie algebras
$\mathfrak g \to End(V),$ which is the corresponding Lie algebra representation.

Another way to do this, closer to the differential point of view (and which you will
need anyway to identify the Lie algebra of $GL(V)$ with $End(V)$) is as follows:
$G(k[\epsilon])$ acts on $V[\epsilon]$ (take dual number-valued points of the
original morphism). In particular, the Zariski tangent space at the identity
(which on the one hand is $\mathfrak g$, by definition, and on the other hand
is the subgroup of $G(k[\epsilon])$ consisting of elements mapping to the identity
under the specialization $\epsilon \mapsto 0$) acts on $V[\epsilon]$ by endomorphisms
which reduce to the identity after setting $\epsilon = 0$. One checks that such
a map is of the form $v \mapsto v + L(v)\epsilon,$ where $L \in End(V)$.

Sending it to $L$ gives the required map $\mathfrak g \to End(V)$.

(Note: we are taking $g \in G(k[\epsilon])$ lying over the identity, applying
the representation $\rho$ to get $\rho(g)$, then forming the difference quotient
$(\rho(g) - 1)/\epsilon.$ Hopefully the connection with differentiation is clear.)

One has to be slightly cautious about going back from $\mathfrak{g}$ to $G$,
since there are the following subtleties which any approach has to take into account:
the field $k$ had better have char. 0; the group $G$ had better be linear algebraic,
and furthermore either nilpotent, or simply connected semi-simple; and the representation
has better be finite-dimensional.

One could try the following: take a finite dimensional representation $V$ of
$\mathfrak{g}$; extend it to a rep. of the universal enveloping algebra $U(\mathfrak{g})$;
use the fact that $V$ is finite-dimensional to extend the rep'n to a certain completion
of $U(\mathfrak{g})$; inside this completion, look at the group-like elements under the
canonical co-multiplication on $U(\mathfrak{g})$; show that these elements form a linear algebraic group $G$ with Lie algebra $\mathfrak g$. (The intuition is that we can map
$\mathfrak{g}$ into a well-chosen completion of $U(\mathfrak{g})$ via a formal version
of the exponential map.)

[This last suggestion is based on a discussion in Serre's Lie algebras/Lie groups book, but I don't remember
if he carefully treats this algebraic group context; it may be that he is rather focussing on the Lie group setting.]