Pulling back the lisse-etale structure sheaf via the inclusion of a point For a scheme $X$, let $LE(X)$ denote the lisse-etale site on $X$. This is the full subcategory of $\textbf{Sch}/X$ consisting of smooth morphisms to $X$, equipped with the etale topology. Let $\mathcal{O}_X$ denote the presheaf on $LE(X)$ sending
$$(U\rightarrow X)\mapsto \Gamma(U,\mathcal{O}_U)$$
This is in fact represented by $\mathbb{A}^1_X$, and hence is a sheaf (of sets)
Now let $k$ be an alg. closed field. Let $X := \mathbb{A}^1_k$ with coordinate $t$. Let $pt := \text{Spec }k$.
Consider the map $i : pt\rightarrow X$ sending $pt\mapsto (t=0)$. By Yoneda, it is possible to show that $i^*\mathcal{O}_X = \mathcal{O}_{pt}$ (ie, the inverse image sheaf of the functor of points of $\mathbb{A}^1_X$ in $LE(X)$ is the functor of points of $\mathbb{A}^1_{pt}$ in $LE(pt)$).
In particular, $i^*\mathcal{O}_X(pt) = k$. I don't see how this agrees with the definition of $i^*$.
By definition, $i^*\mathcal{O}_X$ is the sheafification of the presheaf defined as follows:
For every $Z\in LE(pt)$, consider the category $I_Z$ consisting of objects $(U,\rho)$ where $U\in LE(X)$ and $\rho : Z\rightarrow U_{pt} := U\times_X pt$ is a morphism in $LE(pt)$. A morphism $(U,\rho)\rightarrow (U',\rho')$ is a morphism $g : U\rightarrow U'$ in $LE(X)$ such that $g_{pt}\circ \rho = \rho'$.
Then, we consider the presheaf on $LE(pt)$:
$$(i^*)^{pre}\mathcal{O}_X : (Z\rightarrow pt)\mapsto \varinjlim_{(U,\rho)\in I_Z^{op}} \mathcal{O}_X(U) = \varinjlim_{(U,\rho)\in I_Z^{op}}\Gamma(U,\mathcal{O}_U)$$
where given $h : Y\rightarrow Z$ in $LE(pt)$, we get a restriction map given by the functor $I_Z\hookrightarrow I_Y$ sending $(U,\rho)\mapsto (U,\rho\circ h)$. Then, we define $i^*\mathcal{O}_X$ to be the sheafification of this presheaf.
(So far this is the setup on p45-47 of Olsson's "Algebraic spaces and stacks")
My issue is: by the definition of $i^*\mathcal{O}_X$, there is a natural map
$$k[t] = \mathcal{O}_X(X)\rightarrow i^*\mathcal{O}_X(pt) = k$$
Which can't be injective, and for example presumably sends $t,t^2$ both to 0. However, the global sections of the presheaf:
$$(i^*)^{pre}\mathcal{O}_X(pt) = \varinjlim_{(U,\rho)\in I_{pt}^{op}}\mathcal{O}_X(U),$$
contains $\mathcal{O}_X(X)$ as a subset (since every $U\rightarrow X$ is smooth, hence flat, hence induces injections on global sections). Thus, the images of $t,t^2$ are distinct in $(i^*)^{pre}\mathcal{O}_X(pt)$. This means that if $t,t^2$ both map to 0 in $i^*\mathcal{O}_X(pt)$, they must agree in some etale covering of $pt$, but this seems impossible since $k$ is alg. closed, so the etale covers carry no additional information.
EDIT: Here is the argument that $i^*\mathcal{O}_X\cong\mathcal{O}_{pt}$. We have, for any sheaf $F$ on $LE(pt)$:
$$Hom_{LE(pt)}(i^*h_{\mathbb{A}^1_X},F) = Hom_{LE(X)}(h_{\mathbb{A}^1_X},i_*F) = (i_*F)(\mathbb{A}^1_X) := F(i^*\mathbb{A}^1_X) = F(\mathbb{A}^1_{pt}) = Hom_{LE(pt)}(h_{\mathbb{A}^1_{pt}},F)$$
where we've used Yoneda twice, in the second and final equalities. But then, since this holds for any sheaf $F$ on $LE(pt)$, comparing the last term with the first, by covariant Yoneda we find that $i^*h_{\mathbb{A}^1_X} = h_{\mathbb{A}^1_{pt}}$ Since $\mathcal{O}_X$ is represented by $\mathbb{A}^1_X$ and $\mathcal{O}_{pt}$ is represented by $\mathbb{A}^1_{pt}$, this shows that $i^*\mathcal{O}_X \cong \mathcal{O}_{pt}$.
 A: EDIT
I just realized that my previous version of the answer was totally misleading. Here is what really is problematic in your argument:
Just because the images of $t$ and $t^2$ are distinct in $\mathcal{O}_X(U)$ it does not mean that they map to distinct elements in $\varinjlim \mathcal{O}_X(U)$.
Here is a concrete description of
$i^{-1} \mathcal{O}_X\cong i^*\mathcal{O}_X$
in terms of the colimit construction you describe.
Given a an object $Z \to pt$ in $LE(pt)$,
the elements of $(i^* \mathcal{O}_X)(Z \to pt)$ are given by
commutative squares
$\require{AMScd}$
\begin{CD}
Z @>\rho>> U \\
@VVV  @VV \pi V\\
pt @>>> X \\
\end{CD}
together with a section $\sigma \in \mathcal{O}_X(U)$.
Another element described by
\begin{CD}
Z @>\rho'>> U' \\
@VVV  @VV \pi' V\\
pt @>>> X \\
\end{CD}
and $\sigma' \in \mathcal{O}_X(U')$ is considered
equivalent to the first element if there exists
a morphism $h\colon U \to U'$ sucht that the obvious
diagram commutes and such that the restriction
of $\sigma'$ along $h$ equals $\sigma$.
Now since $\mathcal{O}_X$ is represented by $\mathbb{A}^1_X$
in $LE(X)$, there is a universal global section $u \in \mathcal{O}_X(\mathbb{A}^1_X)$ such that for any global section
$\sigma \in \mathcal{O}_X(U)$, there is a unique morphism
$\widetilde{\sigma}\colon U \to \mathbb{A}^1_X$ over $X$ such that
$\sigma$ is the pull-back of $u$ along $U \to \mathbb{A}^1_X$.
It follows that the element in $(i^* \mathcal{O}_X)(Z \to pt)$ as above has a canonical representative given by the commutative square
\begin{CD}
Z @>\widetilde{\sigma}\circ\rho>> \mathbb{A}^1_X \\
@VVV  @VVV\\
pt @>>> X \\
\end{CD}
and the universal global section $u \in \mathcal{O}_X(\mathbb{A}^1_X)$.
Clearly the set of such diagrams is in canonical bijection with the set of 
morphisms $Z \to \mathbb{A}^1_{pt}$ over $pt$.
I keep my old answer below.
END EDIT
Your description of the lisse-étale site $LE(X)$ and the structure sheaf $\mathcal{O}_X$ for a scheme $X$ is correct.
Given a morphism $f\colon X \to Y$ of schemes you get a functor $f_* \colon Sh(LE(X)) \to Sh(LE(Y))$ on the corresponding categories of sheaves together with a morphism
$\mathcal{O}_Y \to f_*\mathcal{O}_X$ of sheaves of rings. This basically follows from the fact that the categories $LE(X)$ and $LE(Y)$ both are restrictions of the big étale site, and it is very easy to see what this functor does.
The functor $f_*$ (considered as a functor of sheaves of sets) has a left adjoint $f^{-1}$, which you describe (and call $f^*$). By adjunction, you get a corresponding morphism $f^{-1}\mathcal{O}_Y \to \mathcal{O}_X$ on $Sh(LE(X))$.
A priori this is a morphism of sheaves of sets, but since $f^{-1}$ preserves finite products (which is something which needs a proof), you get a ring structure on $f^{-1}\mathcal{O}_Y$ and the morphism $f^{-1}\mathcal{O}_Y \to \mathcal{O}_X$ is a homomorphism of sheaves of rings.
However, this is not an isomorphism in general.
EDIT: In fact it is an isomorphism when we are working on the lisse-étale site since by the argument given in the question. 
To construct the left adjoint $f^*$ of $f_*$ as a functor of sheaves of modules, you let $f^*\mathcal{M} = \mathcal{O}_X \otimes_{f^{-1}\mathcal{O}_Y} f^{-1}\mathcal{M}$ (with the tensor product taken as sheaf of rings). From this description, it is clear that $f^*\mathcal{O}_Y \cong \mathcal{O}_X$.
But this does not follow from the Yoneda lemma as you suggested.
EDIT: Well, it does, but not in the direct way suggested below.
The reason is that $f\colon X \to Y$ is not an object of $LE(Y)$ in general, so the Yoneda lemma does not apply. In particular, this is the case in your example as $\iota\colon pt \to X$ is not smooth. However, if $f\colon X \to Y$ happens to be smooth, then the Yoneda argument works and $f^{-1}\mathcal{O}_Y \cong \mathcal{O}_X$.
A: Thanks to Daniel Bergh's answer, after an hour of thinking about this, I finally understand the problem.
Fundamentally, the confusion stemmed from a bad intuition about inductive limits (even of sets!). This, in turn, led to the bad intuition that $(i^*\mathcal{O}_X)(pt)$ is "essentially a union" of the global sections of $U$ for smooth morphisms $U\rightarrow X$ whose image contains $pt$
Let $I$ be a small index category, and suppose we have a functor $F : I\rightarrow Sets$. Then, the limit $\varinjlim F$ is the set 
$$\left(\bigsqcup_{i\in I} F(i)\right)/\sim$$
where for any $x_i\in F(i), x_j\in F(j)$, we declare $x_i\sim x_j$ if there is a morphism $i\rightarrow j$ such that $F(i\rightarrow j)(x_i) = x_j$. Note that we do not require that morphisms $i\rightarrow j$, if they exist, need be unique. Ie, $Hom_I(i,j)$ may have cardinality greater than 1.
For every $i\in I$, we have a canonical map $F(i)\rightarrow\varinjlim F$.
Suppose now that:
($\ast$) $I$ has an initial object $i_0$, such that for all $i\in I$, $F(i_0\rightarrow i)$ is an injection (in $Sets$). Note that this is the case for the diagram given by taking global sections of $I_{pt}^{op}$ - the initial object being (the global sections of) $(X,id_{pt})$, the maps being injections since $X = \mathbb{A}^1_k$ is irreducible.
Note that this condition $(\ast)$ implies that any map $i\rightarrow i_0$ in $I$ must be a retraction of the unique map $i_0\rightarrow i$.
It is NOT TRUE that ($\ast$) would imply that the canonical map $F(i_0)\rightarrow \varinjlim F$ would be injective. (This was my bad intuition) For example, suppose $F$ is given by the diagram consisting of two objects $A,B$, and 3 (nonidentity) morphisms given as follows:


*

*A unique injection $f : A\hookrightarrow B$ and

*two maps $g,g' : B\rightarrow A$ satisfying $g\circ f = g'\circ f = id_A$


then this satisfies our hypotheses ($\ast$) ($A$ is the "initial object" here), and in this case $\varinjlim F$ is actually the coequalizer of $B\stackrel{g,g'}{\rightrightarrows} A$, so if $g\ne g'$, then the canonical map $A\rightarrow \varinjlim F$ is not injective. 
Indeed, this can happen in our situation. In particular, let $X = \mathbb{A}^1_k = \text{Spec }k[x]$ (we've changed the coordinate from $t$ to $x$), let $i : pt := \text{Spec }k \rightarrow X$ be the inclusion of the point $x = 0$. Now let $\mathbb{A}^1_X = \text{Spec }k[x,y]$ with the structure morphism given by $k[x]\hookrightarrow k[x,y]$. Then, the two sections $X\rightrightarrows\mathbb{A}^1_X$ given by $y\mapsto x,y\mapsto x^2$ (this exists in $LE(X)$, and can be extended to objects of $I_{pt}$) at the level of global sections corresponds to
$$k[x,y]\rightrightarrows k[x]\qquad y\mapsto x,x^2$$
Then, in the inductive limit of this diagram, we have $x = y = x^2$.
We note that this situation cannot happen if we replace $LE(X)$ with the small etale site $X_{et}$. Indeed, in this case, for the inclusion of $pt$ into any irreducible scheme $X$, $\Gamma(X,\mathcal{O}_X)$ would sit as a subring of $(i^*\mathcal{O}_X)(pt)$, essentially because any section in $X_{et}$ is an isomorphism onto its image. Though, in this case $i^*\mathcal{O}_X\ncong \mathcal{O}_{pt}$, which is morally related to the fact that $\mathbb{A}^1_X\notin X_{et}$.
