Question: "What is a dual action of the group?"
Answer: For simplicity, if $X:=Spec(A)$ and $G:=Spec(R)$ is a linear algebraic group over a field $k$ acting on $X$ via $\sigma^*: G\times_k X \rightarrow X$ you get a "dual action"
$$\sigma: A \rightarrow R\otimes_k A.$$
If $L\in Pic(A)$ is an invertible module with $\mathcal{L}:=\mathbb{V}(L^*):=Spec(Sym_A^*(L^*)):=Spec(B)$, a $G$-linearization of $L$ gives an action
$$\gamma^*: G\times \mathcal{L}\rightarrow \mathcal{L}$$
and a "dual action"
$$\gamma: B \rightarrow R\otimes_k B.$$
There are canonical maps $A\rightarrow B$ and $R\otimes_k A \rightarrow R\otimes_k B$ and all diagrams are to commute.
The map $\sigma$ is a local version of your map on global sections
$$H^{0}(X,\mathcal{L})\rightarrow H^{0}(G,\mathcal{O}_{G})\otimes H^{0}(X,\mathcal{L}).$$
Question: "Can we talk about invariant sections of the line sheaf? How? We have a correspondence between sections of an invertible sheaf, and sections of the associated line bundle, that is, map of schemes X→L which are sections of the projection map L→X. In this sense, what is an invariant section of the line bundle?"
Answer: You formulate the notions "invariant function" and "invariant section" using the dual actions $\sigma, \gamma$ defined above.
Example: If $X \subseteq \mathbb{P}^n_k$ is a quasi projective scheme with an action $\sigma^*:G\times_k X \rightarrow X$, and where you can give $X$ an open affine cover $U_i:=Spec(A_i)$ of $G$-invariant subschemes, you get induced dual actions
$$ \sigma_i:A_i \rightarrow R\otimes_k A_i.$$
You use these dual actions $\sigma_i$ to study the action $\sigma$ and the "quotient" $X/G$.
When $G$ is "finite" such an affine open cover always exist. Moreover the quotient $X/G$ "exist" and you define it locally as $Spec(A_i^G)$.
Example: if $C:=\mathbb{P}^1_k$ is the projective line, there is an action of the symmetric group $S_n$ on $C^{\times_k n}$, and you may prove that if $n\neq char(k)$ there is an isomorphism
$$Sym^n(C):=(C^{\times_k n})/S_n \cong \mathbb{P}^n_k.$$
You embed the product $C^{\times_k n}$ into a projective space and use an open affine $S_n$-invariant cover to construct the quotient $Sym^n(C)$ and to prove it is isomorphic to projective $n$-space. So you cannot construct the symmetric product $Sym^n(C)$ as $Spec(A^G)$ for some group $G$ acting on a ring $A$, since projective space is not affine.
Example: An affine example and a "set theoretic paradox". If $G:=S_n$ is the symmetric group on $n$ letters acting on the polynomial ring $A:=k[x_1,..,x_n]$ it follows the invariant ring $A^G \cong k[s_1,..,s_n] \subseteq A$ where $s_i$ are the elementary symmetric polynomials. The polynomials $s_i$ are algebraically independent over $k$ hence $A^G$ is a polynomial ring and there is an isomorphism
$$ \pi: \mathbb{A}^n_k \rightarrow \mathbb{A}^n_k/G \cong \mathbb{A}^n_k.$$
The map of schemes $\pi$ gives at the level of topological spaces a surjective and non-injective endomorphism
$$\pi: \mathbb{A}^n_k \rightarrow \mathbb{A}^n_k$$
of affine $n$-space. Since the ring extension $A^G \subseteq A$ is an integral extension, it follows $\pi$ is surjective. Given any prime ideal $\mathfrak{p} \subseteq A$, it follows the orbit of $\mathfrak{p}$ under the action of $G$ has $n!$ elements in general. Hence the map $\pi$ is not injective. To some people (not all) this is a "paradoxical situation".
Note: When working with schemes over fields that are not algebraically closed you will need this construction. Some people are "sloppy" when considering such actions and do not use the dual action.
