The quotient of relative spec and proj Suppose $X=spec\, R$ for some nice ring $R$, and $G$ is a finite group acting on $X$ freely, then I can write the quotient as $Y=spec\, R^G$. 
Now what if $X$ is defined as relative $\mathbf{Spec}$ or $\mathbf{Proj}$ over some scheme $A$? For example, suppose $X=\mathbf{Spec}_A\, R$ for some sheaf of $\mathcal{O}_A$-algebras $R$, and I have group actions defined by $T_g^*R\to R$ over $A$ ($G$ is also acting on $A$ freely) for each group element $g\in G$. Then $G$ is acting on the total space $X=\mathbf{Spec}_A\, R$. Then is it possible to write the quotient $Y=X/G$ as a relative spectrum of some invariant subsheaf of $R$?. 
 A: As long as the action of $G$ on $A$ is not trivial, this is not going to work, because $X/G$ will not be an $A$-scheme. 
Here is a concrete, but somewhat general example: Let $Z$ be an arbitrary affine scheme and $X=A\times Z$. Define the $G$-action by acting on $A$ with the given free action and act trivially on $Z$. This gives a free $G$-action on $X$. The quotient $X/G$ is just $(A/G)\times Z$, which will not map to $A$ in general. 
The context you may have a chance of making this work is if you want $X/G$ to be a relative $\mathrm{Spec}$ over $A/G$. That actually seems to work. Since $G$ is finite, $X$ is also affine over $A/G$, so you may actually assume that $G$ acts trivially on $A$. In that case, you can just look at the $G$-invariant part of $R$ and take the relative $\mathrm{Spec}$ according to that. This should give you what you want, since locally you just have $\mathrm{Spec}$'s and in that case you already know what you want and the base is fixed, so everything is dandy.
A: If $G$ is a finite group, and $X$ is the relative spectrum of $R$ on $A$, and $\pi: A \to A/G$ is the quotient map, then $X$ is also the relative spectrum of $\pi_* R$ on $A/G$. This should work for any affine morphism, in fact.
$\pi_* R$ carries a natural $G$ action, where $G$ now acts trivially on the base $A/G$. The invariant subsheaf will give you $Y$. Proof: The question is local, and locally on affines this is just the original question you stated.
An alternative way of viewing this is that, as the quotient $A \to A/G$ is etale, the action of $G$ on $R$ gives a descent datum to descend the coherent sheaf $R$ from $A$ to $A/G$. This descent is the sheaf of algebras that gives $Y$.
One will not be able to write $Y$ as a relative spectrum of something on $A$ in general, because $Y$ has no natural categorical map to $A$.
