Note that if f: X-> Y →Y is a categorical quotient in the category of schemes which is stable under base change by open immersions, then the second condition (ie. O_Y-> (f __* Y→(f* O_ _X)^G X)G is an isomorphism) is automatically satisfied.
In the paper "Examples and counterexamples for existence of categorical quotients" by A'Campo-Neuen and Hausen, there is an example of a categorical quotient f: X-> A^1 →A1 such that f^{-1}(A^1 f-1(A1 - 0)-> A^1 →A1 - 0 is not a categorical quotient. I haven't checked but I believe this should also give an example where condition (2) fails.
I don't know of example of a categorical quotient where condition (1) fails.
