The abelian category of quasicoherent sheaves on a schemes determine the scheme. This is an old result of Gabriel ("<a href="http://www.numdam.org/item?id=BSMF_1962__90__323_0">des categories abeliennes</a>" 1962), proved in full generality by <a href="http://www.mpim-bonn.mpg.de/preprints/send?bid=3948">Rosenberg</a>. This means that, QCoh(X) does not only tell you the open subschemes of X but also gives you the structure sheaf ! I've known this result for some time but I had never looked at it in detail until today. I'll sketch what I have just learned hoping not to make big mistakes...
An abelian subcategory B of an abelian category A is said to be a <b>thick subcategory</b> if it is full and for any exact sequence in A
0 ---> M'---> M ---> M'' ---> 0
M belongs to B if, and only if M' and M'' do.
If B is a thick subcategory of A there is a well defined localization A/B, which is again an abelian category. A/B has the same objects as A and a morphism f:M--->N in A/B is an isomorphism if, and only if ker f and coker f belong to B.
Let T:A ---> A/B be the localization functor. Then B is said to be a <b>localizing subcategory</b> if B is thick and T has a right adjoint. The condition of being localizing can be rephrased only in terms of A and B. see Gabriel's thesis above (proposition 4 in chapter III).
Finally, if M is an object of A, we denote by <M> the smallest localizing subcategory containing M.
Now let X be a scheme, j:U ---> X an open embedding and i:Y ---> X its closed complement. Then there is a bunch of adjunctions between the categories of quasicoherent sheaves of U,X,Y: i<sub>* </sub>:QCoh(Y) ---> QCoh(X) has a left adjoint i<sup>* </sup>:QCoh(X) ---> QCoh(Y) and a right adjoint i<sup>! </sup>:QCoh(X) ---> QCoh(Y). On the other hand, the functor j<sup>* </sup>:QCoh(X) ---> QCoh(U) has a left adjoint j<sub>! </sub>:QCoh(U)--->QCoh(X) and a right adjoint j<sub>* </sub>:QCoh(U) ---> QCoh(X). This is sometimes called a recollement.
Let's assume that X is Noetherian and let A = QCoh(X). We have an exact sequence of abelian categories
0 ---> QCoh(Y) ---> A ---> QCoh(U) ---> 0
in the sense that the category QCoh(Y) happens to be a localizing subcategory of A and its quotient is identified with QCoh(U). The first map in the exact sequence is i<sub>* </sub> and the second j<sup>* </sup>. Moreover, I think that QCoh(Y) is the smallest localizing subcategory of QCoh(X) containing i<sub>* </sub>O<sub>Y</sub>. Gabriel proves that there are no more such localizing subcategories, that is closed subschemes of X correspond exactly to localizing subcategories <M> generated by a single coherent sheaf (i.e. Noetherian object in A). Moreover, irreducible closed subsets (the points in the underlying topological space of X) are given by localizing subcategories <I> for I an indecomposable injective. We have described the points of X and its closed sets in terms of only the category A, so we can recover the underlying topological space of X from A.
In particular, an open subscheme U of X gives a complementary closed subscheme Y, which is in correspondence with a localizing subcategory <M> and, moreover, QCoh(U) = A/<M>. So, responding to the queston above, for any f:U ---> X, U is an open subscheme if, and only if the kernel of f<sup>* </sup>:QCoh(X) ---> QCoh(U) is a localizing subcategory of the form <M> for a coherent sheaf M.
Regarding the structure sheaf O<sub>X</sub> there is an isomorphism O<sub>X</sub>(U) and the ring of endomorphism of the identity functor on QCoh(U) (which happens to be A/QCoh(Y)), so the structure sheaf can be recovered only in terms of the category A.
Finally, just say that there are other results in the spirit of reconstructing a scheme from some category of sheaves on it. This is the starting point for using such categories of sheaves as a definition of noncommutative scheme. There is more information on <a href="http://ncatlab.org/nlab/show/noncommutative+algebraic+geometry">this entry</a> in nlab.