The abelian category of quasicoherent sheaves on a schemes determine the scheme. This is an old result of Gabriel ("des categories abeliennes" 1962), proved in full generality by Rosenberg. This means that, QCoh(X) does not only tell you the open subschemes of X but also gives you the structure sheaf ! I've known this result for some time but I had never looked at it in detail until today. I'll sketch what I have just learned hoping not to make big mistakes...
An abelian subcategory B of an abelian category A is said to be a thick subcategory if it is full and for any exact sequence in A
0 ---> M'---> M ---> M'' ---> 0
M belongs to B if, and only if M' and M'' do.
If B is a thick subcategory of A there is a well defined localization A/B, which is again an abelian category. A/B has the same objects as A and a morphism f:M--->N in A/B is an isomorphism if, and only if ker f and coker f belong to B.
Let T:A ---> A/B be the localization functor. Then B is said to be a localizing subcategory if B is thick and T has a right adjoint. The condition of being localizing can be rephrased only in terms of A and B. see Gabriel's thesis above (proposition 4 in chapter III).
Finally, if M is an object of A, we denote by <M> the smallest localizing subcategory containing M.
Now let X be a scheme, j:U ---> X an open embedding and i:Y ---> X its closed complement. Then there is a bunch of adjunctions between the categories of quasicoherent sheaves of U,X,Y: i* :QCoh(Y) ---> QCoh(X) has a left adjoint i* :QCoh(X) ---> QCoh(Y) and a right adjoint i! :QCoh(X) ---> QCoh(Y). On the other hand, the functor j* :QCoh(X) ---> QCoh(U) has a left adjoint j! :QCoh(U)--->QCoh(X) and a right adjoint j* :QCoh(U) ---> QCoh(X). This is sometimes called a recollement.
Let's assume that X is Noetherian and let A = QCoh(X). We have an exact sequence of abelian categories
0 ---> QCoh(Y) ---> A ---> QCoh(U) ---> 0
in the sense that the category QCoh(Y) happens to be a localizing subcategory of A and its quotient is identified with QCoh(U). The first map in the exact sequence is i* and the second j* . Moreover, I think that QCoh(Y) is the smallest localizing subcategory of QCoh(X) containing i* OY. Gabriel proves that there are no more such localizing subcategories, that is closed subschemes of X correspond exactly to localizing subcategories <M> generated by a single coherent sheaf (i.e. Noetherian object in A). Moreover, irreducible closed subsets (the points in the underlying topological space of X) are given by localizing subcategories <I> for I an indecomposable injective. We have described the points of X and its closed sets in terms of only the category A, so we can recover the underlying topological space of X from A.
In particular, an open subscheme U of X gives a complementary closed subscheme Y, which is in correspondence with a localizing subcategory <M> and, moreover, QCoh(U) = A/<M>. So, responding to the queston above, for any f:U ---> X, U is an open subscheme if, and only if the kernel of f* :QCoh(X) ---> QCoh(U) is a localizing subcategory of the form <M> for a coherent sheaf M.
Regarding the structure sheaf OX there is an isomorphism OX(U) and the ring of endomorphism of the identity functor on QCoh(U) (which happens to be A/QCoh(Y)), so the structure sheaf can be recovered only in terms of the category A.
Finally, just say that there are other results in the spirit of reconstructing a scheme from some category of sheaves on it. This is the starting point for using such categories of sheaves as a definition of noncommutative scheme. There is more information on this entry in nlab.