Here are two answers to this question based on different interpretations of the meaning of ``Koszul.''

One reasonable way to define things is that Koszul duality depends not only on algebraic data (operad, modular operad, algebra, whatever) but also on a presentation of the algebraic data. This is both elegant and a bit of a cheat. For example, in algebras, operads, or properads (let me just say operad for simplicity), take the "maximal" presentation of your operad. That is, take all operations as generators and then all relations of the form [(infinitesimal) composition of two generators] = [first generator] composed with [second generator]. Then the Koszul dual is precisely the bar construction. So you've totally diluted the meaning of Koszul duality by allowing every operad to have a "Koszul" presentation which doesn't tell you anything new. But I think if you're a homotopy theorist, this is a pretty reasonable definition.

On the other hand, maybe you want something more rigid or more amenable to calculation. So let's start over. One motivation for Koszul duality for an operad $P$ is that you have a canonical (large) resolution of $P$, the cobar bar resolution $\Omega B P\to P$ but you'd like something smaller if you can get it without losing the simplicity of the differential. Typically you hope that your resolution will be of the form $\Omega P^!$ for some cooperad $P^{!}$ (these exclamation points should be upside down but I can't figure out how). If $P^!$ has no differential of its own, this will be the unique-up-to-isomorphism minimal model for $P$. So one natural candidate is for $P^!$ to be the homology of $BP$. This inherits cooperadic structure from $BP$ and as such lives in the right category to apply the cobar functor $\Omega$.

However, there are three problems. The first one, which is serious, is that we don't know that $BP$ is formal, i.e., quasi-isomorphic to its homology not just linearly but as a cooperad. Typically the homology will have to be given higher homotopy cooperations and then there's no hope for $\Omega H_*(BP)$ to be a resolution of $P$. But maybe you get lucky and you can construct this quasi-isomorphism. The second problem is computing this homology. In the classical definition of Koszul duality for operads, you don't take $H_*(BP)$ as a definition, you take one particular linear summand of $H_*(BP)$ which is easy to compute, based on a grading that comes from a quadratic presentation. Then you cross your fingers and hope this summand exhausts the homology. So say it does. The third problem, which is not such a big deal, is that $\Omega$ doesn't preserve all quasi-isomorphisms. In practice you can use the same grading on $BP$ that passes to its homology to ensure that this is one of the good quasi-isomorphisms preserved by $\Omega$.

So let's try to mimic this in modular operads. Define $P^!$ as the homology of the Feynman transform (in this paragraph the exclamation point should not be upside down because of the dual involved in the Feynman transform), and call P Koszul if $P^!$ and the Feynman transform of $P$ are weakly equivalent as modular operads. As you point out, it's not obvious what quadratic means in this context, so let's skip the second problem altogether. Then it should be an exercise to ensure that even though you haven't used the grading of a homogeneous presentation, in most (all?) cases of interest, the Feynman transform preserves quasi-isomorphisms so we get the desired behavior from $P^!$, that it generates the minimal model for $P$.