Certainly one can ask for the data of a (unital or nonunital) $A_\infty$-algebra $A$ equipped with a "comultiplication" map $A \to A \otimes A$ of $A_\infty$ bimodules. Of course, this is infinitely much data --- all the higher coherences explaining how the comultiplication intertwines with the left and right actions. I will call these higher coherences "(higher) Frobeniators".
Such data will not, in and of itself, give the notion you are after, because it does not include the coassociativity data. It also clearly does not suffice to ask that the underlying map of vector spaces $A \to A \otimes A$ be part of a co-$A_\infty$ structure, as then you may not have good coherence between the (higher) coassociators and the (higher) Frobeniators. Recall that the coassociator is a homotopy between two different maps $A \to A \otimes A \otimes A$. What I expect is that you want to ask that it be a homotopy of maps of $A_\infty$-bimodules.
Almost surely that will give you a good notion of "open $A_\infty$ Frobenius algebra", although I don't know if/where it's been studied in the literature. If it has, almost surely it's in the context of Poincare duality.
Here's another option. Following Gan, call by the name dioperad a collection of chain complexes $\{P(m,n)\}_{m,n\in \mathbb N}$ with symmetric group actions and associative ``composition'' maps indexed by directed, but not necessarily rooted, trees. Dioperads are one possibly many-to-many generalization of operads which, unlike cyclic operads, admit a straightforward theory of Koszul duality. The open and coopen Frobenius dioperad is the one for which $P(m,n)$ is empty if $mn=0$ and otherwise is the free bimodule for the symmetric groups on $m$ and $n$ letters. It is generated by a class in $P(1,2)$ and a class in $P(2,1)$ subject to "associativity", "coassociativity", and "Frobenius" relations.
This dioperad is known to be Koszul (which follows e.g. from the Koszulity of the associative operad along with some results of Vallette), and its homotopy algebras are a reasonable notion of "homotopy Frobenius algebra". I believe, but have not checked, that they are precisely the same as the "open $A_\infty$ Frobenius algebras" mentioned above.
Dioperads only involve trees, and so everything in, say, "homotopy Frobenius algebra" when defined in terms of dioperads only involves trees. So let me actually call such things "tree-level homotopy Frobenius algebras". There is another reasonable many-to-many generalization of operads known as PROPs, which involve graphs. In particular, there is a PROP who describes Frobenius algebras. PROPs do not admit as good a Koszul duality theory, making it hard to work out "homotopy algebra", but pretty much every PROP in nature is the universal enveloping PROP of a properad, a notion due to Vallette, and moreover Vallette has shown that the universal enveloping functor from properads to PROPs is exact. In particular, there is a properad of Frobenius algebras (generated by the same presentation as the dioperadic version), and PROPic and properadic homotopy algebras thereof are equivalent. I will call homotopy algebras for the properad/PROP of Frobenius algebras "graph-level homotopy Frobenius algebras", since PROPs and properads involve non-tree graphs.
An aside: It is very hard to prove that the properad of Frobenius algebras is Koszul, and I don't think a proof is in the literature. Its commutative cousin is known to be Koszul by arXiv:1402.4048, but that paper was a long time coming and required some new techniques. Without Koszulity, you can still work out what a large presentation of the notion of "homotopy ...", but not a small one.
Now for the warning: the universal enveloping functor from dioperads to properads is NOT EXACT. So you should not expect the notions of graph- and tree-level homotopy Frobenius algebra to be equivalent.
I studied some related questions in arXiv:1412.4664 and arXiv:1307.5812, where I showed that tree-level homotopy "shifted commutative" Frobenius algebras can fail to extend to graph-level ones, or if they do extend then the extension can fail to be canonical. Indeed, the extension from tree- to graph-level is a "perturbative quantization" problem closely related to Feynman diagrams in quantum field theory.
The punchline is just that you should pay attention to which version of "homotopy Frobenius algebra" you want — graph- or tree-level — and be careful not to assume you have one when you've built the other.