Generalised operad structures We can naively consider an operad as a collection $\{P(n)\}_{n\geq 0}$ of vector spaces $P(n)$ consisting of "functions" with $n$ inputs and one output, equipped with a number of compositions
$$P(m)\times P(n)\to P(m+n-1)$$
given by attaching the output of an element of $P(n)$ to one of the inputs of an element of $P(m)$.
A dioperad generalises this to allow multiple outputs. So we have a collection $\{P(n,m)\}$, and a composition
$$P(n_1,m_1)\times P(n_2,m_2)\to P(n_1+n_2-1, m_1+m_2-1)$$
where we attach one output to one input.
My question is whether there is a similar structure in the literature where we allow ourselves to attach multiple inputs to multiple outputs? Some sort of poly-operad
 A: There is a general framework for working with these generalized operad structure in the book A Foundation for PROPs, Algebras, and Modules (called Foundation below).  A pre-publication version is here.  In Foundation Chapter 10, the concept of a G-PROP is defined, where G is a pasting scheme (Foundation Chapter 8).  Foundation Chapter 11 shows that operads, dioperads, half-PROPs, properads, PROPs, wheeled operads, wheeled properads, and wheeled props, among others, are examples of G-PROPs for appropriate pasting schemes G.  For attaching multiple inputs to multiple outputs, you should consider properads, PROPs (as Denis Nardin pointed out above), and wheeled PROPs in Foundation Sections 11.7-11.9.
The original concept of a PROP is due to Mac Lane in Natural associativity and commutativity, Rice Univ. Stud. 49(1) (1963), 28-46.  Notice that this paper predated both Lambek's multicategories (in Deductive systems and categories. II. Standard constructions and closed categories, 1969 Category Theory, Homology Theory and their Applications, I (Battelle Inst. Conf., Seattle, Wash.), pp. 76–122, Springer, Berlin, 1969) and Szabo's polycategories (in Polycategories, Comm. Algebra 3 (1975) 663-689).
