It doesn't "stop being a mathematical operation". The syntax, itself, is a mathematical structure that can be, and is normally, formalized as a magma or as its generalization to different signatures, which are called term algebras for that signature. The elements of a free object in a term algebra are, in fact, one and the same as an abstract syntax tree over that signature, and it is here that you formalize such notions as abstract syntax tree. This is where the "syntax" or "meta level" - as you refer to it - actually lives.

This gets more directly to the question: what type of functional or operator is this: $λx(\_)$; i.e. $y ↦ λx(y)$, when we want bound $x$'s in $y$ to be regarded as such? I think you need to fall back to term algebras, and to magmas (or whatever the generalization of magma's is called) to formalize that as a function that respects the handling of bound variables. It's just easier to skip the formalization and describe it as just a syntactic functional, instead; somewhat like Landin's "let (_) = (_) in (_)".

You need to be more specific. This one: $\frac{d}{dx}(y) = Dλx(y)$? The $(\_)$ place-holder is not an argument in the usual sense, because it can take bound variables. So it has to be read as something at the "alpha level", so to say, rather than "beta level"; and the place holder is more akin to being of what's referred to as a "reference type" in programming languages. It is a syntactic functional, maybe formalized as such in the Magma (or generalization thereof) underlying the Term algebra.

Ok. I'll remove the editorial, if you wish.