Here are three research traditions that both illustrate how the problem can be approached, and give rather different perspectives on what counts as a logic.
(this really is just to complement Dan Piponi's answer).
- Tarski's consequence relations and abstract algebraic logic
Tarski's basic idea was to define a logic as an abstract pair of the form $\langle \mathcal{F},C\rangle$ where $\mathcal{F}$ is a free algebra of formulas and $C$ is an operator on $\mathcal{P}(\mathcal{F})$ [I write $\mathcal{F}$ for the domain of the free algebra]. For any set of formulas $A$, the set $C(A)\subseteq\mathcal{F}$ is meant to represent the 'consequences' of $A$ -- so that $C$ generates a consequence relation $\vdash_{C}$ defined as $A\vdash_{C} B$ iff $B\subseteq C(A)$.
Next, Tarski gave several structural conditions that the operator $C$ ought to satisfy in order for the resulting consequence relation to count as well-behaved, or logical (roughly, those conditions consist of reflexivity, monotonicity, compactness, as well as invariance under uniform substitution of variables). See here for more detail.
This view really treats logic as a (very special) branch of abstract algebra. One idea is to try to differentiate between logics, and classify them, by looking at their different algebraic properties. It was one of the earliest systematic attempts at answering the question of 'what a logic is' in such general terms.
On the other hand, this framework is generally too weak to account for quantification of any sort; the attention is almost exclusively restricted to propositional logics.
(NB. Tarski's approach eventually gave rise to some very interesting work on the general process of algebraization of a logic, under the guise of Abstract Algebraic Logic -- see, e.g. here. Interesting monographs on the topic include Rasiowa and Sikorski's An Algebraic Approach to Non-Classical Logics as well as Blok and Pigozzi's Algebraizable Logics.)
- Model-theoretic logics, generalized quantifiers
For an introduction see this book. The model-theoretic approach studies various extensions of first-order logic: predominantly infinitary logics of the form $\mathcal{L}_{\alpha\kappa}$, where $\alpha, \kappa$ are ordinals (the logic $\mathcal{L}_{\alpha\kappa}$ allows conjunctions/disjunctions of less than $\alpha$-many formulas, and quantification over less than $\kappa$-many variables). It also covers topics like abstract characterisations of first-order logic (cf. Lindstrom's theorem mentioned in John Goodrick's answer) as well as connections with probabilistic logics.
There is related research on what makes a quantifier 'logical'. The idea is to characterise the logical quantifiers as operations of a certain type that are invariant under certain groups of transformations -- there appears to be some controversy about what exact transformations truly define the 'logical' operations (see here).
- Applied logics
Another approach that gives a slightly different perspective on what counts as a logic is work in `applied' logic: this is a broad field of study which has at its root a dynamic view of logic (see here). Here, one uses so-called 'dynamic' modal logics to model processes that change over time, such as transitions between the states of a program (see Propositional Dynamic Logic) or informational states of agents (see Dynamic Epistemic Logic). Those logics are studied either for their intrinsic mathematical interest, or can also be applied to the study of information exchange protocols in game theory, cryptography, and various topics in formal philosophy.
The approach here is less algebraic, but focused more on model-theoretic and computational aspects. This research often bears close links to computer science and philosophy.
