@quid. In all cases, thank you much for the fruitful conversation. I won't accept your answer only because I would like to hear other voices on the same issue, and possibly some categorist. Lastly, I must really add N. Hegyvari and I. Ruzsa to the list of the people working on the subject. I feel it as necessary. :)

You have nothing to be sorry about! As a minor remark, there is no loss of generality in assuming that the $\varepsilon_i$'s are all equal to 1, as far as the abelian case is concerned. I deliberately phrased the whole stuff in the form that I did to highlight that everything goes through verbatim in the non-commutative setting. On another hand, it is somewhat apparent, I think, that the true problems arise when trying to develop these (basic) ideas further and approach the question from the top (and not from the bottom). In any case, I'm happy to know that this has good chances to be new.

a $s$-morphism. For if $x_1, \ldots, x_s, y_1, \ldots, y_s \in X_1$ and $\varepsilon_1, \ldots, \varepsilon_s \in \{\pm 1\}$ are such that $\sum_i \varepsilon_i x_i=\sum_i \varepsilon_i y_i$ then $\sum_i \varepsilon_i\phi(x_i) =\sum_i \varepsilon_i\phi(y_i)$ (since $\phi$ is a $s$-morphism from $(\mathfrak A_1, X_1)$ to $(\mathfrak A_2, X_2)$), to the effect that $\sum_i\varepsilon_i\psi(\phi(x_i))=\sum_i\varepsilon_i\psi(\phi(y_i))$ (since $\phi(x_i),\phi(y_i)\in Y_1$ and $\pi$ is a $s$-morphism from $(\mathfrak B_1,Y_1)$ to $(\mathfrak B_2,Y_2)$.

$X_i \subseteq \mathfrak A_i$ and $\phi: X_1 \to X_2$ a function (I'm slightly abusing notation, but it's not a great issue), such that blah blah blah. Thus, I don't see any problem here: If $\phi: (\mathfrak A_1, X_1) \to (\mathfrak A_2, X_2)$ and $\psi: (\mathfrak B_1, Y_1) \to (\mathfrak B_2, Y_2)$ are two $s$-morphisms, one defines the composition of $\phi$ with $\psi$ iff $(\mathfrak A_2, X_2) = (\mathfrak B_1, Y_1)$, as expected, accordingly setting it equal to the arrow $h: (\mathfrak A_1, X_1) \to (\mathfrak B_2, Y_2)$ for which $h = \psi \circ_{\bf Set} \phi$. This is still (...)

Thank you once more for your (rich) comments. Indeed, I've just realized that also Tao and Vu work out only the abelian case (I had forgot that they speak of additive groups wherever commutativity is implied). As for the mathematical question that you pose, I don't see a problem here. It really depends on the way you're defining your category. Though it doesn't use explicitly this wording Additive Combinatorics defines - let me say, very appropriately - a Freiman $s$-morphism as an arrow $\phi: (\mathfrak A_1, X_1) \to (\mathfrak A_2, X_2)$, with $\mathfrak A_i$ a (commutative) group (...)

Well, it seems that I've already checked them all. As for your question on possible applications of a categorical framework to approach the basic theory, I think that there is already broad evidence that looking at (fundamental, foundational) questions from a categorical point of view is likely to spread our horizons beyond any apparent imagination and open new paths towards unknown universes waiting for being explored. It may sound perhaps too romantic, but looks more than sufficient (to me) to deserve our efforts.

(...) of the 1996 ed. of ANT - Inverse Problems and the Geometry of Sumsets (as a side note, only in the abelian setting). (3) Grynkiewicz, in his presentation, deals uniquely with abelian groups. In the first part, he introduces the idea of a Freiman morphism of a sum-set (together with other basic material). In the second part, he discusses problems relating to what he refers to as the universal ambient group (shortly, UAG) of a sum-set. (4) Idem for Konyagin and Lev, who develop a linear algebra method (Theorem 4) to solve a couple of questions raised by Freiman himself.

Thanks for the references, I will check them and come back with more comments. For the moment, some preliminary considerations: (1) Additive Combinatorics introduces Freiman morphisms in Sect. 5.3 (at the least, in the 2006 ed. on my shelf), but it is strictly focused on the case of groups (as you suspected). There is just a minor remark pointing out an analogy with the differential geometry of manifolds, that's all. I may be wrong, but I guess that the 2010 ed. has not been extended in this respect. (2) It goes the same with Nathanson, who presents Freiman morphisms in Sect. 8.2 (...)

...a decade before Carmichael himself realized an error in his earlier "proof" (R.D. Carmichael, Note on Euler's $\phi$-Function, Bull. AMS, Vol. 28 (1922), pp. 109-110), with the result that the question is still today broadly open. In spite of the fact that Dickson, in the 2005 edition of his History of the Theory of Numbers (Vol. 1, p. 137), states that the conjecture was proved in Carmichael's original 1907 paper.