Wikipedia credits Bourbaki with coining it, but doesn't provide a source. Does anyone happen to know the motivation for using this term?
The second definition in http://www.larousse.com/en/dictionnaires/francais/magma/48543 would seem to answer your question and agrees with the wiki page.
Update: here is what the link said to avoid broken links as YCor pointed out.
nom masculin (latin magma, résidu, du grec magma)
Mélange formant une masse pâteuse, épaisse : Magma informe.
Mélange confus, inextricable de choses abstraites : Ces propositions constituent un magma incohérent.
Matériau constitué de liquide et de cristaux en proportions variables, qui se forme à l'intérieur de la Terre et qui, en refroidissant, forme une roche.
Wikipedia also says that "magma" is used by Serre in his book Lie algebras and lie groups: 1964 Lectures given at Harvard University. This seems to be the case (at least for the 1992 Springer reprint that I have access to.) Is this the earliest use in print of "magma" in this sense?