Here are some examples, ranging from the comical to the debatable.
Comical: Pretty much any mention of mathematics in Jacques Lacan. To give you an idea, here is a typical passage:
This diagram [the Möbius strip] can be considered the basis of a sort of essential inscription at the origin, in the knot which constitutes the subject. This goes much further than you may think at first, because you can search for the sort of surface able to receive such inscriptions. You can perhaps see that the sphere, that old symbol for totality, is unsuitable. A torus, a Klein bottle, a cross-cut surface, are able to receive such a cut. And this diversity is very important as it explains many things about the structure of mental disease. If one can symbolize the subject by this fundamental cut, in the same way one can show that a cut on a torus corresponds to the neurotic subject, and on a cross-cut surface to another sort of mental disease. [Lacan (1970), pp. 192-193]
And here's another one:
Thus, by calculating that signification according to the algebraic method used here, namely $$\frac{S(\text{Signifier})}{s(\text{signified})} = s(\text{the statement})$$ with $S=(-1)$ produces $s=\sqrt{-1}$[...]Thus the erectile organ comes to symbolize the place of jouissance, not in itself, or even in the form of an image, but as a part lacking in the desired image: that is why it is equivalent to the of the signification produced above, of the jouissance that it restores by the coefficient of its statement to the function of the lack of signifier -1. [Lacan (1971); seminar held in 1960.]
Interesting/Rigorous but still quite a stretch: The work of Alain Badiou on set theory, although more rigorous and advanced, also provides a very good resource for misapplications of formal mathematics in order to draw non-mathematical conclusions, cf. especially Being and Event which is his magnum opus, in which he uses set theory to support the tagline that 'Mathematics is Ontology'. Unlike Lacan, Badiou at least knows his stuff when it comes to the statement and development of formal results. That said, his interpretations and conclusions are often huge stretches.
Here's a related MO post on Badiou:
Badiou and MathematicsBadiou and Mathematics
Interesting/Philosophy: I don't know if you'd call these misapplications, but they are certainly attempts to use formal results to draw philosophical conclusions that are not in any formal way entailed by those results. Here are some examples:
- Michael Dummett on how Godel Incompleteness might/might not threaten the thesis that meaning is use (philosophical anti-realism):
The philosophical significance of Gödel's theorem, M Dummett - Ratio, 1963
- Hilary Putnam on how the Lowenheim-Skolem Theorem proves that reference is underdetermined by all possible theoretical or operation constraints (i.e. that the meaning of our mathematical vocabulary can never be accurately understood in order to fix an intended model):
http://www.jstor.org/stable/2273415
Pretty much anything philosophical that has been written about the so-called Skolem Paradox involves formal-to-informal entailments.
- Roger Penrose in The Emperor's New Mind again using Godel to draw conclusions about consciousness and mechanism