Lagrange launched a challenge via the French Academy in the 1780s calling on authors to provide a clarification of the use of infinite and infinitesimal numbers in analysis. Carnot wrote an essay that did not win the competition but became a popular text on the foundations of the calculus.
An earlier challenge was issued anonymously by Pascal and involved the solution of certain problems related to the cycloid curve. A small number of mathematicians submitted solutions including the jesuit Lalouvere who was being informally advised by Fermat (who did not submit a solution himself). The result was of course a victory by Pascal and an embarrassment to Lalouvere who committed some errors along the way.