It depends on context. In the physics literature, there is a term "exactly solvable" meaning that a closed form for the solution can be written; it is never used to indicate that the solution exists in an abstract sense. E. g., see Baxter's classical book "Exactly solvable models in Statistical mechanics". So, in this context "exact solution" does mean "closed form solution".
Edit inspired by Alexandre Eremenko's answer: more precisely, for a physicist, "exact solution" should be explicit enough to answer all the questions they care about. In practice, these questions often boil down to the asymptotics at singular points and/or infinity. Say, Baxter's exact solutions are primarily interisting because they provide the critical exponents. In that sense, Painleve functions should count as closed form, since the asymptotical expansions and connection formulae are known for them, and Sundman's solution is not.
In other context, you may talk about approximate or perturbative solution, and then I feel it is fine to contrast it to the "exact solution" even when the closed form for the latter is not known.